University Information Technology Services

The UITS Research SP System

Configuration of the IBM Teraflop SP System

The IBM Teraflop SP System, also know as the UITS Research SP, is comprised of two physical IBM RS/6000 SP systems connected to form one logical system which will have a total theoretical peak compute capacity of 1.005 TeraFLOPS (1.005 trillion FLoating OPerations per Second), a total memory capacity of 452 gigabytes (452 billion bytes), and a total disk capacity of 5.3 terabytes (5.3 trillion bytes). The currently installed ten SP frames hold a total of 616 processors within 143 SP nodes. These nodes are interconnected via two low-latency high-speed (150 megabytes/second) networks using crossbar switch technology, referred to as the SP Switch. An SP Switch router, specially designed to support SP Switch adapter cards, links the two networks. The two SP systems which comprise the IBM Teraflop SP System are referred to as the Aries Complex and the Orion Complex.

The Aries Complex includes eight frames of 4-cpu Power3+ Thin nodes, providing a homogeneous parallel processing environment for 508 processors with a total of 453 gigabytes of distributed memory.

The Orion Complex houses two frames: one frame of 4-cpu Power3+ Thin nodes, and another frame of 16-cpu Power3+ High nodes. With 8 gigabytes of memory, these Power3+ High nodes provide an excellent symmetric multiprocessing (SMP) environment for large-memory parallel jobs. Indiana University will add yet this fall another powerful SMP system, a 16-cpu Power4 Regatta node, to the Orion Complex. The Regatta incorporates IBM's newest processor and node technology, designed to provide excellent performance for both scientific and commercial/database workloads. (Further information about Power4 and Regatta technology is available at http://www.ibm.com/servers/eserver/pseries/news/pressreleases/2001/oct). The Orion Complex is currently located with the Aries Complex in Bloomington, but will be relocated to Indianapolis in the near future to better support researchers at the School of Medicine.

Within the IBM TeraFLOP SP, each node runs its own copy of the IBM's Unix operating system, AIX, but system managment of this large cluster is integrated and facilitated via IBM's SP management software, PSSP (Parallel System Support Program). The entire system is accessed via a single batch scheduler, IBM's LoadLeveler package, such that a user may log in to any of the SP nodes and submit a batch job, and this job is then dispatched to a suitable node with the required hardware, software and available cycles to run the job. While a large majority of the cycles consumed on the IBM Teraflop SP are used by parallel programs (i.e., programs which run simultaneously on several processors), there is also a significant amount of serial (i.e., single-cpu) work performed on this system. A wealth of software is available, including several commercial statistical and mathematical packages, scientific/numerical libraries, and database applications.

Of benefit to all users but especially to parallel programmers is a high-performance parallel filesystem, IBM's GPFS (General Parallel File System), which supports the reading and writing of data simultaneously to and from several nodes across several disks, but which also supports the usual Unix file system commands. The Aries Complex access a 570 gigabyte GPFS filesystem, while a 144 gigabyte GPFS filesystem is accessible to the nodes of the Orion Complex. In addition, the Teraflop SP is connected via the SP Switch router to the mass store system, another IBM RS/6000 SP system which runs the High Performance Storage System (HPSS) for fast archival and retrieval of large amounts of data to and from disk cache and tape.

Research SP Home Page

UITS | Advanced IT Lab