Appro plunks Opteron super into US fission lab

Supercomputing upstart Appro International has scored a relatively big win, getting a large cluster of its Xtreme-X parallel machines installed and running at the Idaho National Laboratory, one of the little known labs of the US Department of Energy.

The INL is relatively unknown because nuclear fusion research gets all of the glory and headlines, and the Idaho lab, which was founded in 1949, was created to do research on nuclear fission, the much less glamorous but nonetheless actually usable form of nuclear energy here on Earth in the 21st century.

Appro's Xtreme-X Opteron-Linux super

INL researchers are beside themselves with glee upon taking delivery of the new Appro super, which weighs in at 91 teraflops and which they have nick-named "Fission" – what else? – because it will allow them to simulate a nuclear fuel rod as it operates within a simulated fission reactor. This was not possible on the INL's prior supercomputer, which had only one-sixth the number crunching oomph of fission. With the Fission machine, INL researchers are simulating how the metal cladding that wraps around a uranium fuel rod behaves as heat, pressure, and other forces perturb it during operation. The 3D simulation running on fission will cover the first 18 months of a fuel rod's operation; it is not clear how long these simulations will take to run on the machine.

To run its fission simulations, INL chose a fourteen-rack configuration of Appro's Xtreme-X machines, which have four sockets per server node. The Fission machine has 391 nodes with 1,586 of Advanced Micro Devices' Opteron 6136 processors running at 2.4GHz. These are the eight-core variants of the Opteron 6100s, which have an 80 watt thermal envelope. INL could have gone with the twelve-core Opteron 6100 series chips, in either 80 watt or 105 watt thermal bands, but the clocks on the twelve-core variants are a little slower and the price is a lot higher. INL could have used the new Opteron 6140, which was announced in February, but in 1,000-unit quantities, each of these 2.6GHz chips costs $989, compared to $744 for the Opteron 6136.

The Fission machine at INL has a total of 12,512 cores running at 2.4GHz to deliver that 91 teraflops. The server nodes are connected to each other using 40Gb/sec (quad data rate) InfiniBand switches from Mellanox Technologies. The Xtreme-X racks have liquid cooling in their doors to trap heat before it leaves the racks and get it out of the data center before it diffuses.

The Fission cluster uses Appro's Cluster Engine Management System, which runs either Red Hat Enterprise Linux or SUSE Linux Enterprise Server in a stateless mode. The Cluster Engine tool can rollback an OS image to a previous state as well as control patching of OSes on the cluster nodes. It also automatically discovers new nodes as they are added to the cluster and puts them to work.

In addition to announcing the INL win, Appro was one of the first vendors out the door saying that it supported Nvidia's new Tesla M2090 GPU co-processors for servers. The Tesla M2090 is a double-wide, fanless GPU co-processor that sports 6GB of GDDR5 memory and delivers 655 gigaflops of peak double-precision floating point performance.

Appro is sliding the M2090s into its GreenBlade blade servers, which come in a 5U rack-mounted chassis and which can house five two-socket servers and ten GPU co-processors – all without melting. The GreenBlade machines were launched last may when Nvidia was just starting to ship the earlier Tesla M2050 GPU cards. The M2090s can also be used in the Tetra supercomputer nodes, which cram four GPU co-processors and a single two-socket system into a 1U pizza box. The Tetra servers were also launched last May. Customers can also attach Tesla co-processors to their Xtreme-X and HyperPower clusters, and it is a wonder why INL has not done just that. ®