'Climate change' event dishes up sous vide supercomputers

Call it climate or call it unusual weather, but last week's five-day heat wave in the Australian city of Melbourne got the sweat running at one of the country's major supercomputing facilities, the Victorian Life Science Computation Initiative (VLSCI).

The problem for sysadmins at VLSCI was simple: with temperatures remaining above 40°C for four days in a row, Melbourne's water got too hot, exceeding the intake specification for the facility's cooling system.

First to go was AVOCA, the 838.86 teraFLOPS Blue Gene/Q system that went live in 2012. On January 14, as the Melbourne City temperature approached 43°C, the VLCSI sys admins stated that “the water inlet temperatures have exceeded the safe limit for keeping AVOCA operational”.

Ten minutes later, the smaller MERRI and BARCOO (both IBM iDataplex x86 systems) also fell at risk of a sudden shutdown, and to cope, administrators stopped scheduling new jobs. So it remained overnight.

While the smaller supers were brought back online at 10 am on January 15, the near-42°C temperatures still kept AVOCA offline. With more work on January 16 – and in spite of the mercury getting close to 44°C, a single 8,192-core midplane of the machine was fired up.

And that's how things stayed through most of last Friday, which replicated Thursday's peak. A second midplane was fired up at 3pm. The machine was finally brought back into service after 5pm, after a cool change arrived.

The facility is now assessing modifications to the cooling system to help it cope with future extreme heatwaves.

Extreme weather has prompted Australia's Bureau of Meteorology to launch a pilot heatwave forecast service.

Meanwhile in the outside world, the heatwave was reportedly responsible for hundreds of cardiac arrests, according to Ambulance Victoria (one call every six minutes at the peak), and five hundred calls for heat exhaustion. It's not yet known how many deaths occurred as a result of the extreme weather.

A heatwave of last week's magnitude has not been recorded for decades in Australia, leading some to label the event a result of anthropogenic climate change. That claim's of course not been proven. But like most other data centre operators, Australia's seek to use low-energy cooling options whenever possible. When even the water supply can't deliver cold comfort, hotter days become a real problem for Reg readers, no matter the source of the heat. ®