The server's down. At 3AM. On Christmas. You're drunk. So you put a disk in the freezer

On-Call The On-Call inbox is full to overflowing, so we felt like letting another reader's tale of late-night weirdness into the wild.

In this tale, reader “Nic” told us that “Back in the 1990s when being a web designer was a job and ISDN was considered hot stuff I worked as a sysadmin at an outfit strategically located above a guitar shop.”

That location made for some interesting moments when yet another version of Smoke on the Water drifted upstairs. At least Nic seems to have been spared Stairway to Heaven.

Anyway, one Christmas Eve Nic behaved appropriately by bending his elbow. Repeatedly. He then headed home, crashed out safe in the knowledge that the next day was Christmas … but was then rudely awakened to be told that a Sun server had died and all customer sites were down.

Nic staggered downstairs, hopped in his car, hopped out again because he was clearly too drunk to drive but was also drunk enough to decide that a bicycle he spotted was a great way to get to the office.

“And so at 3am I am bowling along through Camden and Regents Park to the office,” Nic recalls. “Or possibly cycling very slowly singing Merry Christmas Everybody and slowly sobering up, I can't recall.”

He made it to the office where he was joined by his boss, also in a less-than-fully-capable state due to festive revelry.

Nic's head cleared enough to figure out that the server's disk was making “little chuntering noises” while the “PROM monitor was doing a Gandalf and refusing to let me pass and boot the beast.”

“3am on Christmas Day is not a good time to look for SCSI disks that'll work in a Sun,” Nic tells us and we don't doubt it's a lousy time to expect service personnel in a hurry. Nic's pickled brain reached that conclusion, but also dredged up a memory of a disk remedy that involved freezing a disk and then whacking it hard.

“On December 23rd this would have been considered sacrilegious, but we had no choices,” Nic explained to us in his email to On-Call. So with his boss in attendance Nic's bag full of useful stuff was spilled onto the floor in a rapid search for a screwdriver, the disk came out of the server and “into the office's mini fridge freezer compartment.” Nic and his boss waited an hour, making the time 4:30am, then popped the disk back into the server in the hope of restoring service by 6:00AM as required.

The server wasn't happy: Nic says its filesystem was “totally trashed” and fsck was “offering to 'fix' each problem by deleting the inodes, essentially forgetting about the files.”

So Nic and his boss re-installed Solaris and got Apache up again. Which just left customer data to restore – the small matter of “a few thousand pesky web pages, all the configuration and a whole bunch of mail.”

At which point Nic and his boss just about gave up, starting with the symbolic putting-away-of-the-screwdriver.

Which was when Nic spotted something else in the bag of useful stuff: a DAT tape from a previous backup made just two days previously.

Next step: ufsrestore and crossed fingers, which turned to work because the data flooded back.

Nic says he and his boss shared what sound like surprisingly uninhibited hugs at this point, then disappeared into the morning to enjoy Christmas Day.

There's even a Christmas miracle in this story, because Nic says that come Dec 27 his budget was increased by the cost of “a real system with proper redundancy and backup processes.”

“Sometimes it takes the CEO looking destruction in the face before arcane requests for budget have meaning.”

Silly us: we blew a Christmas On-Call story months before the day. If you've a story from 3am on any day, festive or not, write to us and you may appear in a future edition of On-Call. ®