Sysadmin ignores 25 THOUSAND patches, among other sins
'If I ever see this bloke on a job I'll punch him in the face' says Reg reader
On-call And that's one of the easier chores our reader found himself faced with in a new temp job. Most weekends, our On-Call feature looks at the odd situations readers find themselves in when called to do something on a client site or in the dead of night.
This week we're making an exception for reader “Bill”, who rates himself as “just your average support engineer” with experience on the front lines at companies big and small.
“I have learned the hard way about things over the years as we all do,” Bill says. But nothing prepared him for a recent gig he describes as “support for some very, very important persons in a small office that accounted for about 40 per cent of the money flowing out of a large multinational”.
The job looked simple: for a month, he'd be supporting Windows Server 2003 and 2008 for 50-60 users with onsite Exchange, Active Directory and BlackBerry Enterprise Server. At the end of the month, a new full-timer would be aboard and Bill could move on to his next contract.
Day one was encouraging. After “all the spiel and introductions companies normally muster for more important people on their first day”, plenty of it in the boardroom, Bill was “handed an A5 piece of paper with the layout of the servers on site and some IPs and hostnames”.
Which is where the trouble started, because that document was maybe 40 or 50 per cent accurate.
Once inside, Bill found things in quite a state. A WSUS server he felt should really have its own box was running on the company's main SQL server that also hosted core financial applications. The WSUS box also had 25,000 patches – Bill's certain it was thousands, not hundreds – awaiting approval or declination.
Getting that sorted was his first day's work, at the end of which he noticed another small SQL box with historical stuff on it, which was running Windows Server 2000. The physical server was made in 2001 and boasted a built-in floppy drive.
Will left the WSUS server “buzzing the 5 Mbps WAN link overnight” and arrived to find "users not happy that they had to restart their machines, and some had a few hundred updates to apply”. Some were even running Windows XP and had also been left unpatched.
For about four years.
Next? Check the anti-virus. It was 15 months out of date because “at some point a firewall rule for it was disabled and now couldn't be re-enabled”. Bill's predecessor hadn't left the firewall's password behind, starting with a need to swamp the connection daily to access new virus signatures.
Scorched earth IT
Things got worse: Bill also found that the call system's default passwords had been changed, historical call reporting had been switched off and call management software erased.
“This meant that every day I needed to be the last one out to run the call stats. Every working sodding day!”
The VoIP provider was no help, as Bill found “they could not find the old default master/config passwords for the device and they couldn't manage it remotely as the idiot has also nobbled that firewall rule as well”.
Back-up was badly implemented so it wasn't deleting log files, which meant the Exchange box was full. So was a filer that, for no apparent reason, stored a copy of the ancient SQL server. And the back-up was also configured not to touch the filer if it was full.
Around this time, Bill learnt that his predecessor had given two days notice after a seven-year stint. Which might have been why the server room was a rat's nest and he spent his days on site “firefighting from one issue to another”.
Last week came the “best” mess of them all.
“A certain server hadn't been rebooted since well before idiot boy left,” Bill explains. “After all the updates, I decided I had a spare hour so would reboot it... and then the trouble began. Before I rebooted, it complained about disk space and the good old Windows disk space optimiser appeared claiming to be able to claw back three gig in 'un-needed' files.”
Bill trusted Windows and left it to do the job. He then rebooted and... nothing happened.
“The server came up OK but there was no logon box,” Bill says. After “checking the other boxes, it looked like most services had come online and after some digging I found why... the gina.dll had been borked as all the other Windows Server 2003 servers had the same data and time for their versions, this one had them set for the day idiot boy left”.
“As this was the server handling the back-up I decided to play it safe and see if I could restore it in some way to a back-up,” a tactic that sadly didn't work. “In the end, at 23:30 I ended up copying the needed files and the winsxs folders from a very similar box and then using the winpe bootdisk to dump them onto the relevant machine.”
Fortunately, “the machine booted up and server services returned to normal. There is now a sticker on the front saying not to power it off until migration work is completed”.
Bill now wishes to commit violence against his predecessor.
Over the years I've worked with some characters, but this idiot boy really needs to be shown how to use machines and I really REALLY feel for his current employer as they don't know what they are in for when he decides he's had enough and wants to go to another job like he did at that office.
If I ever come across this guy at a contract job I am simply going to punch him square in the face and walk off. I don't care if he makes Shrek look tiny, professional standards and morals should have kicked in before he did what he did!
Is there someone you'd like to punch? Keep yourself out of jail by instead telling us your story so we can turn it into a therapeutic future edition of On-Call. ®