nav search
Data Center Software Security Transformation DevOps Business Personal Tech Science Emergent Tech Bootnotes BOFH

YOU are the threat: True confessions of real-life sysadmins

Who will save the systems from the men and women who save the systems from you?

By Stuart Burns, 19 Nov 2014

Some sysadmins will go to extremes to secure a network, viewing it (wrongly) as their property.

For proof, look no further than Terry Childs, the City of San Francisco sysadmin who lost his job and subsequently refused to give over the system's virtual keys to his superiors in 2008.

It took just under a million dollars, several weeks, and the concerted efforts of several equipment vendors to put things right.

Childs had configured the equipment (predominantly Cisco) so securely that not only did no other administrator have rights to the switches and routers, but configs were not saved – so any power loss or attempt to reboot the switch or router into recovery mode would not work.

Childs ended up serving four years' jail time. It was also generally assumed that his management also bore some responsibility for what happened and the way in which they handled the situation, ie, very badly! For businesses big and small the question becomes how does one actually defend or monitor these threats, real or perceived?

It is not always directly the administrator’s intention to do wrong. Sometimes a wrong command executed against the system can take out an entire infrastructure. Earlier this year we read about a university that saw an entire portion of its network destroyed when a wrong command was apparently given.

"One admin said that given the right amount, he would compromise the system. Interestingly, the administrator stated that the amount had to be big enough so that they would not have to work again. This decision was based on the fact no one would ever employ them again."

This kind of error is not unusual and I have seen it happen several times over the years. There is a certain amount of mitigation that can be done to help to reduce the possibility of such a failure. Basic steps include reviewing the system defaults, using intelligent settings and having elevated accounts for just this sort of action. Frequently, you will find that the program makers' version of "sane defaults" and sysadmins' versions of sane defaults are very different!

Digital spycraft

The final portion of this threat is what one could term “digital spy craft.” Everyone knows about United States Army soldier Bradley Manning and rogue NSA sysadmin Edward Snowden. The decisions they made were their own and they exposed damaging government secrets.

The scary fact is that everyone has a price and it isn't always a monetary one.

This is a fact that should scare administrators as principles and beliefs are not always obvious or easy to defend against.

I gathered a small group of admins who were prepared to talk informally and without attribution. The conversation lasted two hours and we talked about if they’d done anything and why.

The session revealed attitudes towards system compromises or handing over information that they shouldn't have.

Most system administrators said they would not compromise their principles or systems for any price. A few administrators, however, said that they would do so for the right price.

Silly money

One admin said that given the right amount, he would compromise the system. Interestingly, the administrator stated that the amount had to be big enough so that they would not have to work again. This decision was based on the fact no one would ever employ them again.

Some administrators saw it as a purely commercial decision based on the fact that even if they were caught, they would think of jail time as time out before they could collect on their ill-gotten gains. (All the above was based on the premise that the loot would be available after any incarcerations.) The general figures mentioned by those prepared to compromise systems were between 10 to 20 million in local currency.

According to one: “If I manage to get the money in such a way that I can reasonably hide it and use it when I get out… what's five years in the big house?”

One overriding aspect was that not one administrator would compromise critical life support systems, no matter what the price. There was also a sentiment of exactly the opposite where banks and multi-nationals were concerned.

As noted, not all systems administrators are driven by monetary gain. One administrator I spoke with, whom we will call Joe, had an interesting story. After he grew tired of non-stop cold calling by an offshore company selling a junk product and getting nowhere with pleas to stop, he decided to teach them a lesson.

Victim turned hacker

Being a security specialist, employed by one of the big names, he knew how to sniff out vulnerabilities. He decided on his plan of attack. The first thing Joe decided to do was to take out the phone infrastructure. The provider in question was using IP-based phones and was unfortunate enough to have a very badly configured infrastructure that was exposing a lot more than it should be.

Within a couple of hours the phone system had been compromised. Joe, now an unwanted intruder, misconfigured the system to cause them annoyance but not so much that the provider would be crippled beyond repair. The last thing he did before he logged out was to remove his phone number, along with several others from the system, as plausible denial. To cover his tracks, Joe had used several proxies in a chain where the penultimate jump was from China. China was chosen for the specific reason of making it more difficult to trace him.

This seems to be a common type of scenario within the world of the clued-up IT people. One of the most interesting books I have read – called Stealing the Network: How to own a Continent – covers just this type of scenario in the first chapter, where someone sought to compromise the web vendor that had wronged them by getting all the customer credit cards.

Perhaps the reason this form of retribution happens more with IT-related people is there is a ready avenue for reprisals against a company, without having to be physically near the company.

Inside the office, a normal user can create problems, but those problems can easily be overcome: deleted files can be restored from backup, for example. You have backups, right? Once an employee is out of the company, they should have no access to the system. VPN access should be terminated before they are!

Frequently, though, it’s sysadmins who have been able to slip back into the system and delete critical components of the infrastructure.

How can businesses defend against such threats? Securing the human is truly not that easy. Some bigger companies now implement more stringent background checks including financial screening and crime screening. The general view on these checks is that they have limited use.

When removing a sysadmin from their position, there are a few things that should be done.

How do you avoid this becoming a problem in the future?

Another administrator should disable the leaving administrator accounts, before or during any dismissal meeting. Don't forget the mobile phones and physical access, passes and such and any access to infrastructure. As anyone would understand, when this happens, both parties want the incident to be over. It would be prudent however to shadow them, just in case.

"Some bigger companies now implement more stringent background checks including financial screening and crime screening. The general view on these checks is that they have limited use."

Another tip I have learnt from speaking to a few forward-thinking administrators is to sit with the new administrator who is taking over from them and make sure that the root passwords are changed and that the former administrator's account is disabled. It helps ensure that an above-board administrator does not have the finger pointed at them should anything untoward happen.

This should go hand in hand with a formal HR process that is tested and followed to the letter. Not infrequently this would have stopped these data deletion issues and also saved the aggrieved administrator from becoming unemployable and from a possible penalties. It removes the temptation of 10 minutes of getting even resulting in a much longer jail term.

Writing this article, a few things came to light that I really didn't expect to see. We all know that accidents happen, and they can, to a degree, be dealt with as I outlined above. I also assumed, rightly or wrongly, that administrators usually wouldn't compromise or divulge information, other than for whistle-blowing as in the Snowden case. That assumption proved to be incorrect.

As for management dealing with the threat of a rogue administrator? There are limits to what can be done, despite what purveyors of data loss prevention software and other security providers might say. ®

The Register - Independent news and views for the tech community. Part of Situation Publishing