Someone has broken into your systems. Now what?

So, you've been hacked. Compromised. Breached and violated. Some criminal Goldilocks has been inside your network and found that your data was neither too hot nor too cold but just right. What are you going to do about it?

This could happen to any organisation and what you do to mitigate the problem could define your public image and legal position for years to come.

Here is a playbook to help you get through the worst of it – and maybe even come out wiser on the other side.

Be prepared

If a CIO has been doing it right, the incident response process will have begun long before a system compromise was detected. Good preparation is key to eliminating headaches later on.

Let’s start by assuming that you’ve been hacked already. It seems to work for the NSA, which treats its network as constantly compromised.

Deborah Plunkett, head of the organisation's Information Assurance Directorate, told attendees at a cyber security forum in 2010: "There's no such thing as 'secure' any more.”

“Companies are split into two kinds: those that know they have been breached and those that don’t,” agrees Lior Arbel, CTO at IT security consulting group Performanta.

He adds that attackers these days go for different bounty, targeting data rather than simply wreaking havoc as they used to when he started working in security.

Your incident response plan will vary according to the degree of severity you are facing, says Mark Osborn, senior consultant at IT and security consulting firm MWR InfoSecurity.

“You may have one response if a salesman says his laptop has been infected with malware. You have a different set of processes if your main corporate database has been breached,” he says.

“So you know in advance which situations might make you call the CEO at 3am, and which simply require you to fire off an incident report to the support team.”

The details of an incident response plan may change but the big rocks remain the same for a serious event, says Robert Horton, European managing director of security consulting at NCC Group.

Follow the drill

“The army puts you through drills. It gets you to understand what high-level scenarios you should follow because there are certain steps that will remain the same across all incidents,” he says.

“You should get the big things right and stick to the plans, even if each incident should be slightly different at the sharp end.”

With that in mind, it is a good idea to practise the plan periodically so that a team can be ready to jump into action at the first sign of a significant breach.

One thing is constant: the need for a competent team. “There’s a lot of knowledge and skill that you must have inside the firm, and it should involve your own people,” says Rolf von Roessing, former international vice president of governance and IT security at non-profit ISACA.

The team should include all the usual suspects: incident response experts, of course, and a technical person or two. PR and legal representatives play an important role in avoiding lawsuits and limiting brand damage.

Corporate security should be involved because there may be effects on the non-IT infrastructure, Von Roessing adds. And senior management should also be on call for serious events.

Whoever goes on the response team should be given the proper access to do their jobs, says the security research SANS Institute in its handbook. That is particularly important if some of the team are from outside the company, which may be the case if your company is too small to muster the necessary resources.

That access doesn’t just involve setting and removing account permissions, either. Horton says an important first step is putting a team under legal privilege, which means those working on an incident response can't be forced to disclose details.

“If an outside firm comes in and does its work for two or three weeks, and then you decide that all the stuff that has been uncovered should have been privileged, it’s too late. Get it in as soon as possible,” he warns.

Spot the culprit

You have your plan and your team but one day, in spite of your best efforts, someone prises open your network and causes some damage. This is where the identification and investigation phase kicks in.

“The realisation might come days, weeks or months after the event,” says Osborn.

Be prepared to do some digging after alerting the response team, he advises. Solid data collection is key so that you can drill down into the data and understand what has happened.

“You should be collecting the logged information all day, every day,” he says.

There are various tools to do this, including Splunk, which provides operational intelligence based on logs. Security information and event management tools can be useful, as they aggregate log data from a variety of sources and alert you when different events correlate to indicate a threat.

Properly used, these tools will enable you to analyse historic data to find the root of a particular network incident.

The key rule here is that everything should be documented in anticipation of using it in court, says SANS. The document should cover what team members did in their investigations, in addition to who, where, when, why and the all-important how.

Damage limitation

Containment is where things get really interesting. The aim, of course, is to mitigate any damage that has already been done and to stop the attacker doing any more.

Throttling attackers involves understanding how they work. Experts explain that they gain access to a system and then escalate their privilege to gain access to more sensitive information.

Then when they have found the data they want, they steal it via exfiltration. In their quest for valuable data, some can siphon it off stealthily for long periods.

““You must ask yourself how can we split off the compromised user account? How can we lock down the server that was breached?” says Paul Nguyen, president of global security solutions for automated threat response firm CSG Invotas.

"What about the database or application layer: how can we close off whole event levels?”

Companies should take forensic backups of infected machines to preserve evidence

SANS breaks down containment into various stages. One is short-term mitigation – cutting infected networks off from the rest of the network, for example. This needs to be done as quickly as possible.

Nguyen advocates automation at this stage to expedite protection. His company provides tools designed to harden internal resources against the spread of a threat using automated scripts that configure devices on the fly.

SANS recommends system backup as a second stage. Companies should take forensic backups of infected machines to preserve evidence.

And finally, long-term containment involves fixing affected systems temporarily so that they can still be used as the team prepares to completely eradicate the threat. This includes removing backdoors and patching systems against the threats.

Waiting game

Von Roessing has his own take on the containment phase, which he calls “response”. In addition to mitigating damage, he advocates secret observation to find out as much about the intruder as possible.

“Resist the temptation to beat them over the head because you will lose a lot of intelligence if you do that,” he says.

“You must set up a honeypot to keep them distracted, while having your forensics team secure the evidence."

Everything you can find out about attackers will help you to get closer and work out where they are coming from. What IP ranges are they using? Is there a digital “handwriting” that they are using?

Technological approaches to containment are just one aspect of mitigating the effects of a breach. Another is notification. Companies must tick their legal and ethical boxes, notifying customers and regulators where necessary.

Paul Simmonds, CEO of the Global Identity Foundation and one of the original architects of deperimeterisation at the Jericho Forum, is sceptical about companies’ capabilities here.

“There is a playbook, which is don’t make it public unless you have to. If you go to your PR department, that’s what they will advise you,” he says.

“The next question is: what laws are in place to mandate notification? And the one after that is: can we get away with it if we don’t notify?” And that’s if the company is an ethical one, he adds.

Eradicate, eradicate

With the containment taken care of, it is time to eradicate the threat from your network entirely. When taking this step, document human resources and other costs that go into the effort.

This will help you to understand how much this part of the breach cost you, says SANS, while also providing proof that the toxins were removed from your systems.

Wiping and re-imaging systems is important, as is patching the re-imaged systems against the vulnerabilities that allowed the attack to happen in the first place.

Von Roessing’s team treats this as an exercise in disaster recovery because companies are taking systems out of operation, at least for a short while.

“Once you do that, you may have killed the attacker but you may have also killed yourself unless you have a strong disaster recovery and business continuity capability,” he says.

Many systems may need to be patched. Some hardware may need to be replaced entirely, and components of the network may have to be reconfigured.

“In between you run things in alternative or emergency operations mode,” he says.

Restored to health

The final two steps in the incident response process are recovery and improvement. Ensuring that the systems are clean and working properly involves a set period of monitoring them for abnormal behaviour.

The improvement process is your way to potentially come out ahead. Reviewing how the incident response was managed will hone your skills so that next time (and let's not fool ourselves here) you will be able to act even faster and more decisively.

The improvement phase goes beyond refining your incident response into an analysis of your security protections.

The attacker's behaviour may give you clues about how you can tighten up your security operations and tools to make another attack more difficult. Look for systematic weaknesses in your infrastructure and processes.

"Rather than closing a few loopholes, we should take a big step back and think holistically. What was the degree of vulnerability? Did we deter them?” says Von Roessing.

Without this broader analysis, the same attacker may be back for another try, he warns.

From preparation through to transformation, the incident response process should be a cyclical one. What is learned from one incident should be fed back into your preparation phase, creating a positive feedback loop.

Be sure of one thing: your attackers are busy perfecting their skills and finding new ways into your system.

Unless you are constantly improving, you will be at a disadvantage in a game with ever-growing stakes. ®