Why IR Fails

Seems like just about every week the news tells us of a new critical computer vulnerability, or a new data breach, or a new hack that is either bring a company to it's knees or threatens the entire internet. What most of the world doesn't get to hear about is the multiple work streams and heroics that goes on within companies once these vulnerabilities or breaches get identified.

This is the world of incident response. And, for the most part - there is a lot that companies get right. But there are also a lot of ways that companies could improve to make this process more efficient.

As I work with companies either as part of an external incident response team assisting a companies core information security specialists, or working as an extension of a company's internal team I have come up with some general themes about what causes IR to be more challenging that necessary.

Team work:
Without the right core team, any project is doomed, however companies should remember that the IR team needs more than just the 3-50 people classified as information security or incident response.
It all starts with the team itself:


1 - People and staff management
What kind of people do you need for an IR team? Do you expect that the already overworked information security staff will have the necessary time to respond to bad things while still maintaining the security of the network? Have you made time to ensure that the main skills for IR (network investigation, forensic investigation, malware analysis, system administration) are covered by the IR team? What about cross training in the event that some of the team is out of reach during an incident. Who manages the schedule for the team to ensure that they are not all on vacation at the same time?

2 - The team is not empowered to get the assistance necessary during an incident to investigate, isolate and remediate
Has leadership conveyed (in writing and verbally) the all areas of the company are expected and required to work with the IR team? When an incident happens - is there a reminder sent by management to emphasize this point. While the IR team is running 24x7, is there support from different company lines (this will; include system administration, networking, workstation admins, AD team, IT, help desk, legal) that is readily available to assist whenever necessary?


3 - Communication is key!
Prior to the 'bad thing' happening, have you as a company looked at the many moving parts that make up your networks? Do you know who to contact to get a block added to a firewall quickly? Do you know what kind of equipment is running at all of the various sites and who is responsible for managing these assets? If the call needs to be made to fully disconnect from the internet for a time - who can make this decision and how do you get in touch with them and how do they communicate this decision?

4 - The users. Users should be considered as part of the team. In fact, they are the first line of defense (and failure).
If users are not aware of the IR team, they may be unwilling to work with them. Additionally, users can destroy evidence by trying to fix security problems themselves. Aware users can also be the first indication of a problem. And the sooner an incident is identified, the quicker the IR plan can be engaged, and hopefully damage can be contained quicker, data loss and downtime minimized, and a return to 'run the business' happens faster.

Planning
It is often said that every incident is different, so there is no way to really plan. It is also said that failure to plan is planning to fail.

1 - The IR plan is created to satisfy an audit requirement
Otherwise known as 'check the box' security. In this case, there is an IR plan that has been written, but the plan itself does not reflect what is truly necessary for response. Searching the web for "Incident Response Plan" and re-branding someone else's idea of best in class does not make for a successful plan.

2 - The plan was written, approved, then archived
If the plan itself is not updated on a fairly consistent (in my opinion every month would be great but I am an overachiever), then it may not be worth the paper it is printed on. Networks are ever-evolving beasts. What was the standard configuration when the plan was written may not apply - you may not even have the same networking gear anymore. Also, the people who are documented may have left the team or even the company, a reorganization may have moved responsibilities to a different team, distribution lists may have been removed, etc.

3 - The drill that never happens
There is a reason fire drills happen. This gets people accustomed to what they should do in the event of emergency and cuts down on the panic reactions. If your IR plan has never been exercised, when the real thing hits - do you really want people spending time panicking or looking for the documents? Make sure that you try out the plan periodically when people aren't in panic mode to get everyone comfortable with their role(s). This also lets you build on the plan, determine what works (and what doesn't) and as a side benefit will identify communication paths that are out dated.

The technical details
Tools & technology
Some tools used by the information security team are part of everyday operations (AV scanning, IDS, firewalls). Others may only be used when an incident is declared or an event requires in-depth investigation (malware analysis, forensics). What can happen is that the IR-specific tools fall into disuse. The licenses are not updated, or the specific hardware needed is lost (or walks out the door when staff leave). Keeping a solid inventory of the tools, who has access to them, who maintains the contract, and where all the moving parts is key.