Why IR Fails

Seems like just about every week the news tells us of a new critical computer vulnerability, or a new data breach, or a new hack that is either bring a company to it's knees or threatens the entire internet. What most of the world doesn't get to hear about is the multiple work streams and heroics that goes on within companies once these vulnerabilities or breaches get identified.

This is the world of incident response. And, for the most part - there is a lot that companies get right. But there are also a lot of ways that companies could improve to make this process more efficient.

As I work with companies either as part of an external incident response team assisting a companies core information security specialists, or working as an extension of a company's internal team I have come up with some general themes about what causes IR to be more challenging that necessary.

Team work:
Without the right core team, any project is doomed, however companies should remember that the IR team needs more than just the 3-50 people classified as information security or incident response.
It all starts with the team itself:


1 - People and staff management
What kind of people do you need for an IR team? Do you expect that the already overworked information security staff will have the necessary time to respond to bad things while still maintaining the security of the network? Have you made time to ensure that the main skills for IR (network investigation, forensic investigation, malware analysis, system administration) are covered by the IR team? What about cross training in the event that some of the team is out of reach during an incident. Who manages the schedule for the team to ensure that they are not all on vacation at the same time?

2 - The team is not empowered to get the assistance necessary during an incident to investigate, isolate and remediate
Has leadership conveyed (in writing and verbally) the all areas of the company are expected and required to work with the IR team? When an incident happens - is there a reminder sent by management to emphasize this point. While the IR team is running 24x7, is there support from different company lines (this will; include system administration, networking, workstation admins, AD team, IT, help desk, legal) that is readily available to assist whenever necessary?


3 - Communication is key!
Prior to the 'bad thing' happening, have you as a company looked at the many moving parts that make up your networks? Do you know who to contact to get a block added to a firewall quickly? Do you know what kind of equipment is running at all of the various sites and who is responsible for managing these assets? If the call needs to be made to fully disconnect from the internet for a time - who can make this decision and how do you get in touch with them and how do they communicate this decision?

4 - The users. Users should be considered as part of the team. In fact, they are the first line of defense (and failure).
If users are not aware of the IR team, they may be unwilling to work with them. Additionally, users can destroy evidence by trying to fix security problems themselves. Aware users can also be the first indication of a problem. And the sooner an incident is identified, the quicker the IR plan can be engaged, and hopefully damage can be contained quicker, data loss and downtime minimized, and a return to 'run the business' happens faster.

Planning
It is often said that every incident is different, so there is no way to really plan. It is also said that failure to plan is planning to fail.

1 - The IR plan is created to satisfy an audit requirement
Otherwise known as 'check the box' security. In this case, there is an IR plan that has been written, but the plan itself does not reflect what is truly necessary for response. Searching the web for "Incident Response Plan" and re-branding someone else's idea of best in class does not make for a successful plan.

2 - The plan was written, approved, then archived
If the plan itself is not updated on a fairly consistent (in my opinion every month would be great but I am an overachiever), then it may not be worth the paper it is printed on. Networks are ever-evolving beasts. What was the standard configuration when the plan was written may not apply - you may not even have the same networking gear anymore. Also, the people who are documented may have left the team or even the company, a reorganization may have moved responsibilities to a different team, distribution lists may have been removed, etc.

3 - The drill that never happens
There is a reason fire drills happen. This gets people accustomed to what they should do in the event of emergency and cuts down on the panic reactions. If your IR plan has never been exercised, when the real thing hits - do you really want people spending time panicking or looking for the documents? Make sure that you try out the plan periodically when people aren't in panic mode to get everyone comfortable with their role(s). This also lets you build on the plan, determine what works (and what doesn't) and as a side benefit will identify communication paths that are out dated.

The technical details
Tools & technology
Some tools used by the information security team are part of everyday operations (AV scanning, IDS, firewalls). Others may only be used when an incident is declared or an event requires in-depth investigation (malware analysis, forensics). What can happen is that the IR-specific tools fall into disuse. The licenses are not updated, or the specific hardware needed is lost (or walks out the door when staff leave). Keeping a solid inventory of the tools, who has access to them, who maintains the contract, and where all the moving parts is key.

IR and the help desk

Here's the scenario:
Get an email from the help desk saying that a user's machine has been infected with a Trojan (not just any malware, a Trojan!)

The machine was located in a different state from myself, and there is a lack of forensic tools for the organization, but I was able to get a colleague to create a memory image and disk image.  He then sent me the memory image to investigate.

Trying to get a little more detail, went back to the help desk and asked for details about what the user reported, what the help desk did, any key dates, & why they thought the machine was infected.  Turns out, the user said that files were disappearing and reappearing from her desktop.  The help desk then said that they had installed & run malwareBytes on the machine, but that didn't fix the problem.

The help desk said that they did not have any documentation of what they had done or when, but if I looked at the logs for malwareBytes that should tell me what I needed to know. (!)

With the idea that desktop files were the issue, I created a timeline from the memory image and used 'Desktop' as my pivot.  I was able to find that the user had been accessing the desktop files the day the issue had been reported, but also noticed that the Hidden attribute was set.  Hmmmm.... back to the help desk.

This time, they told me that they had gone into folder properties and set the 'Show Hidden Files..' option to true.  But that the files were still shadowed after they did this, so the machine must have a Trojan. 

Shadowed files and folders?  I had to go back and look at my machine to figure out what they meant by this one:

See how those files are lighter, faded looking?  That's shadowed (files that are marked as hidden files). Now we come to the idea that the files weren't being removed, they were just set as hidden. 

Back to the help desk to ask when the user first reported having problems.  They forwarded me an email from the user dated 4 days prior to the contact with InfoSec where the user said her issues with email had been fixed.  I still have no idea how this relates to the problem of hidden files. The also mentioned at this point that since running malwareBytes had not fixed the problem, they restored everything that had been quarantined.


The date of the initial report didn't really help, so working backwards in the timeline, I was able to find a point in time where the Users folder appears to have had the 'Hidden' attribute set.  When this attribute is set manually (right-click, properties, check hidden), the pop up screen gives you the option to make this change recursive (default) or only to the folder in question.

Looking at the timeline, the change was definately recursive!

Once I had that point in time, I tried to work backwards to find if there was an executable that was run, or a web page - anything that could explain how this was set accidentally.  Unfortunately, I did not find any smoking guns. I could see that after the properties were recursively set, the user logged off fairly quickly and the next set of events line up with when the help desk started installing tools.  It is possible that the user herself actually changed this attribute without realizing the effect this would have.

I went back to the help desk to see if they could "help" anymore.  They gave me a few more details (the user said she was working fine and then stuff disappeared, the help desk received the laptop the following morning).  They also said that they could no longer restore the user's files.

I never received a copy of the disk image, so could not help in restoring the files. 2 months later, I still haven't received the disk image from the company and the user had not asked for her machine or the files back!

So here's the recap of what the help desk did:
1. Installed a tool to "jiggle" the mouse so the machine wouldn't log out
2. Installed and ran malwareBytes
3. Changed registry properties (show hidden files)
4. Restored everything that malwareBytes had quarentined
5. Called infoSec

The actions taken by the help desk aren't bad - but the many, many conversations to get the information about what they did was painful.  They never mentioned the mouse jiggle tool that I saw in the timeline. And the idea of restoring quarantined files before they called infoSec is strange - I don't know what they thought that would do for us.

If the help desk had a standard process, or were instructed to take detailed notes - this would have made life much simpler.  It may not have allowed me to find the root cause, but it would have cut a few days out of the process where I was waiting for the help desk to give me more information.

In my ideal world, the help desk would take a memory image before starting to work on machines they think are infected with malware.  I know this would take time, and that help desks are busy, but in those instances where they can't solve the problem, that image could be invaluable.

Decoding Vawtrak\NeverQuest Traffic

This is not a post about the capabilities of this particular malware - this is about decoding the network traffic generated from an infected machine. StopMalvertising has a really nice write-up here: Analysis of vawtrak if you want to learn more about the malware.

Like all good malware, once a machine is infected, Vawtrak will send communications to it's C2 servers .  In this case, the traffic is encoded:

Observing the behavior of the malware on my system - I noticed that the malware only generated traffic when I had a browser open.  Looking at CloudStrike... I could see that the malware has injected itself into my browsers.

Looking at the malware in IDA and Olly provide no useful information, and I could not get the maware to execute within the debugger.

My next step was to use memory analysis to see if that provided any clues.  I took a memory dump of my infected machine after I had started IE and then started analysis using volatility.

Using the malfind plugin, I could see several injected processes, but the one I really wanted was in the IE and FireFox processes. 


Now that I had a memory address, I reverted back to a clean VM and tried again.  Using Olly (opened before I infected the machine), I attached to the IE process and put a breakpoint on what I thought was the infected memory section.

... insert lots of trial and error ...

What I was able to discover is that the traffic I saw was encoded using XOR with a random number generator with the seed included in the first 8 bytes of the payload:

Using python I was able to develop this decoder:

Decoding the above traffic yields:
C:\Users\dkilman\Dropbox>decode.py
id=68F33EA70000006700000000000F00000000&iv=00000022&av=00000000&info=020000020501010100030A28&proxy=none
!SCORE!

Link to the code: http://cybersecuritymave-techie.blogspot.com/2014/07/decodepy.html
Link to a pcap parser that will decode the vawtrak data: https://github.com/dominique97/vaw_decode

decode.py

## Author: Dominique Kilman
## Name: decode.py
## Copyright (c) 2014 Dominique Kilman. All rights reserved.

def rand(seed1):
    seed = (seed1 * 0x343fd) + 0x269ec3
    newSeed = seed & 0xffffffff
    ran = ((seed >> 16) & 0x7fff)
    modifier = ran & 0xff
    return [newSeed, modifier]

#test2 = encoded payload (after the first 8)
test2 = "\x9f \x0a \xc6 \xde \x21 \xcf \x53 \xea \x5a \x6a \xad \x38 \x70 \x30 \x8b \x39 \x60 \xe2 \x5f \x8e \x0a \x71 \xdb \x2d \x99 \xbb \xb3 \x03 \x38 \xf8 \xb8 \x59 \x9d \x74 \xba \x37 \x61 \x13 \xf9 \xd0 \xad \x3e \x1e \xfa \xee \x46 \x50 \x30 \x89 \x87 \x44 \xa5 \x7c \x8e \xa2 \x45 \x0c \xcc \x9f \x1d \x90 \x43 \x95 \x5a \x08 \x0c \xe6 \x5a \xa6 \xf8 \x66 \xdc \x2f \xb0 \xac \xcd \x5b \x58 \xe7 \xe1 \xe2 \xf3 \x0c \x6c \x39 \xec \x3b \xda \x89 \xdd \xde \x2e \xa0 \x18 \x5c \x5f \x15 \xe9 \xfa \x47 \xb1 \xd1 \x47 \x27"
seed = 0x2c4f5a543d6d6910 # first 8 of payload

newTest = test2.split(' ')
modchar = ''
resultLine = ''

for i in range(0,len(newTest)):
#for i in range(0,72):
    res = rand(seed)
    seed = res[0]
    modchar += str(int(res[1])) + ','
    try:
        test2 = ord(newTest[i]) ^ res[1]
    except:
        test2=0
    try:
        resultLine += str(unichr(int(test2)))
    except:
        test2=0

print resultLine

The BIOS Plot

So the first question is what is BIOS really?  Wikipedia will tell you that it's the Baisc Input/output System that all modern PCs use as their initial start-up tool.

Access to the BIOS setup is usually obtained by vendor-specific key press during the boot process (DEL, F2, F1, etc).  The key must be pressed prior to any OS starts up.  With the speed of modern computers now, it can be a challenge to figure out the key and get into the BIOS setups.  In the olden days, start-up took longer and you had time to read the boot up screens!

Here's some picures showing different BIOS setup screens on the machines in my house:

What you can do with the BIOS setup varies by manufacturer, but most often includes system settings including date time, the devices that can boot the machine (hard drive, CD/DVD, USB), as well as hardware specific settings and security options. 

The BIOS can be updated by a process called 'flashing' (no, not that kind of flashing!)

• There have been some historical attacks that impacted the BIOS:The first known BIOS attack released in 1998 was called the Chernobyl virus. Part of this virus payload attempted to flash the BIOS.  If it was successful, the machine would no longer boot.

• In 2011, a Chinese Security firm announced that they had discovered malicious software now known as Mebromi that targeted the Award BIOS as part of its infection vector.

• In 2013 while dealing with persistent malware infections in his environment, researcher Dragos Ruiu theorized that he was dealing with something he class BadBIOS - mawlare that was spreading to the BIOS of other machines via speakers and microphones (scary!)

Then the government gets in the act.  During a 60 minutes interview in 2013, the NSA broke the story of the BIOS Plot - evil nation states *almost* had the ability to control every computer on the planet - using BIOS vulnerabilities and mawlare.  Then there's the leaked details of the US government's own BIOS malware tool (DIETYBOUNCE).

With all the potting to take over my BIOS, what's a girl to do?  As far as I know, there are no AV scanners for the BIOS, and I don't know of any tools that would let me look at the BIOS on the fly without hosing my system accidentally.  The only almost workable answer I could think of was to take a hash of my BIOS and check it periodically to make sure it isn't changing.  

I have my plan, now I need a way to hash the BIOS... which means I need to be able to get a copy of the BIOS since I can't hash it while my machine is actively running.

Figuring out my BIOS version is pretty easy ... running msinfo32.exe on my windows machine brings up a handy view with all the information about my system, including the BIOS.  I could also get this from the BIOS setup screen.

Getting a copy of the BIOS proved to be more interesting.  Different vendors provide different tools to update the BIOS.
 
My ASUS machine has an EZ Flash tool that is part of the BIOS which lets me update the BIOS and create a copy of the current BIOS anywhere I want (score!).  My Lenovo machine let me download a tool to update my BIOS that included a program called WinPhlash which also has a handy copy the BIOS option (yay for us!).  

But once I got to my HP machines I hit a brick wall... the BIOS update too says that it saves a copy - but I couldn't find the file it supposedly saved. A little more research revealed that the copy option may rely on having the HP_TOOLS partition... unfortunately, my corporate build machine and the one that I installed TrueCrypt on no longer had this partition.  So a fail here, but 2 out of 3 isn't too bad.

One other option that looks like it could be pretty cool is something called Copernicus from Mitre.  The tool includes a script that will get a copy of the BIOS and then stores some statistics.  Sadly, this one didn't work on my HP machines either... sigh.

But, all in all - I learned way more about the BIOS that I though.   Some of the complexity in getting tools to work for securing (or scanning, or copying) the BIOS also provide it's own kind of mitigation to attacks -- getting an attack to work across all BIOS is not a trivial task.  To date, all the know attacks target very specific BIOS vendors and versions.

Creating a malware management system

During my time as a malware analyst, the methods for keeping track of the malware I have analyzed has always been problematic.  There's the excel approach (nice big spreadsheet of hashes and names, maybe some analysis details), the email approach (search through sent items to see what I did months or years ago), the word files approach (search folders and file shares for the write-up of that malware), and finally - build a database.

I started building the data base several years ago using MS Access.  Access is easy to set up (initially), portable, works almost anywhere.  This is what I used for several years across several different companies.  I stored basic analysis details, built a switchboard for searches, had a tabbed entry page for different details of analysis, etc. 

But as the information I wanted to store became more complex (many to many relationships in Access is painful!) the interface and the query design started to breakdown.

I started thinking that I needed to design a real database, but balked at designing a front-end that would be usable without months (or years!) of design and programming - after all, this was a project done on my own time or while I was between projects at work.

Then, a co-worker asked if I had ever used django.  Short answer was no, but I liked python and so looked into what django was and what it had to offer.  Almost like an answer to all my database design needs - semi-automatic database implementation, built-in web front end,  even the site title (The web framework for perfectionist with deadlines) seemed designed for exactly what I needed.

Now, we all know that nothing is that easy.  But after working through the tutorials, and putting some thought into what the database should look like.  I had a working model.  Then came the hard part - customizing exactly what I wanted and changing looks and feels to make me happy.

The default django admin worked pretty well for the basic data entry pages:

And then I decided to get a little fancy and add in some graphs to show statistics about the malware world I had analyzed.

I also added details to turn this into a forensics case management tool -- details about machines, disk images, and memory images and the case (or engagement) that ties them all together:

This is (and probably always will be!) a work in progress.  Now that I have the database semi-stable, I am working to add integration with automated tools like mastiff.py so I can pull the information from the mastiff database into my database instead of manually copying the data, plus reporting - easily pulling the information into something that I can turn into a nice report without too much editing.  Of course, the report will only be as good as the data I put into the summary and analysis parts!  But - with the django pieces in place - searching by any field is easy, showing statistical information is done, and best of all - it's an interface that some day I might be able to use across my team so we all put data into a central repository that can be used for quicker analysis in future projects.