Tracking the hackers

by Simon Edwards

You've been hacked, so now what? Scream "fire!", call the FBI or pretend it never happened?

You've installed the latest firewall, patched every workstation with the new security updates and located every unauthorised wireless LAN in the building - but you've still been hacked. Do you call the police, fire your systems administrator, reinstall and pretend nothing happened? Or take down your Web and e-mail servers (and, therefore, business) for a prolonged period of examination? What does your emergency response plan say? You've got one, right?

This article is about tracking down the person or persons who have successfully attacked one or more of your computer systems. We will not be explaining how to secure your Web servers but rather how to prepare for the eventuality that they fall under someone else's control.

You need to be prepared for an attack so that when someone breaks into your essential systems you can respond as quickly and rationally as possible. Panicking can result in lost forensic evidence or, even worse, lost business. You can't leave your compromised Web host visible (and risible) on the Internet but you shouldn't blindly restore a backup and assume that the hacker won't repeat his actions either. There's been a problem and your job is to fix it as fast as possible and ensure it never happens again. After that you can choose whether or not to track down the perpetrator. But before you kick off a major police investigation there are some serious issues to consider.

Peter Sommer, Visiting Research Fellow at the LSE Computer Research Centre and legal expert witness in computer forensics, believes that a lot of thought should go into whether or not the news of your security breach should be allowed to reach the law and, therefore, the public. "It depends on your organisation and how inconvenienced you are prepared to be. Your internal procedures will be held up to scrutiny and disclosure puts off many companies." More than anything business must continue as usual, and any steps taken to gather forensic evidence must be as sound as possible while minimising the time that systems are offline. "Preserving the evidence is important but the company must continue. Calling in a specialist will help."

For the purposes of this article we will assume that you want to track down the attacker and bring a legal case against him or her.

Getting ready

It doesn't matter how many books and magazine articles are published advocating safe computing practices, how many free tools are made available for download and how many companies offer security services and advice. Computer systems are hacked all the time.

The CERT Coordination Center, an organisation dedicated to working with the Internet community in handling security incidents, dealt with over 52,000 such incidents in 2001 alone and received nearly 119,000 e-mail messages and 1,417 hotline phone calls from people reporting incidents or seeking advice during this same period. This is a 160 per cent increase in hacker activity over the previous year. New security holes emerge all the time, and CERT predicts that a recently announced vulnerability in the Simple Network Management Protocol (SNMP) will cause widespread problems with routers being taken down or even controlled by attackers.

Clearly the most sensible approach is not one of complacency, and preparations must be made for the nasty eventuality that an attacker succeeds. CERT has published a list of steps designed to help administrators recover from a system compromise and you'd be advised to read these now before you need them. The first tasks you'll need to perform are not to pull the plugs and start analysing the log files - "We suggest you contact law enforcement before attempting to set a trap or tracing an intruder." Instead, the person in charge of the systems should consult existing security policies or consult with the management, seek legal advice and only then make contact with a law enforcement agency. Only at this point should you consider a forensic investigation. You can find the document here.

If you want to involve law enforcement, and you are operating in the UK, you should call your local police force, which will use its own Computer Crime Unit to investigate. If the crime appears to be extremely serious they may escalate the case to the National High Tech Crime Unit. And if you'd rather have someone else do the forensics, rather than your own team, companies such as Vogon , EnCase and Computer Forensics are available to help.

The investigation

The aim of most forensic investigations is to establish a modus operandi and the identity of the attacker. The MO is important because if you don't know how the attack was made there's no way you can prevent it happening again. Tracking the hacker's methodologies can help strengthen your defences, and may lead you to discover who was behind the hack in the first place. There are millions of ways to attack a computer system, but if you discover two of your systems have been abused in identical (and unusual) ways you can be fairly sure that the same person was involved in both cases.

First steps - disk imaging

Before you start trawling through files looking for clues begin by creating an exact copy of the compromised system's hard disk. According to Peter Sommer, "most scene of crime work is extremely imperfect," whether it be at a murder site or a Web site. Don't boot the computer up for the examination, as you'll be doing the digital equivalent of running a herd of cows through the crime scene. When a computer starts up it makes, deletes and changes files which may cause essential clues to disappear. You will damage a prosecution's case if the defence can argue that evidence has been tainted through mishandling. Instead, remove the hard disk and mount it as a read-only volume on another system. Programs such as Symantec's Ghost and Unix's dd can make an exact bit-for-bit copy of the disk rather than just grabbing a copy of all the files. This means that even empty space is replicated, which is important because empty disk space is rarely truly empty (see Examining deleted data below).

Once you have a disk image of the suspect drive store the original somewhere secure, such as in the IT manager's safe. Always document who has had access to any evidence as an uncertain chain of custody can be called into question should a court case arise. Create at least one more copy of your copy, either to another drive or to tape. This is your working 'original', from which you can take further copies should something go wrong without disturbing the master copy.

The first disk image you created can now be examined by installing the drive as a slave on another PC system. The PC need not be running Windows as FAT32 and NTFS file systems can be mounted directly from Linux systems. One very good reason to use another machine to analyse the compromised disk is that programs on hacked system may have been altered to help hide the hacker's log entries and files. Run software such as Tripwire to generate signatures for important files and store these elsewhere. Then, when you suspect a hack has occurred, compare your record against the disk. You may find that simple programs such as Dir or Netstat have been replaced with customised versions that hide certain files or network connections.

Disk images can also be mounted by VMWare, an incredibly useful virtual machine that can undo all changes made during a session on the suspect's Windows, Linux or BSD installation. If you are investigating a Unix server but prefer to run Windows on your main desktop PC, VMWare will allow you to create your report in a familiar environment. If you mount a disk in Nonpersistent mode all changes you make to the data, intentionally or not, are lost when powering off the virtual machine, which helps to keep your working copy of the disk clean.

Logs are your friend

Unix and Windows servers can log almost any activity that they are requested to do. If the server was set up to create logs for important actions you should have plenty of evidence at your fingertips, including when an intruder logged on, his IP address (and, therefore, where he logged in from) and possibly errors that can indicate the sort of attacks he's tried. If you see something like the following garbage in the 'messages' log file you can be fairly sure someone has tried a buffer overflow attack on your system:

Apr 11 17:10:24 rogue bsd-gw[1212]:
[ID 315218 lpr.error] Invalid
protocol request (66):
BBBìóÿ¿íóÿ¿îóÿ¿ïóÿ¿XXXXXXXXXXXXXXXXXX%.172u%300$ns
ecur%301$nsecurity%302$n%.192u%303$n\220\220\220\2
20\220\220\220\220\220\220\220\220\220\220\220\220
\220\220\220\220\220\220\220\220\220\220\220\220\2
20\220\2201Û1É1À°FÍ\200\211å1Ò²f\211Ð1É\211ËC\211]
øC\211]ôK\211Mü\215MôÍ\2001É\211EôCf\211]ìfÇEî^O'\
211Mð\215Eì\211EøÆEü^P\211Ð\215MôÍ\200\211ÐCCÍ\200
\211ÐCÍ\200\211Ã1É²?\211ÐÍ\200\211ÐAÍ\200ë^X^\211u
^H1À\210F^G\211E^L°^K\211ó\215M^H\215U^LÍ\200èãÿÿÿ
/bin/sh

However, you may find that you have no log files, or none that supply any useful information. There are two likely reasons for this, the first being that the systems administrator has not tuned the logging to provide critical information. Or has not turned it on at all, in the case of many Windows servers. Also, any hacker worth his salt will have disabled or deleted logging and log files.

There is a good case to be made for having log files saved to a computer other than the Web or mail server in question. The Unix Syslog daemon is quite capable of sending logs to a Syslog server, and Windows 2000 is also able to log externally with the help of extra tools, such as NTsyslog. While it is possible to recover deleted logs, the quicker and easier your analysis of the hack, the better it is for business.

You can log network traffic using an Intrusion Detection System (IDS) such as ISS' RealSecure Network Sensor or a free application such as Snort. These products watch the network packets as they pass between hosts and can record everything. Popular attacks are usually recognised and flagged so a systems administrator can look back and see what has really happened. There is always a chance that the hacker may be caught in the act and prevented from doing any real damage. Other popular IDS' include Network Flight Recorder (NFR) and Big Brother.

Finding the perp

Once you've got lots of data, hopefully including time of logins, usernames used, locations where they logged in from and so on you can begin to dig for information. On Windows PCs you can use tools such as Sam Spade to start running simple Whois queries. If you're lucky you'll find your hacker has been stupid enough to launch his attacks from a home ADSL connection that features a static IP address range. This isn't likely, though, and commonly you'll find the attack came from a dial-up account at an ISP or from another hacked machine.

Whois servers can provide enough information to find out who to contact for help, and can even identify the attacker by name in some circumstances.

In the first case you could approach the Internet Service Provider used in the attack and ask for cooperation. If they have kept their log files and are willing to help you'll be able to discover which subscribers look like being the attacker. The attitude of some ISPs can be less than helpful, but if legal action is taking place they will respond more usefully than when some unknown Webmaster makes a vague complaint.

Bear in mind that just because an attack comes from Fred Bloggs's PC doesn't mean that Mr Bloggs knows anything about it. Hackers usually use a chain of compromised systems to hide their tracks, and if Fred's Windows PC is misconfigured to act as an anonymous proxy, and he has no logging turned on, the trail will become decidedly chilly. Similarly, dial-up accounts can be stolen or hijacked. Your international chase for justice could end at the doorstep of a victim as innocent as yourself.

On balance

So when is it worth tracking the hacker? Dan Cuthbert, Senior security consultant at IDSec, says, "it all depends on the nature of the attack - how seriously the company intends to follow through actions against the person or group who initiated it. Something as serious as stolen money or fraud should be reported to the National High Tech Crime Unit, which is run by a brilliant bunch. They are very clued up and eager to help."

But Web defacements and other relatively minor, if potentially embarrassing, scenarios are not worth following up. "Not only don't you want to lower your public image," he says, "you're admitting your security setup is weak. This can open the floodgates for others to have a pop at hacking you." Peter Sommer agrees, but has seen some cases where small-time hacking has made it to the courts. "There was a case of a former employee who placed a badly hidden backdoor on a development system. He accessed the system and left a note criticising the site's security. It crashed the next morning and the company prosecuted. It was a misconceived prosecution and he was found not guilty in 45 minutes."

The lesson here is don't let your ego, or the ego of your IT staff, blow things out of proportion. If you lose business and money you have a good reason to chase the attacker, but if it's just a case of losing face you'd be better turning your energies to fixing the weaknesses in the systems.

Practice practice practice

If you are going to do your own forensic examinations, or want to put together a dedicated team to do so, you'll need to get some practice in. You don't want to realise your limitations, or the limitations of your toolkit, when faced with a 48 hour deadline. There are a number of good books available to introduce technical users to the world of computer forensics, including the excellent Computer Forensics: Incident Response Essentials by Kruse and Heiser and Mike Schiffman's Hacker Challenge, which contains descriptions and clues of digital break-ins. The reader attempts to solve the mystery and can read in-depth descriptions of the solution afterwards.

For an even more hands-on approach, the increasingly well-known Honeynet Project provides disk images that may be downloaded, mounted on your forensic analysis computer and examined. If you want to enter the competition and submit your report you might even get credit on the site. For example, in May 2001 the Honeynet Project challenged beginners to recover a deleted root kit from a disk image. Even just reading the reports submitted by other users can be very educational and interesting.

If you have sufficient resources, and regularly test security measures in your own laboratory, you can kill two birds with one stone and run penetration tests against lab machines, followed by a forensic analysis. IT managers could find it helpful to pit teams against each other, where one attempts to hack a server and another tries to clear up the mess afterwards. Ensure that none of this activity leaks on the production network, however, or you could cause problems for your users (and almost certainly fall foul of any security policies already in place).

Emergency response teams

While it may be tempting to train your IT staff to become cybercops, bear in mind that their everyday job of replacing printer toner, backup tapes and configuring servers is what they are paid for and anything else they learn is a bonus. Can you afford to pay salaries to half a dozen fully-trained ethical hackers? And will they have time to fulfil their normal roles if they devote sufficient time to learning about IDS and disk analysis properly?

If you don't have the budget you may need to rely on outside help. We've already mentioned CERT, but if you want someone to actually take over, look at your server and do the analysis you'll need to pay for a consultancy like Computer Forensics of DIBS fame, which will do everything up to and including giving evidence in court.

Rugby-based Computer Forensics is, in fact, one of a very few UK companies that specialises in post digital crime examinations. It supplies two main hardware products, the DIBS Portable Evidence Recovery Unit for copying disks and the DIBS Forensic Workstation for analysing the data. The overall cost of kitting up a forensics team is usually around £10-15,000, which is why banks and government agencies are their usual customers.

For those departments that can afford to pay two or three people to specialise in forensics, and have them work around an on-call shift system, there are training courses and books, as well as the practice methods mentioned above, that can be used to bring the skill levels up to scratch. Computer Forensics runs courses, including a general technical course for £888 (£1,043), which do not revolve around the company's DIBS product line. It also claims to provide a high level of technical support to customers of its hardware and software products.

To gain a wider perspective on building up a team grab a copy of O'Reilly's Incident Response by R.van Wyk and Forno.

Keep it legal

According to research commissioned by Oracle most security breaches are caused by internal employees, while Internet Security Systems claims that 19 per cent of digital crime and data loss is due to insiders, compared to the one per cent of external attacks. Life would be so much easier if you could keep a fully open eye on all the systems and network traffic under your control. Every action your users take could be logged and used as evidence should someone commit a misdemeanour.

RIPA vs the Human Rights Act

But while the Regulation of Investigatory Powers Act 2000 appears to give employers carte blanche when it comes to watching the activities of its workers, allowing a more-or-less free reign over monitoring e-mail and phone calls, the Human Rights Act of 1998 grants everyone "the right to respect for his private and family life, his home and his correspondence."

IT manager should discuss the finer points of this obviously incompatible legislation with the company lawyers before deploying key-stroke logging software and phone taps. And while you may be given complete autonomy over your network, you should also be made utterly aware of where the network's boundaries lie. You should not, for example, start routinely bugging contractors' laptops.

The county line

Jurisdictional boundaries don't just exist within companies - even the police can't go strolling across to Bulgaria to arrest the hacker who attacked your mail server. Learning where your hacker attacked from should be part of the intelligence you try to gather before calculating whether or not pursuing him is worthwhile. While it may be possible to extradite someone from France, your chances to getting to grips with a hacker in China is clearly more limited.

In June 2000 Attorney General Janet Reno spoke at the Information Technology Association of America (ITAA) Cybercrime Summit saying, "although borders are meaningless with respect to cybercrime, we have got to effect alliances around the world that will ensure that there are no rogue nations, no rogue jurisdictions, that permit cyber attacks around the world." The relatively new Convention on Cybercrime now features signatories from the UK, the USA, Bulgaria, France, Germany, Romania, Japan and South Africa as well as many others.

This Convention is aimed at promoting the sort of legislation necessary to arrest computer criminals and to aid international cooperation. You can find out more about this subject at the Council of Europe's Web site here.

Examining deleted data

An experienced forensics examiner will look at both the so-called empty disk space, where deleted files or sections of deleted files can reside, as well as slack space. Slack space is the unused disk space in a disk cluster. Clusters on FAT32 partitions are 4k in size, so a 6k file will create 2k of slack space. If this area of the disk once contained part of an important file it may be possible to recover some data. This might include clues to help crack encrypted files on the disk, or to incriminate the system's user. Most importantly, for when trying to track a hacker, they may include deleted log files.

Deleted data can be recovered even if it has been partially overwritten.

There are a number of tools professionals use to examine and recover deleted files. Those who regularly examine Unix-based platforms will be acquainted with The Coroner's Toolkit, which can recover deleted information and even run on a live host, gathering data from volatile memory. Clearly there are potential tainted evidence issues when running software on the actual compromised machine, but in some cases this is necessary. ForensiX is a Linux-based suite of programs that support a huge number of file systems and can quickly index a disk image for fast text searches. EnCase is a very popular Windows forensics package that allows you to build case files, which makes creating your final reports relatively simple.

Public relations vs retaliation

Many companies believe that the value of tracking their hackers rapidly decreases when balanced against the inevitable bad publicity that comes with admitting that the computer systems were vulnerable in the first place. While it is only realistic to believe that no computer is truly secure (unless it's locked in a safe, and switched off), the public and the national media doesn't accept this fact and panics every time users' credit card details are stolen from databases, or customers' billing details appear on the Web for all to see. That's not to say that companies shouldn't try their utmost to protect these details, but a successful hack is never impossible.

If you call the police you are simultaneously admitting that your computer or recruitment system was flawed but, some argue, you are also showing that you are not ignoring the hack and are committed to improvement and prosecution. Just make sure that the hack you are highlighting is worth bringing to the public's attention. Is an internal breach, where a disgruntled employee momentarily defaces your Web site as important as an external genius managing to clean out your accounts? The line between responsible reporting and a PR disaster is usually finer than that obvious example. If the employee was using company servers to host child pornography should you report? Ensure that you know where that line is, beforehand, so that management's judgement doesn't become clouded when nasty things hit the fan.

The ultimately responsible attitude is to report any break in that appears to come from an innocent party's computer or network. That way, while you may not wish to press charges, you have at least helped prevent further attacks on yourself and others via this particular source.

First Published in PC Pro, August 2002.

The above article is © Dennis Publishing Limited 2002. UK property of Dennis Publishing Ltd. This article may not be reproduced or transmitted in any form in whole or in part without the written consent of the publishers.

Special Report: