Backdoors
An attacker gained access to one of my systems in the course of a successfull attack. He will want to gain access again at a later time without having to exploit the same vulnerability again. For example if he used an SSH attack, the password might habe been changed in the meantime. This is why an attacker will likely install a so called backdoor.
What is a backdoor?
A backdoor provides an alternative mechanism for remote access to a compromised system. A backdoor can be a new account with a password of the attackers choosing. Though, an administrator will easily detect a new account. There are more refined backdoors which are not as easy to spot and close.
Backdoors can be implemented in a variety of ways. Often, new network services are installed like an additional SSH server on a high port. Or an existing service is manipulated - e.g. by exchanging the binary. Sometimes a tiny configuration change can provide the attacker with reliable access to the system. Thus, providing a backdoor. A backdoor can also be implemented within the files and directories of common users. An additional entry in the authorized_keys file or manipulation of a user owned script can be enough. A list of examples can be found in [1] and [2].
What is the problem?
In the course of the reaction to a security incident the compromised system needs to be cleaned up after analysis. There are different ways to do this. One way is deleting the malware found and reverting all changes done by the attackers. The other way is rebuilding the system completely from scratch. The latter is more time consuming but is still the generally recommended way. Lack of time and limited resources most often lead to the former: deleting all traces found and hope for the best.
The problem is, you never know if there was a backdoor. Maybe you just didn't find it yet! Clever attackers leave several backdoors behind, so they don't lose access when a single one is found. If the vulnerability that lead to the incident was openly accessible for a longer period of time, chances are good there were several groups of attackers on your system. All of these could have left backdoors behind; independantly of each other. The root of the problem is, that you cannot prove there is no backdooor. You can only prove there is a backdoor by finding it.
Pragmatic approach
The common guides (e.g. [3] and [4]) suggest to reinstall the system from original media. In the field, this is seldomly done as one might guess. Often, one of these reasons are put forward:- The system needs to be up and running all the time.
- Our administrators do not have the time necessary to reinstall.
- The configuration is not really documented and a reinstall would be extremely time consuming and risky.
- Limit the attack surface: The attackers will try to access the backdoor from the internet or from another compromied system on the local network. Ideally, only the absolutely necessary network services are provided. Outgoing connections should also be limited to what is known and necessary since a backdoor might work by periodically initiating an outgoing connection.
- Log and analyse: A backdoor can be well hidden but the attacker accessing the backdoor should get noticed. Local log data should be synchronised to central log servers in order to protect it from manipulation. In addition, netflow data and firewall logs can be a valueable source of information. Analysis and alerting should be automated to enable a timely reaction to threats.
- Be prepared to reinstall: Unfortunaltely, most often you can prove that the system was compromised but not the opposite. Therefore, we must live with a remaining risk. If a certain level of uncertainity has been reached (e.g. suspicion has risen after several unexplained incidents) a reinstall should be planed. Ideally, this should not be a big effort due to standard installation images even for servers. This would also be a good time to create the missing documentation.
Backdoors can have very different forms and are sometimes extremely hard to find. Seldomly, one can say for sure, that there has been no compromise. Even a reinstall is not always a save bet since a backup might have been infected or there has been a backdoor in user data. But since the attackers normally want to do things with our systems, the situation is not that grave. Have close control over your networks and analyse your log data and you will find the attackers.
[1] Andreas Bunten: Kompromiss nach Kompromittierung?, <kes> Die Zeitschrift für Informations-Sicherheit, Ausgabe 4, August 2011
[2] Andreas Bunten: 99 Backdoors on my UNIX Host, Tagungsband des Frühjahrsfachgesprächs der German UNIX User Group, März 2012 (lokale Kopie)
[3] First Responders Guide to Computer Forensics, Carnegie Mellon Software Engineering Institute, CERT Training and Education, März 2005
[4] Karen Scarfone, Tim Grace, Kelly Masone: Computer Security Incident Handling Guide, National Institute of Standards and Technology, Special Publication 800-61 Revision 1, März 2008