It security revolves mostly around not letting security incidents happen. Unfortunately, this will not always be successful. How do I react to a IT security incident? There are some helpful guides from different organisations (e.g. NIST ) and they all share some core elements. The reaction to an incident is divided into phases which can be arranged as a cycle:
Sometimes it's obvious there is a security incident. At other times there are just hints to a possible problem. It might not be easy to decide if there is a security incident or just a technical glitch: A network service might have stopped because there was an unusual accident. The incident might also be a side effect of a compromise.
It's important to find out if a problem was caused by a fault or by a security incident. To do this systematically and not to forget to check things, check lists are used. The check lists are just a basis from which to proceed and do further testing. Examples for check lists can be found at DFN-CERT  .
During analysis the critical questions need to be answered:
- When did the attack occur?
- How did the attackers gain access?
- What exactly happened?
- Who are the attackers?
As soon as it is clear this is a security incident, other parties need to be informed. The own management needs to be informed. If the organisation is large enough, its press team and attorney might need to be briefed, too.
Following analysis, the compromised system should be disconnected from the network to make sure the attacker has no access to the system anymore. If this is a denial of service attack, one would try to easy the load on the affected systems. If passwords have been compromised, these would need to be changed or the corresponding accounts locked.
Now might be a good time to contact other involved parties. This might be the system administrators of the attacking site (the attacking system might just have been compromised) or maybe the police.
If the attackers had full access to the system, it should be reinstalled from original distribution media. It is possible to just erase the malware left behind by the attackers but there is always a risk that backdoors stay behind and the attackers easily gain access again. Also, this is the moment to fully patch the system and remove all vulnerabilities the attackers used to compromise the system.
After the incident the persons involved should consider and document the lessons learned. Which counter measures worked? What is missing (e.g. documentation) and what did not work at all? This is the only way to get better.
Preparation consists of doing the missing things identified earlier. This is the time to write down emergency plans, to test the tools used in an emergency and to organise a training session if necessary.
It is not always easy to react to a security incident: On the one hand, you need to be quick. On the other hand, unwise decisions mean that evidence gets lost and the incident can possibly not be resolved. For this reason, you should look into the topic before the incident happens: There will always be another security incident!