Lessons from the Incident Room

The view from inside

Before I moved into detection engineering, I spent years in consulting incident response — ransomware, business email compromise, insider threat, and cloud intrusion, for organisations from family businesses to global enterprises. No client names and no war stories here; the details belong to the clients. But the patterns repeat so reliably that they are worth writing down. These are the lessons that survived every engagement.

Preparation beats heroics, every time

The single strongest predictor of how an incident ends is not the sophistication of the attacker. It is the state of the victim's logging, backups, and identity hygiene before day zero.

The recoveries that went well had boring things in place: offline or immutable backups that were actually tested, centralised logs that predated the intrusion, an asset list that roughly matched reality, and someone who knew where the crown jewels were. The recoveries that went badly were archaeology — reconstructing what existed before deciding what was lost. Forensics can be brilliant; it cannot conjure logs that were never collected.

If you do one thing after reading this: check, today, that your backups restore and your authentication logs are retained somewhere the attacker cannot reach. That hour is worth more than any tabletop exercise.

BEC is a process failure wearing a technical costume

Business email compromise produced some of the largest losses I worked on, and almost none of it was technically sophisticated. The pattern is depressingly stable: a phished mailbox, a quiet inbox rule, weeks of patient reading, then one perfectly timed payment-redirect email in a real thread.

The fix is mostly not a product. Out-of-band verification for bank-detail changes — a phone call to a known number — defeats the entire playbook. The organisations that had that one rule, enforced even when the request looked like it came from the CEO, did not become cases.

Evidence discipline is a kindness to your future self

In the first hours of an incident there is enormous pressure to act — wipe the machine, rotate everything, get the business back up. Sometimes that is right. But every action taken without preserving evidence first is a question you may never be able to answer: how they got in, what they took, whether they are still there.

Chain of custody sounds like bureaucracy until the insurer, the regulator, or the litigation arrives — typically months later, when memory has faded and the rebuilt servers have overwritten everything. Image first, act second, write everything down with timestamps. Your future self, sitting in a dispute about what was exfiltrated, will be grateful.

Executives need a different truth — not a smaller one

The hardest skill in incident response is not technical. It is standing in front of a leadership team mid-crisis and being clear without being falsely certain.

What works: lead with the decision they need to make, state what you know, what you suspect, and what you have ruled out — explicitly labelled as such — and never let optimism outrun evidence. A briefing that says "we believe access began on the 14th, we have not yet ruled out the finance system, you will have an answer on that by Thursday" builds more trust than confident guesses that get revised tomorrow. Credibility, once spent, does not come back during the same incident.

The incident is the cheapest security budget you will ever get

Post-incident, every organisation has a window — typically a few months — when security has the board's attention and remediation money is available. The ones that converted that window into identity hardening, logging coverage, and tested recovery came out stronger than they went in. The ones that patched the single entry point and moved on tended to reappear.

A serious incident is a terrible thing to experience and an excellent audit. It tells you, with complete honesty, what your controls actually do under pressure. The least you can do is take notes — which, in the end, is what this post is.