Detection as Code: Treating Your Rules Like Software

The folder of mystery rules

Every SOC I have seen from the inside has, at some point, run on a folder of mystery rules: detections written years ago by someone who has left, edited live in the SIEM console, with no history of what changed, why, or what the rule was even meant to catch. Nobody dares delete anything. Alert volume only ever goes up.

Detection as code is the unglamorous fix: treat detection content exactly the way software teams treat source code. It is now how I run detection day to day, and I would not go back. Here is what the discipline actually involves, beyond the slogan.

The pipeline

Rules live in Git, in a vendor-neutral format. Mine are Sigma. Every rule is a file; every change is a commit with an author and a reason. The SIEM console is a deployment target, not an editing surface. When an auditor asks "when did this detection change and who approved it", the answer is git log, not an archaeology project.

title: Suspicious OCI IAM Policy Grant to All Resources
status: production
logsource:
  product: oci
  service: audit
detection:
  selection:
    eventName: CreatePolicy
    statements|contains: 'allow group * to manage all-resources'
  condition: selection
tags:
  - attack.privilege_escalation
  - attack.t1098
level: high

Every change goes through review. A detection rule is a small program that runs against your most sensitive data and pages humans at night. It deserves the same scrutiny as production code. Review catches the things authors cannot see: a field name that does not exist in this log source, a condition that matches half the estate, an exclusion wide enough to drive an attacker through.

CI tests every rule before it ships. Three layers, in increasing order of value: syntax and schema validation (cheap, catches typos), true positive tests — replaying known-bad event samples the rule must match — and false positive tests, replaying known-good samples it must not. The test corpus matters more than the framework: every incident and every noisy-alert complaint becomes a new sample, so the suite compounds.

Every rule is mapped to MITRE ATT&CK. Not for compliance decoration — for navigation. The mapping turns "do we have good coverage?" from a feeling into a heat map, and it is how threat hunts get targeted: hunt where coverage is thin, and every gap a hunt confirms becomes a new rule with a test attached.

What changed, measurably

The benefits showed up quickly and in roughly this order:

1. Regressions stopped. Rule edits that would have silently broken a detection now fail CI instead. The class of incident where "the detection existed but someone had tuned it into uselessness" simply ended.
2. Tuning got honest. When suppressing a false positive requires a commit with a justification, exclusions stay narrow and reviewable. The quiet, console-edited exclusion that eats a real alert is the most dangerous artefact in a SOC.
3. Coverage conversations became evidence-based. The ATT&CK map gives leadership a truthful picture — including the uncomfortable parts — and turns budget requests into "this is the gap" rather than "trust me".

What it costs, honestly

The discipline is not free. Building the test corpus is real work — true positive samples have to come from somewhere, which in practice means a lab that can generate attack telemetry and the habit of saving event samples from every incident. Review adds latency to urgent rule pushes, so you need an expedited path for active incidents (ship with one reviewer, backfill the tests within the week). And the first migration — dragging hundreds of legacy console rules into Git, deduplicating and deleting as you go — is a slog with no shortcut.

Worth it anyway. Of everything I have adopted in detection work, this is the practice with the highest ratio of impact to novelty: no machine learning, no new product, just software engineering applied to the place where defence actually happens.

If you are starting now: pick one log source, migrate its rules, wire up the three CI layers, and expand outward. Do not boil the SIEM.