How to Test Whether Your SIEM Detects Real Attacks

KEY TAKEAWAYS

01A rule that has never fired against a real behavior is a hypothesis, not a detection.02Most SIEMs already collect the data. The pipeline between log and alert is what breaks.03Send a safe, tagged behavior through and timestamp all six hops. Where the timestamps stop is the gap.04Send each fix to the owner of the failing hop, then rerun the same behavior.05Report alert rate on validated behaviors, not rule counts.

1. Logging is not detection

A SIEM collects logs. Detection is what turns one of those logs into an alert that a person acts on in time. Most SIEM gaps sit between the two: the data is there, but no useful alert comes out the other end.

A rule that has never fired against a real attack behavior is a hypothesis, not a detection. Counting rules tells you how many hypotheses you have. Validation tells you which ones hold.

ATT&CK TECHNIQUES DETECTED 21% Average across production SIEM deployments.

RULES THAT NEVER FIRE 13% Broken by misconfigured data sources and missing fields.

TELEMETRY ALREADY COLLECTED >90% Of techniques are detectable from data already ingested.

Figures from Hayrok Labs research note HL-RN-2026-05 · source citation to confirm before publishing

Why detection breaks silently

Parsers break A vendor changes a log format upstream, and a field the rule depends on goes empty.

Rules time out Expensive queries are skipped or throttled under load, so the rule exists but doesn't run on time.

Log sources go quiet An agent stops forwarding, a cloud audit trail is disabled or a retention change drops a table.

Tuning overshoots An exclusion added to silence false positives also silences the true positive.

Alerts route nowhere The alert fires but lands in a queue nobody watches, or is merged into an unrelated incident.

COMMON MISCONCEPTION

"We ingest everything, so we'll see it." Collection is usually the healthiest part of the pipeline. The breaks happen later, in parsing, scheduling, tuning and routing, and none of them raise an error.

2. The six hops between an attack and an alert

Every SIEM detection travels the same road. Select a hop to see how it fails and how to check it.

Rule execution OWNER · Detection engineering

The scheduled or streaming rule actually runs over the window that contains the event.

HOW IT FAILSExpensive queries are skipped, throttled or time out under load, or the schedule misses the window.

HOW TO CHECKCheck the rule's run history and health for the test window.

Rule counts measure hop 5 alone. Validation tests all six, so the fix goes to the team that owns the hop that actually broke.

3. The method in six steps

Trace real behaviors, not rules. Steps 04 and 06 are where most teams cut corners.

Pick behaviors, not rules

Start from priority techniques mapped to ATT&CK and choose safe behaviors that exercise each one.

OUTPUT · A behavior list per technique.

Make each run attributable

Tag every run with a unique marker so its events can be found in every system.

OUTPUT · A run ID per test.

Execute from realistic positions

Run behaviors on the endpoints, cloud accounts and identities an attacker would use, through an isolated runner, with approval.

OUTPUT · Executed behaviors with start times.

Timestamp each hop

Record when the behavior ran, when telemetry appeared at source, when it landed in the SIEM, when the rule matched and when the alert reached a queue.

OUTPUT · A hop record per run.

Classify the outcome

Give each run one outcome. Each outcome points to a different owner.

OUTPUT · Outcome and owner per run.

Fix the hop, then rerun

Send the fix to the owner of the failing hop and rerun the same behavior. Close the gap only when the alert arrives in time.

OUTPUT · A verified fix.

4. Worked example: encoded PowerShell in Sentinel

This traces encoded PowerShell (T1059.001) through a Microsoft Sentinel pipeline. The same approach works in Splunk, Elastic or Google SecOps.

Step 1: the detection under test

The rule is written in Sigma so it's portable across SIEMs.

Sigma

title: Encoded PowerShell command line
id: 6b1a3f2e-9c4d-4e1a-8f2b-3d5e7a9c1b20
status: test
logsource:
  category: process_creation
  product: windows
detection:
  selection:
    Image|endswith: '\powershell.exe'
    CommandLine|contains:
      - ' -enc '
      - ' -EncodedCommand '
  condition: selection
level: high
tags:
  - attack.execution
  - attack.t1059.001

Step 2: a benign, tagged behavior

The test does nothing except write a unique run marker. Encode it and run it on an in-scope test server, with approval.

PowerShell

$marker = 'hayrok-run-7f3a91'
$cmd = "Write-Output '$marker'"
$b64 = [Convert]::ToBase64String([Text.Encoding]::Unicode.GetBytes($cmd))
powershell.exe -NoProfile -EncodedCommand $b64

Step 3: find the marker at every hop

Encoding hides the marker on the command line, so search for the encoded value, measure ingestion delay and check whether the rule fired.

KQL · Microsoft Sentinel

let b64 = "<encoded value from step 2>";
SecurityEvent
| where EventID == 4688 and CommandLine has b64
| extend ingested = ingestion_time()
| project Computer, TimeGenerated, ingested,
          ingest_delay_s = datetime_diff('second', ingested, TimeGenerated)
| join kind=leftouter (
    SecurityAlert
    | where AlertName == "Encoded PowerShell command line"
    | project AlertTime = TimeGenerated, CompromisedEntity
  ) on $left.Computer == $right.CompromisedEntity

Reading the result

No 4688 event Hop 1 failed. Process creation auditing or command-line logging is off.

Large ingest_delay_s Points at collection.

Event, but no matching alert Points at parsing, rule execution or rule logic. Check them in that order.

5. Try it: find the broken hop

Four hop records from a sample environment. Read each one and pick the hop where detection broke.

hayrok-run-7f3a91 T1059.001 · Encoded PowerShell

Source telemetry 09:02:11Z —

Collection 09:03:40Z · delay 89 s —

Parsing CommandLine populated —

Rule execution skipped: query timeout under load —

Rule logic no match evaluated —

Alert routing null —

Which hop broke?

Illustrative hop records · synthetic environment

6. Classifying the outcome

Give every run exactly one outcome. Each one points to a different owner and a different next step.

Alerted in time An actionable alert reached a watched queue within the expected window. NEXT · Keep the behavior in the regression set.

Alerted late The alert arrived, but outside the window your SOC needs. NEXT · Find the slow hop from the timestamps.

Logged only Telemetry reached the SIEM, but no alert fired. NEXT · Check parsing, rule execution and rule logic.

Partially logged Some events or fields arrived, but not enough to detect. NEXT · Fix the missing source or field mapping.

Not logged Nothing usable reached the SIEM. NEXT · Fix source telemetry or collection first.

What to record for each run

The technique ID, the run marker, the host and account, the timestamp at each hop, the outcome, the rule that responded and the owner of the failing hop. Close a gap only when a rerun of the same behavior alerts in time.

7. Measuring the outcome

Five numbers worth putting in front of your SOC lead. Each one only improves when detection actually improves.

TRACK THIS WHY IT MATTERS

Alert rate on validated behaviors The share of behaviors that produced an actionable alert. This is the headline number.

Logged but not alerted Behaviors that left evidence in the SIEM with no alert. Usually the most fixable gap.

Median alert latency Seconds from behavior to alert in a queue, per log source.

Log source health The share of required sources that delivered telemetry on the last run.

Regression rate Detections that passed last run and failed this run.

Rerun the same behaviors after every parser, rule or log source change, and on a regular cadence for your priority techniques. A detection that passed last run and fails this run is a regression and should be handled like an incident.

8. Check your understanding

Three quick questions. Pick an answer to see the explanation.

Q1Your test event is in the SIEM with every field populated, but no alert fired. Which hops are still suspects?

Q2Why tag every validation run with a unique marker?

Q3A rule alerted last week. This week the same behavior is logged only. What is it?

9. Glossary

SIEM Security information and event management: the system that collects logs and runs detection rules over them.

Hop One stage of the pipeline between an attacker's action and an analyst's queue.

Run marker A unique tag written by a validation run so its events can be found in every system.

Hop record The timestamps and status of one run at each hop, plus its outcome and owner.

Ingestion delay The time between an event happening and it becoming searchable in the SIEM.

Sigma An open, SIEM-agnostic format for writing detection rules.

Logged only An outcome where telemetry arrived but no alert fired.

Regression A detection that passed on a previous run and fails now.

How to test whether your SIEM detects real attacks