Alert deduplication and correlation

# Alert deduplication and correlation

Separate repeated deliveries from unique alerts, then group related evidence with a visible time-window rule without equating alerts with incidents.

This is an original, fictional, offline exercise for an introductory security learner. It is independent practice, not copied coursework or a claim of professional experience. Completion and competence remain unassessed.

## Before you start

Allow **30-45 minutes**: 5 minutes reading, 10 inspecting and predicting, 10 checking and explaining, and 5-20 for a changed or independent case. You need Python 3.10+ (standard library only), a terminal and a text editor. Comfort opening files, reading a small table and basic evidence notes is enough; programming, administrator rights, a Linux installation and cloud accounts are unnecessary. Previous introductory network or Wireshark practice is useful but not required.

> **Vocabulary:** Delivery: one received record. Deduplication: removing repeat delivery of the same ID and content. Correlation: bringing related evidence together. Anchor window: a period measured from the first event rather than the most recent event. Incident: a conclusion requiring investigation beyond alert counts.

## Evidence and exact rules

The case JSON is the single authoritative input. Fixtures are guided.json, changed.json and challenge.json. All data is synthetic; artifacts is empty to avoid competing editable copies. Edit only a prepared run copy. The analyser is a pure calculation and never executes evidence text.

data.window_seconds is an integer from 1-3600. Each alert has id, fictional lowercase host.test, simple account name, UTC time to seconds and kind auth-failure/auth-success/process-alert. Identical IDs with identical content represent repeat delivery. Conflicting content under one ID is rejected. Different IDs remain distinct.

Deduplicate before counting. Partition by exact host/account, sort by timestamp then ID, and start a session at the first alert. Include alerts up to window_seconds after that anchor, inclusively; a later alert begins a new session. The window does not slide. Keys are host/account/session-number. Three strictly earlier failures before a success in one session give review-auth-sequence. Otherwise process-alert plus either authentication kind gives review-mixed-signals. Remaining groups give retain-context. These are exercise triage rules, not confirmed incident counts.

## Run the exercise

Open a terminal in this lab folder. Use your Python command (`python`, or `py -3` if that is how Python is installed). Commands below use `python` consistently.

```text
python lab.py cases
python lab.py prepare --scenario guided --out run/attempt-01
python lab.py show --case run/attempt-01/case.json
```

Read the evidence. In run/attempt-01/answers.json, fill each decision with an exact label from the rules; add an explanation and evidence references such as record IDs or configuration line numbers. In notes.txt, separate observation, interpretation, uncertainty, next check and help used. Save your prediction before opening the answer.

```text
python lab.py analyse --case run/attempt-01/case.json
python lab.py check --case run/attempt-01/case.json --answers run/attempt-01/answers.json
python lab.py prepare --scenario changed --out run/attempt-02
python lab.py prepare --scenario challenge --out run/attempt-03
python -m unittest -v test_lab.py
python lab.py verify
```

For changed/challenge use the corresponding attempt folder in show/analyse/check. prepare refuses an existing output folder. --out is relative to this lab and must be below run/; explicit input paths are relative to your terminal directory. analyse/show default to the guided fixture if --case is omitted. Exit 0 means command success or decision match; 1 means differences/incomplete written fields; 2 means invalid input or a filesystem problem. The checker verifies decision fields and the presence of writing, not the quality of reasoning. verify checks shipped examples, not learner mastery.

## Deliverable and limits

Keep case.json, answers.json and notes.txt together in the ignored run folder. Deliver a short handover: the question, your conclusion, two precise evidence references, one alternative explanation or limitation, and the next evidence you would request. State whether you used the worked example and whether you completed a fresh case unaided. Do not describe an automated match as assessment or professional experience.

Deduplication uses alert ID and identical full content; different IDs are retained even when text or timestamps match. Clock accuracy, sensor coverage, account ownership and authorised activity remain unverified; no service or endpoint was contacted.

Use WORKED-EXAMPLE.md after making a first prediction, and DRILLS.md for progressively less support. Do not use these rules as an operational security verdict or modify a real host as part of this exercise.

## Primary references

The following documentation supports the underlying concepts; the tiny schema and explicit teaching rules above are specific to this lab. Links were checked during authoring on 23 October 2026. Running the lab needs no internet connection.

- [NIST log management guidance](https://csrc.nist.gov/pubs/sp/800/92/final)

# Worked example: Alert deduplication and correlation

Try the guided case before reading this page. Use `python lab.py prepare --scenario guided --out run/worked-01` if you need a fresh ignored copy.

Six deliveries contain five unique alerts because a1 is delivered twice. a1, a2 and a3 are distinct failures at 00:00:00Z, 00:00:10Z and 00:00:20Z. a4 succeeds at 00:01:00Z, strictly after all three, and a5 is a process alert at 00:02:00Z. All share workstation.test/student and lie within 300 seconds of a1. The group workstation.test/student/1 therefore gets review-auth-sequence, which takes precedence over the mixed-signal rule. It is one review group, not evidence of one confirmed incident. Check whether the account holder retried a password or performed an authorised task.

The reference decision fields for guided are:

```json
{
  "workstation.test/student/1": "review-auth-sequence"
}
```

Now open `python lab.py analyse --case run/worked-01/case.json` and compare observations, not only labels. Record one difference between your first prediction and the reference. If there was no difference, explain the rule that produced each decision rather than copying the label.

Handover prompt: explain what this evidence establishes, cite its exact IDs or lines, give a plausible alternative or remaining uncertainty, and name the smallest next check. Record any help used. A correct label by itself is not evidence of independent understanding.

# Drills: Alert deduplication and correlation

## Guided practice

Prepare guided in a fresh run folder, read the README rules, and make a prediction for every decision ID. Use WORKED-EXAMPLE.md only after saving that prediction. Correct your explanation in a different paragraph so the original reasoning remains visible.

## Prompts only

Close the worked example and use these questions to reconsider the evidence:

- Which record can be removed as a duplicate, and why?
- Would three failures at exactly the success timestamp count as earlier failures?

## Independent attempt

Prepare challenge in a fresh folder. Keep reference outputs, worked example and analyse closed until you have saved answers.json plus a five-sentence handover in notes.txt. Include the observation, interpretation, an alternative, uncertainty and next check. Use the README for syntax if needed and record that help. Afterwards run analyse/check and have a person inspect the explanation. Independent completion is unassessed unless someone actually reviews it.

## Changed case

Prepare changed. Predict which conclusions change and which do not. Explain the changed evidence before checking:

- Why do four deliveries of a1 count as only one failure?
- Why do alerts separated by host form different groups even at nearby times?

Small edit drill: In a copied challenge case, move a4 from 00:05:01Z to 00:05:00Z. Predict the effect of the inclusive 300-second boundary.

## Teach-back

Explain delivery count, unique-alert count, correlation-group count and confirmed-incident count as four different quantities. Use exact record IDs from your attempt. A partner should ask one counterexample rather than accepting a memorised label. Record the question and your response.

## Later recall

After 24-72 hours, reopen only your own notes and a fresh case. Write the central rule from memory, identify one boundary or missing-evidence condition and predict the result. Then check the documentation and record what you forgot. Do not count rereading an answer as an independent attempt.

## Handover

Keep the copied case, completed worksheet and notes together. State the scope, conclusion, two evidence references, unresolved uncertainty and requested next check. Include help used and whether someone assessed the reasoning. Do not claim experience beyond this fictional exercise.