Offline file integrity and change triage

# Offline file integrity and change triage

A fictional community workshop has approved a notice update. Other files may
also have changed. Your task is to describe the evidence, separate explained
changes from unexplained ones, and judge whether the baseline deserves confidence.

This is original guided practice for a beginner. It uses generated dummy text
files, runs entirely offline, and needs only Python 3.10 or later. There are no
accounts, live systems, executable payloads, or dependencies to install. You use
commands and explain results; you do not need to write a program.

Independent completion and understanding have not been assessed. The utility,
fixtures, and supplied answer key are learning scaffolds, not a claim of
professional experience or independent learner authorship.

## The security idea

A SHA-256 hash is a compact value calculated from file bytes. Comparing it with
a previously recorded value helps detect a byte change. A changed hash does not
explain the change: an approved edit and an unauthorised edit both alter bytes.
Even a line-ending conversion can alter a hash.

This lab combines three pieces of evidence:

1. A **baseline** records file names, byte lengths, and hashes before the change
   window, if that history is supported.
2. A **current inventory** records what is present now.
3. The supplied **change list** records approved paths and exact before/after
   hashes. A path or ticket number alone does not approve arbitrary new contents.

Hash agreement does not prove that a file is safe, authentic, or unchanged since
some earlier time that was never measured. An attacker could replace a file and
an unprotected baseline together. This lab's scenario includes a fictional
reference anchor for most variants; it is editable local training evidence,
not a digital signature or an independent trust service.

## Command contract

Run these commands from this folder. `lab.py` is the single maintained entry
point. The default `--run` is `run/` beside the script; explicit paths are
relative to your current working directory. The parent of a new run must exist.

| Command | Input and result | Exit codes |
| --- | --- | --- |
| `python lab.py generate` | Creates a new sandbox with `reference/`, `current/`, scenario, approvals, and answer key. Defaults to `mixed`, seed `23`. Refuses an existing run folder, even an empty one. | `0` created; `2` invalid input or existing folder |
| `python lab.py baseline` | Scans `run/reference/` and creates `run/baseline.json`. Does not modify the reference or current files. Refuses an existing output. | `0` created; `2` invalid input or existing output |
| `python lab.py check` | Scans `run/current/`, compares it with the baseline, and applies the supplied exact change approvals. Prints a report without repairing anything. | `0` no unexplained changes and exercise anchor matches; `1` review needed; `2` invalid or unreadable input |
| `python lab.py manifest --verify` | Checks the shipped source and fixture hashes against `PROVENANCE.json`. | `0` match; `1` difference; `2` invalid or unreadable input |

All commands accept `--json` for structured output. Input errors go to standard
error. Use `python lab.py COMMAND --help` for options.

`generate` accepts `--variant approved`, `mixed`, `replacement`, or
`suspect-baseline`, plus `--seed 0` through `999999`. `baseline` accepts
`--source reference` or `--source current` and `--name NAME.json`. `check`
accepts `--baseline NAME.json`. Baseline names must be a single lowercase JSON
file name inside the chosen run; they cannot point outside it.

```console
python lab.py generate
python lab.py baseline
python lab.py check
```

The default check exits `1` deliberately: three changes need investigation.
Read [the worked example](WORKED-EXAMPLE.md), then use [the drills](DRILLS.md)
with unfamiliar inputs. A check result is an evidence classification, not a
learner grade. Explain your reasoning before opening `expected-findings.json`.

## How the report decides

- **Unchanged:** the name, hash, and byte length match the supplied baseline.
- **Approved:** a changed, added, or deleted path matches both hashes in its
  change-list entry. A missing side is recorded as `null` for an addition or
  deletion.
- **Unexpected:** the observed change lacks that exact approval. It needs
  investigation; the output does not call it malicious.
- **Unused approval:** a listed change was not observed exactly. This can mean
  work has not happened yet, the wrong baseline was used, or contents differ
  from the approved version. It is context, not automatically an incident.

The checker independently reports whether the baseline content matches the
scenario's reference anchor. It canonicalises JSON before hashing, so indentation
changes to the inventory do not create an anchor mismatch. Hashes of the actual
dummy files always cover their exact raw bytes. A matching anchor supports only
the supplied exercise history. An absent or mismatched anchor produces exit `1`,
even when no unexpected file differences appear.

| Variant | What changes in the conclusion |
| --- | --- |
| `approved` | The notice edit matches approval; no unexplained changes. |
| `mixed` | The notice is approved; a config edit, an addition, and a deletion are unexplained. |
| `replacement` | The notice and the exact report replacement are approved; additions and deletions are not inherently suspicious. |
| `suspect-baseline` | A prior config change is already in both trees. The comparison sees no config difference, but the baseline has no independent anchor and conflicts with a written requirement. |

A seed changes synthetic labels, counts, ticket IDs, and their hashes. It does
not change the scenario's security reasoning; select another variant for that.

## Files, safety, and limits

`reference/` is a supplied exercise snapshot, and `current/` is the observed dummy
tree. Preserve both while writing your first findings. Only fresh run creation
and new baseline files are written by normal commands. The checker does not
execute files, contact services, delete, quarantine, or restore anything.

The utility is deliberately limited to generated sandboxes with a valid scenario
file and the named `reference/` and `current/` children. It scans actual directory
entries rather than opening paths from a baseline. It rejects path traversal,
absolute baseline entries, Windows device names, uppercase or unsupported file
names, symbolic links, reparse points (including junctions), hard-linked files,
and nonregular files. Bounds are 100 files, 100 directories, eight directory
levels, 1 MB per file, and 10 MB of file content. An empty current tree can be
reported as deletions; an empty baseline is rejected.

This is a small static-tree exercise, not a race-resistant forensic acquisition
tool. Do not change files while a command is running. Permission errors or
unsupported inputs stop the check instead of producing a reassuring partial
report. Metadata such as owner, permissions, timestamps, alternate data streams,
and extended attributes is outside its model. Renames appear as addition and
deletion. The lab cannot determine actor, intent, execution, compromise, or
whether every required file was included in an originally trusted baseline.

Generated files use deterministic UTF-8/LF bytes, and the local `.gitattributes`
preserves LF for shipped files. `PROVENANCE.json` records original synthetic
provenance and hashes of the utility, documentation, tests, and worked fixture.
It does not authenticate its own contents or establish educational acceptance.
`python lab.py manifest --write` creates a missing provenance inventory for a
reviewed release; it refuses to overwrite an existing one. Updating a release
requires deliberate review and replacement of that inventory by a maintainer.

```console
python -m unittest discover -s . -p test_lab.py -v
python lab.py manifest --verify
python lab.py check --run fixtures/worked-example
```

The shipped fixture check exits `1` with the expected mixed-case findings.
The tests cover changed conclusions, exact approvals, poor baselines, removal
of every current file, deterministic bytes, malformed inputs, output protection,
and link/path restrictions. Link creation tests report a skip if the operating
system denies creation; a separate reparse-attribute test always checks rejection.

# Worked example: a notice update and three unexplained changes

This supplied walkthrough uses the `mixed` variant and seed `23`. It is not a
record of independent learner work. All names and file contents are fictional.

## 1. Establish the scope

```console
python lab.py generate --variant mixed --seed 23 --run run-walkthrough
```

Read `run-walkthrough/scenario.json` and `run-walkthrough/approvals.json` in a
text editor. The supplied exercise history says that `reference/` was checked
before the change window. A fictional owner approved the notice's opening time
changing from `09:00` to `10:00`. The approval records the exact before/after
file hashes. No approval is supplied for the config, temporary note, or archive.

The scenario and approvals are the exercise's authority. In real work, their
source, custody, and protection would also need verification.

## 2. Create the measurement

```console
python lab.py baseline --run run-walkthrough
```

The baseline contains four file records. `docs/notice.txt` is 39 bytes before
and after its edit: a size-only check would miss that change. SHA-256 compares
the bytes, so equal length does not prevent detection.

Creating a baseline is only measurement. Confidence comes from the history and
protection of its source. Do not create a new baseline from current files merely
to remove findings.

## 3. Interpret the comparison

```console
python lab.py check --run run-walkthrough
python lab.py check --run run-walkthrough --json
```

The result is `REVIEW NEEDED`, exit code `1`. Its baseline status is
`matches-exercise-anchor`.

| Path | Observation | Assessment and next action |
| --- | --- | --- |
| `docs/notice.txt` | Modified; same byte length, different hash | Exact hashes match `CHANGE-000023`; record as the approved update. |
| `docs/contact.txt` | Unchanged | No difference from this baseline; make no broader security claim. |
| `config/service.txt` | Modified: `remote_access=disabled` became `enabled` | Unexpected. Preserve evidence and ask the fictional owner whether a change record is missing. This text does not enable a real service. |
| `docs/temporary-note.txt` | Added | Unexpected. Establish its source and purpose before deciding what action is appropriate. |
| `reports/archive.txt` | Deleted | Unexpected. Confirm the retention requirement and whether removal was authorised; do not restore automatically. |

There is **one unchanged file, one approved change, and three unexpected
changes** across five observed paths. The count includes the missing baseline
file and the new current file. The current folder itself contains four files.

A suitable short finding is:

> The supplied baseline matches the exercise anchor. The notice update matches
> its exact approval. The config edit, new temporary note, and missing archive
> have no matching change record. I would preserve these observations and check
> the owner's records before deciding on recovery or escalation. Hashes establish
> differences from this baseline; they do not identify the actor or prove an attack.

## 4. Test a misleading alternative

```console
python lab.py baseline --run run-walkthrough --source current --name baseline-current.json
python lab.py check --run run-walkthrough --baseline baseline-current.json
```

This comparison shows four unchanged current files and no unexpected differences.
However, its baseline status becomes `does-not-match-exercise-anchor`, and it
still exits `1`. The new measurement has absorbed the changes, so it cannot
establish the earlier state. Its unused approval also shows that the expected
notice transition was not observed in this comparison.

The separate `suspect-baseline` variant is harder: no independent anchor exists.
The written requirement and the source history must be considered even when
the file comparison itself looks uneventful. Continue with [the drills](DRILLS.md).

# Practice: make and explain your own finding

Use a fresh folder for each attempt. These are optional practice stages, not
recorded passes or evidence of professional work. Before opening the supplied
`expected-findings.json`, write a short note with: observation, approval match,
baseline confidence, uncertainty, and next action.

## Guided: a routine approved change

```console
python lab.py generate --variant approved --seed 41 --run run-approved
python lab.py baseline --run run-approved
python lab.py check --run run-approved
```

Explain why a hash difference alone is not an incident. Identify the exact
evidence that permits the notice update. What would happen if the notice
contained `11:00` instead of the approved `10:00`? The same path and ticket
would no longer explain those different bytes.

Expected self-check: one approved edit, no unexpected differences, and a
matching exercise anchor; exit `0`. That result says nothing about your
understanding until you can explain the evidence and limits.

## Hints: an addition and a deletion

```console
python lab.py generate --variant replacement --seed 71 --run run-replacement
python lab.py baseline --run run-replacement
python lab.py check --run run-replacement
```

Read the change list before predicting the result. Do additions always need an
alert? Does a deletion always mean lost data? Explain how `null` represents the
missing side of each approved transition. Why would an exact byte check be more
useful than trusting the replacement report's name?

Expected self-check: the notice, archive deletion, and exact replacement report
are approved; no unexplained changes; exit `0`. This is a different conclusion
from the mixed case, even though both contain an addition and deletion.

## Independent: apparently unchanged, but uncertain

```console
python lab.py generate --variant suspect-baseline --seed 83 --run run-suspect
python lab.py baseline --run run-suspect
python lab.py check --run run-suspect
```

Keep the worked answer closed until you have a finding. Read the scenario's
baseline history and inspect `reference/config/service.txt` and
`current/config/service.txt` with a text editor.

- Does the baseline describe a verified earlier state?
- What does the written requirement say about remote access?
- Can matching hashes resolve the conflict with that requirement?
- What independent evidence would you request before calling the state clean?
- Why could simply accepting a new baseline conceal a problem?

Expected self-check: the notice is approved and the config is reported unchanged,
but the baseline has no anchor; exit `1`. Both copies already contain `enabled`,
which conflicts with the written `disabled` requirement. The comparison alone
cannot reveal when that change happened or who made it.

## Teach it back

Use a fresh mixed case with another seed. Explain one approved change, one
unexpected change, and the baseline's limits without reading the answer key.
Describe a plausible benign explanation for an unexpected change, the evidence
needed to test it, and why you would preserve the observation before repair.
Do not infer an attack from a hash mismatch alone.

An honest portfolio note could say: "I practised comparing synthetic file
inventories with an approved change list using a supplied offline utility."
Add your own specific reasoning and difficulties only after doing the exercise.
Do not claim the supplied utility or example findings as independent work.