HTTP access-log triage: original guided practice

# HTTP access-log triage: original guided practice

Read a small web-server access-log export, reconstruct a request sequence and
explain which evidence deserves a closer look. The busiest source is not
necessarily the most concerning. A response status alone cannot tell you what
information a visitor received or whether an account was accessed.

These are original guided materials for someone with introductory networking or
Wireshark experience. Everything in the cases is fictional: the site, addresses,
events, notes and future dates. This is separate practice, not a university
submission or evidence of professional experience. Jeremy's independent
completion has not been assessed. The supplied utility is a tool for the exercise;
the learner is not expected to develop Python or claim independent authorship of it.

## Start here

Use Python 3.10 or newer. Only its standard library is needed. Open a terminal in
this folder; on Windows, `py` can replace `python` if that is the installed command.
No command makes network requests, starts a server or contacts a target.

```sh
python lab.py --help
python lab.py check
python lab.py timeline
python lab.py triage
python lab.py check --report runs/report.json
```

Read [`fixture/context.json`](fixture/context.json) alongside the sorted timeline.
The table itself is [`fixture/access.csv`](fixture/access.csv). Open the generated
`runs/report.json` in a text editor. Write a short interpretation before opening
the [worked example](WORKED-EXAMPLE.md), then try the [drills](PRACTICE.md).

## Compact command contract

`lab.py` is the only maintained command entry point. Input paths are relative to
the terminal's current folder; output paths are relative to this lab folder.
Generated cases and reports must be inside this lab's ignored `runs/` folder.
Every output must be new, including a case directory that would otherwise be
empty. Repeating an output path is refused to protect fixtures and your notes.
Use a fresh name to repeat an exercise.

| Command | Input | Result |
| --- | --- | --- |
| `generate --seed 72 --scenario mixed` | A numeric seed and scenario | New `runs/seed-72-mixed/` containing access CSV, context and manifest |
| `timeline [--case DIRECTORY] [--source-ip ADDRESS]` | Checked-in fixture by default | Requests sorted by UTC time, optionally filtered to one exact source; no file written |
| `triage [--case DIRECTORY] [--volume-threshold 8] [--path-threshold 3] [--output runs/report.json]` | CSV and context from one case | Source summaries, evidence IDs, input hashes and review prompts in one JSON file |
| `check [--case DIRECTORY] [--report FILE]` | Manifest and case files; optional report | Recreate expected bytes from the seed, compare all three files, and optionally recalculate the report |
| `test` | Supplied fixture and temporary cases | Standard-library unit checks; temporary folders removed only from the test's own locations |

`generate --output runs/my-case` sets a fresh case directory. Exit status `0`
means completion, `1` means software tests failed, and `2` means invalid input or
a file problem. These numbers are not learner grades. `check` tests reproducible
file content and calculations; it does not judge an explanation or prove security.

## Reading an HTTP request

**HTTP** is the application protocol used for web requests and responses. An
access log records what the logging component observed. This lab uses a deliberately
small **CSV** export, a text table with comma-separated fields. It is not a raw
packet capture or a parser for every web-server log format.

| Field | Meaning in this export |
| --- | --- |
| `event_id` | Unique fictional record label, such as `HTTP-0017` |
| `timestamp` | Request time in UTC, written `YYYY-MM-DDTHH:MM:SSZ` |
| `source_ip` | Recorded source address; not proof of one device or person |
| `host` | Fictional destination site ending `.test` |
| `method` | `GET`, `HEAD` or `POST` |
| `target` | Requested path, optionally followed by a query starting `?` |
| `status` | HTTP response status recorded by the fictional server |
| `body_bytes` | Number of response body bytes sent in this invented export; excludes headers |
| `user_agent` | A client-supplied label beginning `Practice`; not verified identity |
| `synthetic` | Literal `true`, marking every record fictional |

`GET` requests a representation of a resource. `HEAD` requests similar response
metadata without a response body. `POST` asks a resource to process submitted
content; this export omits that content. These meanings follow
[HTTP method definitions](https://www.rfc-editor.org/rfc/rfc9110.html#section-9.3).

| Status used here | Basic meaning |
| --- | --- |
| `200` | The HTTP request succeeded; interpretation depends on the method and resource |
| `303` | The client is directed to another resource; the destination header is absent here |
| `403` | The server understood the request and refuses it |
| `404` | No current representation was found, or the server is unwilling to disclose one |

See the [HTTP status definitions](https://www.rfc-editor.org/rfc/rfc9110.html#section-15).
For this exercise, `HEAD` with `body_bytes=0` is expected. A `POST /login` response
with status `200` is not an authentication-result field. Body size does not tell
you whether a response was a form, a fallback page or sensitive information.

## What the three review rules count

The program groups requests by the exact `(host, source_ip)` pair over the
**whole collection interval**, including both endpoints. It reports:

1. **Request volume:** at least eight requests by default. This is a count, not
   requests per second. Compare collection durations before comparing counts.
2. **Watched-path variety:** at least three distinct paths from `/.env`, `/admin`,
   `/backup.zip` and `/debug`. These are exercise choices, not a universal list
   of malicious paths. An authorised tester or normal site feature could use them.
3. **Watched-path 200:** any request for a watched path with status exactly `200`.
   This asks what the server returned; it does not establish disclosure.

For rules 2 and 3, only text after `?` is removed before comparison. Matching is
case-sensitive and exact. `/admin?view=1` matches `/admin`; `/Admin` and `/admin/`
do not. Repeated requests and query variations do not increase path variety.
The parser accepts simple paths and queries only. It rejects percent encoding,
network-path references, traversal forms, unsupported methods and live addresses.
It does not decode or normalise paths, classify payloads or detect every attack.

Rules do not trust user-agent labels or automatically excuse a source described
in a context note. They do not combine different addresses. Shared addresses,
proxies, changing addresses and missing logs can change the interpretation.
Several prompts can refer to the same sequence. Neither prompt counts nor
request counts are counts of incidents, attackers or affected people.

## Write an evidence-led handover

Keep three things separate in a short note saved under `runs/`:

- **Observed:** collection interval and coverage, source, methods, targets,
  statuses, counts and evidence IDs. Record the rule settings.
- **Interpretation:** one plausible explanation, one alternative, and what the
  export cannot establish. A context note supports an explanation but needs
  corroboration before being treated as confirmed operational history.
- **Next check:** the smallest additional evidence that would change the
  decision, such as scheduled-job records, route configuration, a corresponding
  application event or the preserved response content. Describe the request;
  do not contact these invented addresses or replay the requests.

There are no authentication results, request or response bodies, cookies, proxy
chains, process activity or file changes in this export. `compromise_assessed`
is `false` because that assessment is not made. It does not mean the site has
been declared uncompromised.

## Variants and reproducibility

A **seed** reproduces the same fictional case. Changing it varies addresses,
site label, date and file order. The **scenario** changes the evidence:

| Scenario | Reasoning change |
| --- | --- |
| `mixed` | A busy crawler and a smaller sequence requesting watched paths |
| `soft-200` | One watched-path response is `200`; a fallback route is a competing explanation |
| `logging-gap` | A known missing interval hides some requests, so a rule can stop matching |
| `crawler-only` | Ordinary crawling still matches the volume rule |

`complete-for-fixture` means all planned events were exported by this generator.
It is not a claim that real access logs could establish completeness. `known-gap`
means the listed inclusive interval has no retained rows. The number and content
of missing events are not inferred by triage. Counts are lower bounds for that
collection period. A lack of prompts is especially weak evidence when records
are missing.

The checked-in fixture uses seed **53**, scenario **mixed**, generator **1.0.0**.
[`fixture/manifest.json`](fixture/manifest.json) records its fictional provenance
and SHA-256 fingerprints of the CSV and context. The program regenerates expected
bytes instead of trusting rewritten hashes. Files are UTF-8 with LF endings;
local `.gitattributes` preserves LF in Git. A hash verifies file identity, not
authorship, global novelty or learner competence.

The addresses use the [IPv4 documentation ranges](https://www.rfc-editor.org/rfc/rfc5737.html#section-3),
and the site uses the [reserved `.test` name](https://www.rfc-editor.org/rfc/rfc2606.html#section-2).
Do not substitute real logs or personal data. Run `python lab.py test` after
changing the utility. Independent learning evidence requires an explanation and
an unfamiliar variant; running the utility or reading the answer is practice.

# Worked example: seed 53, mixed

Try your own explanation first. This is a supplied reference analysis, not a
record of Jeremy independently completing an investigation.

## Establish scope

```sh
python lab.py check
python lab.py timeline
python lab.py triage --output runs/worked-report.json
python lab.py check --report runs/worked-report.json
```

The export contains **21** synthetic requests for `practice-library-053.test`,
with a collection interval of **2032-06-14 09:39:00–09:44:00 UTC**, inclusive.
There are three recorded source addresses. `complete-for-fixture` confirms the
generator's intended export, not real-world visibility. The timestamps are
invented future dates. CSV file order is shuffled; use the sorted timeline.

## Compare the sources

| Source | Evidence | Interpretation and next check |
| --- | --- | --- |
| `192.0.2.65` | 4 requests, `HTTP-0001`–`HTTP-0004`; three `GET` responses of `200`, then `POST /contact` with `303` | Compatible with ordinary browsing and a form redirect. The absent redirect header and request body limit further interpretation. |
| `198.51.100.126` | 12 requests, `HTTP-0005`–`HTTP-0016`, 09:42:00–09:42:11; eleven `GET`, one `HEAD`; nine `200`, three `404`; no watched path | The volume rule matches. Public pages, stale guide paths and the scheduled-job note support a crawler explanation. Confirm the job's timing and requests before treating the note as verified. |
| `203.0.113.138` | 5 requests, `HTTP-0017`–`HTTP-0021`, 09:43:00–09:43:20; four different watched paths, then `POST /login` | Watched-path variety matches despite lower volume. Requests for configuration, administration, backup and debug paths make probing plausible. Check authorisation and corresponding application records. |

The default settings are volume **8** and watched-path variety **3**. The report
contains **two review prompts**, one for volume and one for path variety. These
are questions for review, not two proven incidents.

For the third source, `HTTP-0017` requested `/.env` and received `404`;
`HTTP-0018` requested `/admin` and received `403`; `HTTP-0019` and `HTTP-0020`
requested `/backup.zip` and `/debug`, both with `404`. These records show paths
and outcomes observed by this logging component. They do not demonstrate code
execution, exposed file contents or account access.

`HTTP-0021` is `POST /login`, status `200`, with a 470-byte response body recorded.
The export contains neither the body nor an authentication result. A response
that redisplays a form is one possibility. “The attacker logged in successfully”
would go beyond this evidence.

The crawler and the watched-path source both claim `PracticeCrawler/1.0` as
their user-agent. That text is client-supplied. It cannot transfer the owner's
scheduled-job note from one source to the other or establish a shared actor.

## Example handover

> Review requests from 203.0.113.138 to practice-library-053.test during
> 2032-06-14 09:43:00–09:43:20 UTC. Four distinct watched paths appear in
> HTTP-0017 through HTTP-0020, exceeding the configured threshold of three.
> Probing is plausible; an authorised test remains possible. HTTP-0021 records
> a 200 response to POST /login, with no authentication outcome available.
> Check authorised testing records and correlate the five event times with
> application routing and authentication records. The export alone does not
> establish disclosure or compromise. Separately, corroborate the scheduled
> crawler note for 198.51.100.126 before closing its volume prompt.

This handover names evidence and uncertainty, and requests information that
could change the decision. It does not recommend blocking an address merely
because it sent the most requests.

## How the answer changes

With the same seed and `soft-200`, `HTTP-0017` changes to `200` with 128 body
bytes. A third prompt, `watched-path-200`, appears. The new context note says a
fallback route can return `200` for unknown paths. Both a fallback response and
unexpected content need consideration; the note and byte count do not identify
the actual body. Request the relevant route/application evidence and preserved
response before asserting disclosure.

With `logging-gap`, the export retains **18** requests. The missing interval is
09:43:05–09:43:15 UTC. Only `HTTP-0017` and `HTTP-0021` remain for the third
source, so its variety rule no longer matches. The report retains the gap warning.
A learner can observe the missing interval; the triage report does not reconstruct
the omitted requests. “No rule matched, therefore no probing occurred” is not a
supported conclusion.

With `crawler-only`, **16** requests remain and only the volume prompt appears.
A false positive is a rule match that proves unrelated to the suspected security
problem after investigation. High volume warrants context, not an automatic
malicious label.

# Practice drills

Use the utility as supplied. You do not need to write Python, memorise flags or
replay requests. Keep notes under `runs/`, using a new filename for each attempt.
The generator refuses an existing case directory, even if it contains only notes.

## 1. Guided reading

Run the reference timeline and identify one request method, target, status and
response-body size. Explain why `HEAD /guides`, status `200`, with zero body bytes
is not a failed request. Find the contact-form request and name the missing
header that would tell you where its redirect leads.

```sh
python lab.py timeline
python lab.py timeline --source-ip 203.0.113.138
```

Write the difference between “the server returned a successful HTTP response”
and “someone successfully authenticated.” Check the worked example only after
writing your answer.

## 2. Use a short checklist

```sh
python lab.py generate --seed 72 --scenario mixed
python lab.py timeline --case runs/seed-72-mixed
python lab.py triage --case runs/seed-72-mixed --output runs/attempt-72.json
python lab.py check --case runs/seed-72-mixed --report runs/attempt-72.json
```

Keep the worked answer closed. Record the coverage, busiest source, watched-path
source, evidence IDs, one plausible benign explanation and one next check.
Explain why the two sources can share a user-agent without being the same actor.
The changed seed alters labels and times; it does not create a different reasoning
problem. Complete one of the next variants to test transfer.

## 3. Change the response

```sh
python lab.py generate --seed 88 --scenario soft-200
python lab.py timeline --case runs/seed-88-soft-200
python lab.py triage --case runs/seed-88-soft-200 --output runs/attempt-88.json
```

Identify what changed in the evidence and which prompt was added. Give two
possible explanations for the watched-path `200`. Name the evidence that would
distinguish them. Explain why a byte count and a fallback-route note are not
equivalent to seeing the response body.

## 4. Lose part of the evidence

```sh
python lab.py generate --seed 91 --scenario logging-gap
python lab.py timeline --case runs/seed-91-logging-gap
python lab.py triage --case runs/seed-91-logging-gap --output runs/attempt-91.json
```

Name the gap interval, the retained requests and which rule no longer matches.
Write a conclusion that remains valid despite the missing records. What should
you request before treating the lower count as a reduction in suspicious activity?
Event IDs with gaps alone are not a universal method for proving missing logs;
use the provided coverage record in this case.

## 5. Challenge a noisy rule

```sh
python lab.py generate --seed 102 --scenario crawler-only
python lab.py triage --case runs/seed-102-crawler-only --output runs/attempt-102.json
python lab.py triage --case runs/seed-102-crawler-only --volume-threshold 13 --output runs/attempt-102-threshold13.json
```

Explain why the default volume rule matches and why increasing its threshold
removes the prompt. The records did not change. Does a quieter report establish
that the new threshold is better? Describe what real baseline and missed-case
evidence you would need before deciding.

## 6. Explain an unfamiliar case

Ask a study partner to choose a fresh seed and one of the four scenarios. They
should give you the case folder, not their answer. If working alone, choose a
different scenario after a break and keep the reference answer closed.

Produce a short handover containing:

- Collection scope, coverage and exact settings.
- A finding tied to event IDs and observed methods, paths and statuses.
- A plausible alternative explanation and a limit on what can be concluded.
- A next evidence request that could change your decision.
- One example of a false positive or a case the simple rules could miss.

Then use `check --case YOUR-CASE --report YOUR-REPORT` for file and calculation
integrity. Compare your reasoning with the scenario discussion in the worked
example. Record corrections honestly. A passing software check is not proof of
understanding; explaining a fresh variant without the answer is stronger evidence.