Experiments

Set this up in Phase 1, before your first real number. The point is that in month nine, when a reviewer questions figure 7, regenerating it is one command rather than an afternoon of remembering which spreadsheet it came from.

Two standards, both self-interest rather than virtue:

1. One command per experiment. You will rerun everything more times than you expect.
2. Every figure regenerable from raw data by script. Manual steps decay and drift.

See guide/06-evaluation-playbook.md for the measurement methodology these scripts support.

Layout

experiments/
├── scripts/
│   ├── setup-machine.sh     apply environmental controls (Linux, needs root)
│   ├── capture-env.sh       record the full environment; run before every experiment
│   ├── run.sh               experiment driver template with repetitions built in
│   └── lib/
│       ├── stats.py         median/IQR, CI, correct percentile merging
│       └── plot_style.py    consistent matplotlib defaults for thesis figures
├── exp01-baseline/
│   ├── config.env           all parameters for this experiment, version controlled
│   ├── notes.md             what this experiment is for and which claim it supports
│   └── analyze.py           raw results -> numbers and figures
└── results/                 IMMUTABLE. Never edit or overwrite. Not in git.
    └── exp01-baseline/
        └── 2026-08-26T14-03-12Z/
            ├── env.txt      captured environment
            ├── config.env   the config as it was for this run
            ├── run-01.log   raw output, one file per repetition
            ├── ...
            └── summary.json derived numbers

results/ is append-only. Disk is cheap and a lost baseline is expensive. Never overwrite a result directory; every run gets a new timestamp.

Running an experiment

cd experiments
sudo ./scripts/setup-machine.sh          # once per boot, on the machine under test
./scripts/run.sh exp01-baseline          # captures env, runs N repetitions, writes results
python3 exp01-baseline/analyze.py results/exp01-baseline/<timestamp>

run.sh runs 10 repetitions by default and refuses to run if the machine has not been prepared. Repetition is built into the harness deliberately, so that a single run is not the path of least resistance — the single-run graph is failure mode 7.

Creating a new experiment

Copy exp01-baseline/ to expNN-<name>/ and edit three things: notes.md to say which claim from your claims-to-experiments matrix this supports, config.env for the parameters, and analyze.py for the analysis.

If you cannot name the claim in notes.md, do not run the experiment. See guide/03-research-design.md.

Figure provenance

Keep every figure's generating script next to the figure, and write the mapping into thesis/appendix/a-reproduction.tex as you go. Filling that table in at the end is much harder than filling it in as you go, and it is the thing that saves you when a number is questioned late.

Figures go to thesis/figures/ as vector PDF, generated by analyze.py. Never screenshot a plot.

A note on your development machine

Your laptop is not a measurement platform. macOS in particular gives you no frequency pinning, no core isolation, aggressive thermal management, and no perf. Develop locally, measure on a Linux machine you control exclusively.

If you do not yet have exclusive extended access to such a machine, that is your highest priority this week — it has a lead time you do not control, and discovering the problem in month four is expensive.

Environment capture

capture-env.sh records kernel version and boot command line, CPU model and microcode, governor and turbo state, THP and NUMA configuration, memory layout, software versions, and the git commit of your code and scripts.

Run it for every experiment, automatically, which run.sh does. A result whose environment you cannot reconstruct is not a result, and by month nine you will have results from four machine configurations that are otherwise indistinguishable. "We upgraded the kernel halfway through" is unrecoverable if you did not record which runs came from where.