Guideguide/06-evaluation-playbook

The Evaluation Playbook

The longest document here, deliberately. In systems research, credibility lives in the evaluation, and the difference between a thesis that survives questioning and one that does not is usually measurement discipline rather than the quality of the idea.

Read this during Phase 1, while you are building the apparatus, not in Phase 3 when you are using it.

Control the environment first

Modern machines are hostile to measurement. Frequency scaling, turbo, thermal throttling, SMT, address-space randomization, transparent huge pages, background daemons, and interrupt delivery all introduce variance that can exceed the effect you are trying to detect.

Before your first real number, get variance under control and characterize it. Run the same benchmark thirty times unchanged and look at the distribution. If run-to-run variation is 15% and you are hoping to demonstrate an 8% improvement, you cannot demonstrate anything yet, and no amount of later analysis fixes it. This measurement is a Phase 1 gate for a reason.

experiments/scripts/setup-machine.sh applies the settings below; capture-env.sh records them so you can prove what you did.

Pin the frequency. Set the governor to performance and disable turbo, because turbo makes throughput depend on thermal history and therefore on what you ran ten minutes ago.

sudo cpupower frequency-set -g performance
echo 1 | sudo tee /sys/devices/system/cpu/intel_pstate/no_turbo    # Intel
echo 0 | sudo tee /sys/devices/system/cpu/cpufreq/boost            # AMD

Isolate the cores you measure on. Reserve cores at boot with isolcpus and nohz_full, then run pinned with taskset or cpuset. Move interrupts and kernel threads off those cores. Run your load generator on a different socket from the system under test, or better, on a different machine, because a generator competing for the same cache and memory bandwidth silently distorts what you are measuring.

Disable SMT while establishing baselines. Hyperthread pairing makes results depend on co-scheduling luck. You can re-enable it later as an explicit experimental variable, which is a legitimate thing to study, but not while you are trying to establish a clean baseline.

Decide the memory configuration and hold it fixed. Transparent huge pages materially affect both performance and variance; either setting is defensible, silently changing between runs is not. Pin memory to the local NUMA node with numactl --membind unless NUMA behavior is your subject. Record whether you drop caches between runs, and do the same thing every time.

Quiet the machine. Stop unnecessary services, disable cron and automatic updates, close everything else, and never measure over SSH while also copying files. Check that no other user is logged into a shared machine — trivially obvious and a routine source of ruined result sets.

Record everything. Kernel version and boot command line, CPU model and microcode revision, all the settings above, every software version, and the git commit of both your code and your scripts. Microcode updates have changed performance meaningfully, and "we upgraded the kernel halfway through" is a genuinely unrecoverable situation if you did not record which runs came from which.

Warm-up and steady state

Never include cold-start effects unless cold start is what you are studying. Caches, branch predictors, TLBs, JIT compilation, filesystem caches, connection pools, and — for learned systems — model warm-up all need time to reach steady state.

Discard a warm-up period, and justify its length with data rather than by convention. The way to do that is to plot the metric over time for one long run and show where it flattens. Put that plot in your thesis or your appendix; it converts "we discarded the first 30 seconds" from an arbitrary choice into a documented one, and it takes one figure.

Then confirm you are actually in steady state: no trend across the measurement window, and the first and second halves of the window statistically indistinguishable. If a metric drifts throughout — memory fragmentation, cache pollution, a leak — that drift is itself a finding and needs reporting rather than averaging away.

Repetitions and statistics

Never report a single run. This is the single most common flaw in student systems evaluations and it is immediately visible.

Do at least five independent runs, preferably ten or more, and prefer separate process invocations over iterations within one process, because a fresh invocation captures sources of variance that an inner loop does not.

Report the median with the interquartile range, or the mean with a 95% confidence interval — and state which one you are showing. An error bar with no stated meaning is uninterpretable and readers will assume the least favorable interpretation. Systems measurements are usually right-skewed, which makes the median the more honest summary. Report the number of runs in every caption or in your methodology section.

Two mistakes worth naming because they are easy to make and hard to spot in a finished document.

Do not average percentiles across runs. The mean of five p99 values is not the p99 of the combined data, and the error is not small. Merge the raw distributions and compute the percentile once, which is what HDR histograms exist for.

Do not claim a difference that your confidence intervals do not support. If the intervals overlap substantially, you have not shown an improvement — you have shown you need more runs or better environmental control. A committee member will check this on your headline figure. The honest options are to gather more data, tighten the environment, or state the result as "we observe no significant difference," which is a legitimate finding.

For a formal comparison, the Mann-Whitney U test is usually more appropriate than a t-test for systems latency data, since it does not assume normality. Reporting effect size alongside significance is better practice than significance alone, especially with large sample counts where trivial differences become "significant."

Latency measurement, and coordinated omission

If you measure latency, you must understand coordinated omission, because it invalidates a large fraction of naively collected latency data and it is invisible in the output.

The mechanism: a closed-loop load generator sends a request, waits for the response, then sends the next. When the system stalls for 100ms, the generator also stalls — so instead of recording the many requests that should have been issued during the stall and would each have observed high latency, it records one slow request. The stall is systematically underrepresented in exactly the tail you care about, and reported p99s can be off by orders of magnitude.

The fix has two parts. Use a load generator that compensates, such as wrk2 rather than wrk, or one that issues requests on a schedule independent of responses. And be explicit about your load model, because open-loop and closed-loop measure genuinely different things: open-loop drives a target request rate independent of responses and is the right model for serving systems where clients are independent, while closed-loop maintains fixed concurrency and is appropriate for a fixed thread pool. State which you used and why. Reporting tail latency from a closed-loop generator without qualification is a methodological error a reviewer may well catch.

Related: report latency as a CDF or with explicit percentiles, never as a mean. Latency distributions are heavy-tailed and a mean tells the reader almost nothing about the behavior users experience. Use HdrHistogram or equivalent to record full distributions cheaply, and present p50, p99, p999 in tables and CDFs in figures.

Finally, always report latency together with the load at which it was measured. Latency at 20% utilization and at 90% utilization are different universes, and a latency number without a load level is not a claim. The strongest presentation is a latency-versus-throughput curve, which shows both the improvement and how the saturation point moves.

Explain the result, do not just report it

The difference between measurement and evaluation is causal explanation. "Our system is 18% faster" is a measurement. "Our system is 18% faster because it reduces L3 misses by 40%, and here is the counter data" is evaluation. The second survives questioning; the first invites the reader to invent their own explanation, which may be that you got lucky.

Get the evidence from the layer below your end-to-end numbers: perf stat for cycles, instructions, IPC, cache misses, and branch mispredictions; perf record and flame graphs for where time goes; bpftrace or ftrace for kernel-path latency distributions; PMU counters for the specific resource you claim to be improving.

Watch for two things. Observer effect is real — tracing can cost more than what you are measuring, so validate that your end-to-end numbers are unchanged with instrumentation disabled, and never mix traced and untraced runs in the same comparison. And measure at the layer where your mechanism acts in addition to end-to-end, because a microbenchmark showing your decision path costs 140ns is a much stronger form of evidence than an end-to-end delta that could have many causes.

If you cannot explain a result, do not report it as if you can. "We observe a 4% improvement that we cannot fully attribute" is honest and occasionally interesting; a confident wrong explanation is much worse than an acknowledged gap.

Ablations and sensitivity

Ablations show that each part of your design earns its place. Disable one component at a time and measure. If removing something does not hurt, you have learned something useful: simplify the design. Present these as a table with the full system, each single-component removal, and the baseline. This is also where you find out whether the sophisticated part of your system is doing the work, or whether a simple heuristic inside it is — which is uncomfortable to discover and much better discovered by you.

Sensitivity analysis shows your result is not one lucky configuration. Sweep the parameters that matter — thresholds, model size, history window, learning rate — and show the shape of the response. A flat region means the system is robust and easy to deploy; a sharp peak means it is fragile, which you should report rather than hide, since a system that only works at one hyperparameter setting is a real limitation and an honest one.

Sweep the workload dimensions too: request rate, working-set size relative to cache, read/write mix, concurrency, arrival burstiness. Somewhere in that space your mechanism stops helping. Find that boundary and characterize it — "our approach helps when the working set exceeds L3 and stops helping below it" is exactly the kind of statement that makes a thesis useful to someone else.

For learned components specifically

The measurement issues that are distinctive to angle A and to learned-systems work generally.

Report end-to-end system performance with the model in the loop, always. Offline prediction accuracy is a hypothesis, not a result. A model with 95% accuracy can produce a slower system than an 80%-accurate one if the 5% of errors are catastrophic or if inference costs more than the decision it improves.

Report inference cost at the tail, not the mean. A predictor averaging 100ns but occasionally taking 10µs on a cache miss may be unusable in a hot path, and the mean conceals exactly the behavior that matters.

Report feature-collection cost separately. It is frequently larger than inference and frequently omitted. If you must touch three cache lines of per-task state to build a feature vector, that is your real cost.

Report training cost honestly: data collection, training time and hardware, and how often retraining is needed in deployment. A policy needing nightly retraining on a GPU has a different deployment story from one trained once, and the reader needs to know which you are proposing.

Report cold start. What happens in the window before the model is useful? Is there a fallback to the heuristic? What does the transition cost?

Report behavior under distribution shift, which for a systems audience is often the decisive question. Change the workload mid-run and show what happens. Graceful degradation to heuristic-level performance is a genuinely strong result and worth its own figure. A policy that is 20% better on matched workloads and 50% worse on shifted ones is not deployable, and saying so yourself is far better than having it discovered.

Compare against a tuned heuristic. See guide/03-research-design.md. Beating an untuned default is not a result.

Split your data temporally and by workload, never randomly. The leakage discussion in guide/03-research-design.md is the most important paragraph in that document. Random splits on autocorrelated traces produce inflated accuracy that will not survive review.

Report the simple-model comparison. Always evaluate whether a decision tree, a linear model, or a small table of statistics gets most of the benefit. Reviewers will ask, and if the answer is yes, that is a publishable finding rather than a defeat — "a 12-node decision tree captures 90% of the gain of our neural policy at 1/50th the inference cost" is a better result for a systems thesis than the neural policy alone.

Figures that hold up

Figures carry more of your argument than your prose does, because they are what gets looked at. Standards worth holding:

Label both axes with units, always, including the ones that seem obvious. Start bar-chart y-axes at zero, since a truncated axis exaggerating a small difference is the classic misleading graph and a reader who spots it distrusts everything else. Use log scale for latency distributions and anything spanning orders of magnitude, and say so on the axis. Show variance as error bars or bands, and state in the caption what they represent.

Use CDFs for latency rather than bar charts of means. Draw a horizontal reference line at 1.0 on normalized plots, and state the baseline in the caption. Keep colors colorblind-safe and ensure the figure survives grayscale printing, which means distinguishing series by marker or line style as well as color.

Check readability at final print size: text in figures should be roughly the size of your body text, which usually means setting the figure's font size explicitly rather than scaling a large figure down. And keep the design consistent across the thesis — same colors for the same systems in every figure, so readers learn your visual vocabulary once.

Generate everything as vector PDF from scripts, never screenshots. Keep the script that produced each figure next to the figure. See experiments/README.md.

The methodology section

Your evaluation chapter opens with methodology, and it does more work than any other section in establishing whether readers trust your numbers. Write it in enough detail that someone could reproduce your setup.

It should cover the hardware specification including CPU model, core count, cache sizes, memory configuration, storage, and any accelerators; the software stack with exact versions including kernel, distribution, compiler, and libraries; the environmental controls you applied, referring to the checklist above; each workload with a sentence on what it represents and why it is relevant; how you configured and tuned each baseline; and how you report results, meaning repetition count, warm-up period, discard policy, and which statistic your error bars show.

Write this section early, in Phase 1, when the details are in front of you. Reconstructing your exact THP setting in month nine is unpleasant and error-prone.

Pre-submission checklist

Before the results freeze, verify every one of these.