Research Design: Deciding What Would Convince a Skeptic

Design your evaluation before you build your system. This ordering feels backwards and is the single highest-leverage habit in this guide.

The reason is simple. If you build first and evaluate later, you will unconsciously choose experiments that flatter what you built. If you decide first what evidence would convince a hostile expert, you discover in month two that you need a workload trace you do not have, or a baseline you cannot obtain, or a measurement you cannot make — at a point where all three are still fixable.

The claims-to-experiments matrix

The core artifact. Use templates/claims-experiments-matrix.md.

Every claim maps to at least one experiment. Every experiment supports at least one claim. The second direction is the one that saves you time: an experiment supporting no claim is an experiment you should not run, and in a year you will generate many tempting ones.

The "would falsify it" column is the one people skip and the one that matters most. Filling it in forces the claim to be concrete, and it protects you later: when a result comes in ambiguous, you have already decided in advance what counts as success, rather than deciding after seeing the number. That is the difference between evaluation and rationalization.

Review this matrix with your advisor in Phase 0 and again at the mid-point.

Baselines

Weak baselines are the most common fatal flaw in systems theses, and the easiest for a committee to spot. You need up to four kinds, and the second and third are mandatory.

The naive baseline — the simplest thing that could work. Establishes the floor and is sometimes humbling; if a three-line heuristic captures 80% of your gain, you need to know that before your committee does.

The production default — what actually ships. CFS or EEVDF for CPU scheduling, the existing LRU-ish page replacement, vLLM's default scheduler, whatever the real incumbent is. This is the comparison readers care about most.

The strongest prior work you can obtain — the best published approach in your specific niche. When the artifact is unavailable, say so explicitly and be transparent about how you approximated it, including using their reported numbers with the caveat that hardware differs. Skipping this comparison entirely reads as avoidance.

An oracle or upper bound, where one is definable — a policy with perfect future knowledge. This is optional but valuable, because it reframes your result. "We capture 70% of the achievable gain" is a much more informative statement than "we improve by 12%," and it tells the reader whether the remaining headroom is worth anyone's time.

Tune your baselines, and document the tuning. This is the part that requires discipline, because you are spending effort making your competition look good. Do it anyway, and write down what you tuned and how. An untuned baseline invalidates every number in your evaluation chapter, and a committee member who suspects it will ask — the correct answer is a paragraph in your methodology section that already answers the question.

A note specific to angle A (AI for OS): your baseline is not "no prediction." It is the existing heuristic, which already encodes decades of accumulated wisdom about this exact problem. Treat it with respect and it will make your eventual result far more credible.

Choosing workloads and benchmarks

You need both real and synthetic workloads, for different reasons. Real workloads establish that your result matters. Synthetic workloads let you sweep parameters and explain why your result happens. A thesis with only synthetic workloads is not convincing; one with only real workloads cannot explain itself.

Pick two or three real workloads and one or two synthetic generators. More than that and you cannot cover any of them properly.

Commonly used and defensible choices, depending on your target:

Storage and filesystem: fio for parameter sweeps, filebench for workload profiles, RocksDB with db_bench, YCSB over a key-value store.

Memory and CPU scheduling: a Linux kernel build for a realistic mixed workload, memcached or redis with a proper load generator for latency-sensitive behavior, SPEC CPU if your institution has a license, stress-ng for controlled pressure, PARSEC for parallel workloads.

LLM serving and AI systems: the benchmark scripts shipped with vLLM or SGLang, ShareGPT or similar request traces for realistic arrival patterns and length distributions, MLPerf Inference for standardized comparison, and Azure or Alibaba public traces for arrival processes.

Three things to get right regardless of choice. Justify each workload in writing — one sentence per workload in your methodology section explaining what it represents and why it is relevant; readers should never wonder why a benchmark is present. Include a workload where you expect to do badly, and report it, because finding your own weaknesses is what credibility is made of and a committee will find them anyway. And be careful about arrival processes: a closed-loop generator with fixed concurrency measures something fundamentally different from an open-loop generator at a target rate, and using a closed loop while claiming to characterize tail latency is a genuine methodological error. See guide/06-evaluation-playbook.md.

Metrics: define them before you measure

Write down exactly what you will measure, in what units, and how, before running anything. Deciding afterwards is how results shopping happens — not usually dishonestly, but by gradually gravitating toward whichever metric looks best.

Performance: throughput, and latency at p50, p99, and p999 rather than averaged. Report tails for anything user-facing. Goodput if requests can fail or miss deadlines.

Cost: CPU cycles, instructions, cache misses, memory footprint, energy if you can measure it credibly (RAPL for CPU, nvidia-smi sampling for GPU, with honest caveats about what these actually capture).

Fairness and isolation, if you are multi-tenant: per-tenant slowdown, and how bad the worst-off tenant gets.

For learned components specifically: prediction accuracy, but never alone — accuracy is a proxy and the whole risk of angle A is that the proxy and the outcome diverge. Also inference latency at the tail, model size in memory, training time and cost, and cold-start behavior before the model is useful.

Two rules. Never report a mean without a measure of spread. And be precise about the denominator when you normalize, because "2x faster" is ambiguous in a way that "reduces median latency from 40ms to 20ms" is not.

Overhead accounting

You must report the cost of your mechanism, not only its benefit. For learned-systems work this is often the difference between a thesis and a thesis that gets torn apart.

Account for all of it: inference latency in the critical path at the tail, memory for model weights and features, feature collection cost (frequently larger than inference itself and frequently forgotten), training and retraining cost, cold-start period, and cache pollution — a model that evicts useful data from L2 can cost more than it appears to in a microbenchmark.

Give this its own subsection in the evaluation chapter. Being forthright about your costs makes your benefits believable; burying them invites the reader to assume the worst.

Generalization, and the leakage trap

If your work involves training a model on system traces, read this section twice. It is where otherwise-good theses acquire fatal flaws.

Do not use a random train/test split on temporal data. System traces are time series with strong autocorrelation. Randomly splitting them puts adjacent, nearly identical samples on both sides of the split, and your model gets to interpolate rather than predict. The accuracy you measure will be badly inflated and the result will not survive scrutiny.

Split temporally instead: train on the earlier portion, test on the later. And split by workload as well — hold out entire workloads the model never saw during training. Report both. The workload-level split is the one that answers the question your thesis actually raises, which is whether a learned policy is useful on workloads it was not trained for. That is the entire premise of replacing a general-purpose heuristic, so it cannot be a footnote.

Also report what happens under distribution shift, when the workload changes mid-run, and what a stale model costs. "Our policy degrades to within 3% of the heuristic under shift" is a genuinely strong result — arguably stronger than a large improvement in the matched case, because it addresses the deployment concern a systems audience actually has.

Reproducibility

Two standards to hold yourself to, both of which are also self-interest rather than virtue.

One command per experiment. ./run.sh exp01-baseline produces a timestamped result directory with captured environment, raw output, and derived numbers. You will rerun everything more times than you expect — after a bug fix, after a reviewer question, after a kernel update — and manual multi-step procedures decay and drift.

Every figure regenerable from raw data by script. When you find an error in month nine, the fix should be one command, not an afternoon of remembering which spreadsheet produced figure 7. See experiments/README.md for the layout.

What to capture with every run, automatically: kernel version and command line, CPU model and microcode, frequency governor and turbo state, memory configuration and THP settings, all software versions, every random seed, the full configuration, and the git commit of your code and your scripts. experiments/scripts/capture-env.sh does this. A result whose environment you cannot reconstruct is not a result, and in a year you will have results from four different machine configurations that are otherwise indistinguishable.

Also: make raw results immutable and never overwrite them. Disk is cheap and a lost baseline is expensive.

Negative results

If your idea does not work, that is publishable, defensible thesis material — provided you did the work to explain why.

What distinguishes a good negative result from a failed project is the explanation and its generality. Not "our model did not help," but "learned prefetching cannot pay for itself at this integration point because feature collection costs 340ns against a 180ns baseline decision, and this bound follows from the cache-miss cost of touching per-task state rather than from our particular implementation." That reasoning generalizes, tells the field something, and is a real contribution.

To be able to write that, you need your Phase 1 measurement infrastructure to be good enough to explain failures, not merely detect them. Which is the argument for investing in Phase 1, one more time.