Failure Modes

How theses in this area actually go wrong. Each one below has an early warning sign, because all of these are cheap to fix in month two and expensive or impossible to fix in month nine.

Reread this at the week-24 mid-point review. Several of these are only visible from a distance.

1. The infrastructure sinkhole

You spend five months building a testbed, a harness, and tooling. It is real work, it feels productive, and it produces no thesis. Systems infrastructure has unlimited appetite for effort, and each individual piece of it is genuinely necessary.

Early warning: no baseline number by week 8. Or you are writing a benchmark framework rather than using one.

The fix: treat the Phase 1 gate as a hard deadline. Use existing tools even where they are imperfect — fio and filebench and the vLLM benchmark scripts exist and you should not be writing their equivalents. When you catch yourself building general infrastructure, build the specific thing your experiment needs instead.

2. Scope drift

The thesis statement quietly changes, three or four times, each change locally reasonable. Nine months in, you have interesting fragments of four projects and a defensible version of none.

Early warning: your thesis statement has changed materially more than twice, or you cannot say in one sentence what you are claiming.

The fix: the one-idea rule and the cut list, from guide/01-scoping.md. Reread your one-page problem statement monthly. Narrowing a claim is good; changing it is expensive, and the distinction is worth being honest with yourself about.

3. The strawman baseline

You compare against an untuned default and report a large improvement. Then someone spends twenty minutes tuning the baseline and most of your improvement disappears.

This is the most common fatal flaw in student systems evaluations, and the most likely to be caught in a defense.

Early warning: you cannot say what you tuned in your baseline, or your improvement is suspiciously large — a 10x improvement over a mature production system usually means you are measuring something other than what you think.

The fix: tune your baselines deliberately and document the tuning in your methodology section. Spend real effort making your competition look good. It is the only way your result becomes believable.

4. Overhead denial

Your model has excellent offline accuracy. You report it. You do not report that inference costs 2µs in a path where the heuristic it replaces costs 200ns, so the complete system is slower.

This is the characteristic failure of angle A and it is fatal, because the entire premise collapses.

Early warning: you have accuracy numbers but no end-to-end system numbers. Any version of the thought "I'll optimize the inference cost later."

The fix: measure end-to-end from the first week you have anything working, before you invest in model quality. If the overhead cannot be paid for, you need to know in month three, when you can still change the integration point, shrink the model, or pivot to angle C.

5. Results shopping

You run many experiments, and the ones that look good become the thesis. Rarely deliberate — it happens through the accumulated small decisions of which configuration to explore further and which metric to feature.

Early warning: your metric choice or workload set changed after you saw results.

The fix: the claims-to-experiments matrix, with the falsification column filled in before running anything. Deciding in advance what counts as success is the whole defense against this. Also report the experiments that did not favor you — it costs you a paragraph and buys you credibility for the entire chapter.

6. Reading forever

Month five, still reading. You know the literature very well and have written nothing and built nothing. Reading always feels responsible, which is precisely what makes it dangerous.

Early warning: no written thesis text by month three. Or you are reading papers only tangentially related to your claim.

The fix: the reading budget and the week-20 freeze in guide/02-reading-system.md. Draft your related-work chapter in weeks 8 to 12 — the deadline forces convergence, and you cannot draft a chapter and keep reading indefinitely at the same time.

7. The single-run graph

Every bar in your evaluation is one run. Any reviewer sees this immediately, and it invalidates the chapter, because without variance nobody can tell whether your 8% improvement is real.

Early warning: you have no repetition loop in your experiment scripts.

The fix: build repetition into the harness in Phase 1, so single runs are not the path of least resistance. Minimum five runs, report median and IQR.

8. Data leakage in temporal traces

You randomly split a system trace into train and test. Because system traces are strongly autocorrelated, near-duplicate samples land on both sides, your reported accuracy is badly inflated, and the result does not survive scrutiny.

This one is particularly dangerous because it is invisible in your own results — everything looks excellent. It is discovered by a reviewer or examiner, and it can invalidate your central claim at the latest possible moment.

Early warning: the words train_test_split(shuffle=True) anywhere near trace data. Accuracy above 95% on a hard prediction problem.

The fix: split temporally, and additionally hold out entire workloads. See guide/03-research-design.md.

9. Going dark on your advisor

Six weeks of no contact because things are going badly and you have nothing to report. Silence is read as trouble, it removes the one person able to help, and it means a course correction that could have happened in week twelve happens in week thirty.

Early warning: you are dreading the next meeting. That feeling is the signal.

The fix: the weekly written update, sent whether or not there is progress. "This week went badly, here is what I tried, here is what I am stuck on" is a completely acceptable update and frequently produces the most useful meeting you will have.

10. The lab notebook thesis

The document narrates what you did in chronological order — first I tried this, then that failed, then I tried something else. It reads as a diary. Readers cannot extract the claim, because there isn't one; there is a history.

Early warning: your chapter or section headings are phases of your project rather than components of an argument.

The fix: structure by argument, per guide/04-thesis-structure.md. Your false starts belong in the design chapter as rejected alternatives, which is a much stronger framing — the same information, presented as evidence that you understood the design space.

11. Lost provenance

Month nine, a reviewer questions figure 7, and you cannot regenerate it. You are not sure which code version produced it, or which machine, or whether THP was on.

Early warning: results in ad-hoc directories, numbers transcribed by hand into a spreadsheet, any figure produced by manual steps.

The fix: the layout and environment capture in experiments/README.md, from Phase 1. Every figure regenerable with one command from immutable raw data.

12. No fallback

Your idea does not work. It is month eight. You have no alternative plan and you keep pushing because stopping feels like failure.

Early warning: you have not written down what you submit if the main idea fails.

The fix: design the fallback in week four, per guide/01-scoping.md. It usually routes through angle C — a rigorous characterization of why the approach does not work, which is a legitimate thesis if your measurement infrastructure is good enough to explain the failure rather than merely observe it.

13. Administrative ambush

Week thirty-nine: the template requires a declaration page you do not have, the committee needed the document three weeks before the defense, or the scheduling deadline passed a month ago. Weeks lost to paperwork, and they are the weeks you least have to spare.

Early warning: you have not read your program's formal requirements.

The fix: month one, from the README. All of it is trivial in month one.

14. The unexplained result

Your numbers are good and you cannot say why. The evaluation reports deltas with no mechanism. A committee will not accept this, and reasonably so — an unexplained improvement might be an artifact, and you have given the reader no way to rule that out.

Early warning: you have no performance-counter or profile data, only end-to-end timings.

The fix: attribution, per guide/06-evaluation-playbook.md. Every headline result needs a mechanism, supported by measurement at the layer below.

15. Perfectionism at the endgame

Week thirty-eight, rewriting chapter four for the fifth time, having not started the defense slides. The last 5% of polish costs as much as the first 80% and adds almost nothing to your outcome.

Early warning: revising text that is already clear while a required section is still missing.

The fix: separate revision passes with time limits, per guide/05-writing-process.md. A complete good thesis beats an incomplete excellent one, without exception, because an incomplete thesis does not pass.

The single best predictor

Across all of these, the students who finish well have one habit in common: they produce something written and something measured every single week, from week one. Not a lot — a paper note, a section outline, one experiment run and logged.

The failure modes above are almost all versions of the same underlying error, which is deferring the production of artifacts in favor of activity that feels like progress. Reading, building infrastructure, and refactoring all feel productive and can each absorb a year. Written pages and logged results are the only things that accumulate into a thesis.