Proposal A — Local inference as a bad citizen of the memory hierarchy

Angle C with a small Angle A mechanism · Compute 0 · Kernel 2 · ML 0 · ~6 months full-time-equivalent · workshop to measurement-conference ceiling

Literature verified 26 August 2026. Re-verify before committing; see the shelf-life note at the end, which is unusually short for this topic.

This document is the Phase 0 deliverable described in guide/01-scoping.md — the one-page problem statement, expanded with the evidence you will need in your first advisor meeting. Compress it to one page before you send it.

---

The problem, and who has it

Run a local language model on your own machine and everything else on the machine gets worse. Two distinct mechanisms produce that experience, and the literature has looked at neither from the affected party's point of view.

Cache displacement. Loading a multi-gigabyte model reads that model through the page cache, evicting the resident working sets of whatever else is running — a browser's tabs, an IDE's index, a language server, a warm build cache. The eviction is invisible to the loader and expensive for everyone else, and the cost persists until each victim faults its working set back in.

Reclaim and victim selection. Push the same machine into memory pressure and Linux makes a choice that is difficult to defend. oom_badness() in mm/oom_kill.c scores a process as get_mm_rss_sum() + MM_SWAPENTS + mm_pgtables_bytes/PAGE_SIZE, and RSS includes file-backed mapped pages. For an mmap'd GGUF weight file those pages are clean, file-backed and read-only — reclaiming them costs one re-read from NVMe and loses no data at all — yet they are charged at exactly the same weight as anonymous heap. A llama-server holding seven gigabytes of weights resident on a sixteen-gigabyte laptop is therefore scored as the fattest process on the machine and selected first, despite its memory being the cheapest on the machine to reclaim without killing anything.

The population with this problem is now large and non-specialist: Ollama, llama.cpp, LM Studio and their derivatives on ordinary laptops and workstations. That matters for the "why should anyone care" question, which is the question a measurement thesis lives or dies on.

Why it is hard — why the obvious solutions fail

There are two obvious solutions, and you must dispose of both in writing before your advisor does it for you.

"Just use cgroups." Put memory.high on the loader, memory.low on the victim, POSIX_FADV_DONTNEED on the weight file once residency stabilizes. This is your strongest baseline and it is free, so run it in week two. The argument that survives it is not that these knobs are unset but that they are insufficient in a specific, demonstrable way: protecting the inference process merely relocates the kill to the user's browser. The defect is not which process the kernel chooses, it is that the scoring model has no notion of reclamation cost, so no assignment of priorities to processes can express "reclaim these pages instead of killing anything." Have that experiment ready for the first committee meeting.

"Just use --direct-io." As of llama.cpp PR #18166, Direct I/O is the default on Linux and Windows, and by construction it does not pollute the page cache. That closes the naive version of the cache-displacement story. Your contribution therefore lives in the regime where users deliberately want mmap — repeat loads, model switching, and the shared-page benefits of file-backed residency — and you must say so explicitly rather than measuring a default that no longer exists.

Why now

Three things changed inside the last twelve months, and together they are the "why now" paragraph.

Loading became a live design space with user-visible knobs. llama.cpp now ships four distinct modes — mmap, --no-mmap buffered reads, --direct-io (PR #18166, now the Linux default), and --hugepages (PR #21821) — plus a MADV_HUGEPAGE hint on the weight mapping (PR #22022, merged as dd17481). A cross-mode comparison is meaningful now in a way it was not in 2024.

The reclaim path itself is under active revision on grounds adjacent to yours. Matt Fleming's March 2026 work (LWN 1061060) documents production machines spinning in direct reclaim for twenty to thirty minutes without ever invoking the OOM killer, because should_reclaim_retry() reads zram's thin-provisioned free-slot count as reclaimable capacity. Chris Down's companion piece and the LWN discussion contain Shakeel Butt stating the MM community's position outright: the in-kernel OOM killer is deliberately conservative and aggressiveness is punted to userspace daemons. That is a quotable statement of the assumption you are attacking, and it is calibrated on datacenter fleets where workloads are restartable.

And the fix became expressible as a BPF program. Roman Gushchin's BPF OOM series (LWN 1056177, v3, 26 January 2026) adds bpf_handle_out_of_memory() as attachable struct_ops, system-wide or per-memcg, with bpf_oom_kill_process() and bpf_out_of_memory() kfuncs and a fallback to the in-kernel killer if nothing is freed. You can therefore write the policy without patching core MM, which is what brings this topic inside your skill envelope.

Thesis statement

By discounting clean, file-backed weight-mapping pages in OOM victim selection and gating weight residency on measured victim-side pressure, a BPF OOM policy reduces the completion-time regression of co-resident interactive workloads during local LLM inference on a memory-constrained single-user Linux desktop by more than 30% relative to correctly configured cgroup v2 confinement, at a cost of under 10% decode throughput.

Both numbers are placeholders until Phase 1 tells you the variance, and per guide/01-scoping.md you should expect to narrow rather than raise them. Note that the overhead bound is part of the claim, not a caveat.

Decomposed claims

Claim 3 is the kill experiment promoted to a claim, which is deliberate: it is the thing most likely to be false, so it belongs where you cannot avoid testing it.

The gap, in one paragraph

This is an unmeasured cost plus an unvalidated assumption, which per guide/01-scoping.md are the two stronger kinds of gap. The three closest works all measure the memory hierarchy from the model's point of view rather than the machine's. MAIO/PPC (Liu et al., USENIX FAST '26) builds a programmable page cache with a "Burn-after-Reading" eviction policy explicitly to stop model loading from occupying cache space — and evaluates it by its effect on loading latency, reporting no victim-side metric. Kim, Lee and Bahn (Mathematics 13(22):3689, 2025) decompose mobile LLM file access into one-time init scans, persistent hot sets and looped weight accesses, and conclude that one-time scans "should be evicted quickly to avoid polluting the cache" — again optimizing the loader's objective. Libra / LLMS (Yin, Xu, Li, Liu, SenSys 2026, pp. 377–391) is the closest and the most dangerous: it argues that Android's low-memory killer treats LLM context memory as ordinary app memory and that a lightweight app holding an active LLM context is easily killed. You must distinguish yourself from Libra in one sentence, and the sentence is that Libra addresses KV-cache context on Android under LMKD whereas this addresses weight residency on Linux under oom_badness(), with reclamation cost rather than recomputation cost as the mis-priced quantity. If you cannot say that crisply, you do not have a thesis.

One awkward citation to handle deliberately: STAP (Zenodo, January 2026) states your premise almost verbatim — that kernel memory management "remains demand-agnostic, treating high-priority model weights and volatile KV caches as uniform anonymous pages" — and claims a 7.5× P99 improvement. It is unrefereed, uncited, and its authors have no publication record, so its numbers are not trustworthy. But a reviewer may find it and ask you to differentiate against work you cannot evaluate. Cite it, state its status plainly, and note that no reproducible evidence accompanies the claim. That is the honest move and it inoculates you.

How you measure it

Victim-side, which is the whole point: pgmajfault per cgroup, completion-time regression on a fixed interactive workload suite, page-cache residency via mincore()/fincore maps over a declared victim file set, and residency recovery time after the load finishes. Inference-side: decode throughput and TTFT, so the trade-off is a Pareto frontier rather than a single number. Reclaim-side: /proc/<pid>/oom_score alongside smaps_rollup for the weight mapping, direct reclaim stall duration, thrashing rate, and which process the kernel actually selected.

Pick victim workloads with published, stable baselines — a cold cargo or tsc build, git status on a large repository, an SQLite query set — because "your victim workload is a microbenchmark" is otherwise the easiest available criticism.

What you build on

A bare-metal Linux install, 16 GB deliberately tight, NVMe, with a pinned kernel carrying the BPF OOM series. The bare-metal requirement is not negotiable and is the single most common way this project fails before it starts: the linux-inference-memory-hints repository — someone else's attempt at adjacent work — documents that WSL2 will validate your scripts while producing no reclaim signal whatsoever. Budget a scratch machine you are willing to hard-lock, and budget for repeated hard reboots.

Everything else is eBPF, /proc, /sys, cgroup v2, and llama.cpp with its shipping flags. No GPU is required, which given your cluster's queue times is a feature.

Fallback

The characterization stands alone. If the BPF policy cannot beat well-configured cgroups, the thesis becomes a pressure-ladder characterization of local inference across swap configurations (none, disk, zswap, zram, zram plus systemd-oomd) and the four loading modes, with the oom_badness() mis-scoring documented and the negative result on policy stated honestly. That is a legitimate Angle C thesis and it is available from week eight onward, which is exactly the property guide/01-scoping.md asks a fallback to have.

What must work by week eight

A bare-metal test machine with a pinned, stated kernel version. A one-command victim workload suite with characterized run-to-run variance. A reproducible memory-pressure ladder. And the cgroup v2 baseline result in hand — because if that result kills the thesis, week eight is when you want to know.

How it goes wrong

Shelf life, and it is short. BPF OOM was at v3 in January 2026 and Fleming's RAM-backed swap fix was in flight in March. If both land, part of your pathology disappears and part of your mechanism becomes "write a BPF program." Mitigate exactly as guide/01-scoping.md prescribes: pin and state your kernel version, and make the contribution the characterization plus the policy insight rather than the mechanism. Treat roughly a one-year window as the planning assumption.

The oom_score_adj objection. "Just set oom_score_adj, or run it under systemd with MemoryHigh=, or use systemd-oomd." Answered by claim 3, if and only if you actually run it.

The framing objection. Rejection vocabulary to expect: "this is a cgroups configuration problem," "MGLRU already handles this," "Burn-after-Reading solved it," "no novelty over Kim et al.," "userspace OOM policy is the accepted design." Each has an answer; none of them has an answer you can improvise.

Why this one, for you

All four of your constraints point the same way, which is not true of any other option in ../01-candidate-topics.md or ../04-new-candidates.md. eBPF is both the instrument and the mechanism rather than a convenience. The topic is core memory management, so an OS/kernel advisor is the right supervisor and there is no risk of it reading as software engineering. A single laptop is the correct platform rather than a compromise, which matters because twelve part-time months cannot absorb cluster queue time. And it is measurement-first, so the thesis exists even if the mechanism disappoints.

The cost you accept consciously: your cluster becomes irrelevant. Every topic that would use it needs exclusive, uncontended GPU time for tail-latency work, and shared-with-queue-time cannot supply that. Decide that trade deliberately rather than discovering it in month four.

Venues

Realistic targets given roughly six full-time-equivalent months: HotStorage, EuroMLSys, APSys, the eBPF workshop at SIGCOMM, Linux Plumbers Conference, IISWC, ISPASS. An LKML RFC of the BPF OOM policy is a legitimate and unusually cheap secondary output, and for a topic with a one-year shelf life, planting that flag early is worth more than it looks.