Topic selectiontopic-selection/reports/memory-and-cold-start

Report: Tiered Memory and Cold Start

Produced 26 August 2026 by a literature-check agent instructed toward skeptical calibration rather than encouragement, and required to verify every citation in search results rather than recall it. Reproduced verbatim.

Topics checked: tiered memory (CXL / multi-socket NUMA) placement for model weights and activations; eliminating cold starts for serverless inference via snapshot/restore.

TOPIC A: Tiered memory (CXL / multi-socket NUMA) placement for model weights and activations

1. CROWDEDNESS: 3. The specific insight your framing rests on — that LLM weight/KV access is deterministic enough for proactive placement to beat reactive kernel tiering — is already the stated headline contribution of at least two 2026 papers, and OS-level tiering itself has a decade of entrenched baselines.

2. STRONGEST / MOST RECENT WORKS

PACT: A Criticality-First Design for Tiered Memory, ASPLOS 2026 — https://huaicheng.github.io/p/asplos26-pact.pdf — establishes that access frequency is the wrong signal; per-page stall criticality gives up to 61% over the best of 7 tiering systems with 50× fewer migrations.
ITME: Inference Tiered Memory Expansion with Disaggregated CXL-Hybrid Memories, arXiv 2606.12556 (2026) — https://arxiv.org/html/2606.12556 — explicitly claims weight/prefix-KV determinism as its key insight and uses it for pipelined multi-tier DMA prefetch; validated on SK hynix CMM plus an FPGA prototype.
HybridGen: Efficient LLM Generative Inference via CPU-GPU Hybrid Computing, arXiv 2604.18529 (2026) — https://arxiv.org/abs/2604.18529 — semantic-aware K-in-DRAM / V-in-CXL placement beats DRAM–CXL page interleaving, with the gap widening as models grow.
Memtis, SOSP 2023 — https://cosmoss-jigu.github.io/pages/pubs/memtis-lee-sosp23.pdf — and TPP, ASPLOS 2023 — https://arxiv.org/abs/2206.02878 — the mandatory baselines; TPP is in mainline Linux since 5.18, so "vs. default Linux" is already a weak comparison.
Fisher, Analysis of Memory Access Patterns for Large Language Model Inference, MS thesis, Virginia Tech, 2025 — https://vtechworks.lib.vt.edu/bitstreams/64d13a5f-4f89-4756-86bc-4cb90dad570f/download — profiles llama.cpp access patterns using a remote NUMA node as the CXL proxy and compares bound-local / bound-remote / default Linux.

3. IS THE MVP ALREADY DONE? Yes. Fisher's thesis is your MVP, same tool, same emulation, same baseline — completed and defended in April 2025 under Nikolopoulos, Back, and Huaicheng Li. The semantic-placement-beats-reactive-tiering result is ITME and HybridGen.

4. SURVIVING GAP. Narrow but real: for MoE models, whose per-token expert routing is data-dependent rather than deterministic, does offline layer-level profiling still beat PACT-style online criticality tiering — and at what fast-tier capacity ratio and co-tenant contention level does it stop winning? This is a measurement study that tolerates a negative result, which makes it survivable in 9–12 months.

5. HARDWARE BAR. Split, and this matters for you. Kernel/OS tiering papers still emulate: PACT at ASPLOS 2026 used a remote NUMA node with reduced uncore frequency to hit 190ns. But LLM-tiering and architecture papers now use real silicon — SK hynix CMM (ITME), Micron CZ120, a Niagara 2.0 CXL Type-3 expander (TraCT, arXiv 2512.18194), CXL switches (Beluga, arXiv 2511.20172). Demystifying CXL Memory with Genuine CXL-Ready Systems and Devices (https://tianyin.github.io/pub/cxl-mem.pdf) states emulation differences are large enough to compel revisiting prior conclusions, and reviewers will cite it at you. Emulation is still accepted, but only with explicit justification and a citation to that limitation.

TOPIC B: Eliminating cold starts for serverless inference via snapshot/restore

1. CROWDEDNESS: 3, effectively closed. Your exact target number — sub-100ms restore of a loaded model, with snapshot storage cost addressed by deduplication — was published in 2024, and the underlying mechanism has since become vendor tooling shipped by NVIDIA and three commercial platforms.

2. STRONGEST / MOST RECENT WORKS

gCROP: On-demand and Parallel Checkpoint/Restore for GPU Applications, SoCC 2024 — https://dl.acm.org/doi/10.1145/3698038.3698510 — the first GPU runtime under 100ms startup for a 3.1GB GPT-2-Large, via parallel restore, profile-guided on-demand page fault ordering, and multi-checkpoint deduplication to cut storage cost. 6.4–24.7× over cold boot, 3.9–23.5× over CRIU.
CRIUgpu: Transparent Checkpointing of GPU-Accelerated Workloads, arXiv 2502.16631 (Feb 2025) — https://arxiv.org/abs/2502.16631 — unified CPU+GPU snapshots without API interception, on H100/A100/V100/A6000 and multi-GPU; merged upstream and released in CRIU 4.0. I could not confirm a conference venue for this one.
Medusa: Accelerating Serverless LLM Inference with Materialization, ASPLOS 2025 — https://minhui-xie.github.io/papers/asplos25-medusa.pdf — materializes CUDA graphs and KV-cache init offline across 10 models (139,364 graph nodes): 42.5% lower load latency, 53% lower TTFT tail. Single-GPU only, on 4×A100-40GB.
ServerlessLLM, OSDI 2024 — https://www.usenix.org/conference/osdi24/presentation/fu — the loading-path baseline, 10–200× over prior serverless systems.
HydraServe, NSDI 2026 (arXiv 2502.15524, formerly ParaServe) — https://www.usenix.org/conference/nsdi26/presentation/lou — cold start via parallel pipelined model fetch. I verified the arXiv version directly; the NSDI page I have only from an aggregated reading list.
CRIU-LZ4, EuroMLSys 2026 — https://radostin.io/files/stoyanov-euromlsys-2026.pdf — on-the-fly page compression on the CRIUgpu restore path: 46–59% lower cold start, up to 6× smaller snapshots.

3. IS THE MVP ALREADY DONE? Yes, completely. gCROP (Yang, Du, Song, Xia — SJTU IPADS) did the latency target, the time breakdown, and the snapshot storage-cost question. In production, NVIDIA Dynamo Snapshot (https://developer.nvidia.com/blog/nvidia-dynamo-snapshot-fast-startup-for-inference-workloads-on-kubernetes/) reports 21× startup reduction on gpt-oss-120b, and Modal and Cerebrium both ship GPU memory snapshots commercially.

4. SURVIVING GAP. One, and it is contested: multi-GPU. cuda-checkpoint cannot handle IPC memory from cuMemExportToShareableHandle(), Dynamo Snapshot's release supports single-GPU only, and Medusa's own discussion section lists multi-GPU as future work. The question a student could own: can a tensor-parallel vLLM worker group be snapshot and restored coherently — how are NCCL communicators and IPC-shared KV buffers reconstructed, and how does restore latency scale with TP degree? Blunt caveat: NVIDIA is actively working on exactly this, so a 9–12 month window may close under you.

5. HARDWARE BAR. Low on GPUs, high on storage. A single A100-40GB or H100 clears the published bar (Medusa used 4×A100 but evaluated single-GPU; gCROP evaluated on AMD GPUs). The real requirement is fast local NVMe — restore time is checkpoint size divided by storage bandwidth, so shared NFS or S3 destroys the result. Driver 550+ is required for cuda-checkpoint and 580+ for restoring onto a different physical GPU. GPU state checkpointing is now solved by vendor tooling: NVIDIA's cuda-checkpoint plus CRIU 4.0 plus Dynamo Snapshot. A thesis that re-implements the mechanism has no contribution left.