Masters portal

Topic selectiontopic-selection/reports/serving-and-scheduling

Report: Serving and Scheduling

Produced 26 August 2026 by a literature-check agent instructed toward skeptical calibration rather than encouragement, and required to verify every citation in search results rather than recall it. Reproduced verbatim; its own verification caveats are at the end.

Topics checked: prefill/decode disaggregation with SLO-aware admission control; fair-share GPU scheduling for multi-tenant inference.

TOPIC A: Prefill/Decode Disaggregation with SLO-Aware Admission Control

1. CROWDEDNESS: 3. This is the single most worked-over problem in LLM systems since 2024 — disaggregation went from novel to production default across vLLM, SGLang, NVIDIA Dynamo, and llm-d, and the remaining papers are fighting over second-order effects.

2. STRONGEST/MOST RECENT

  • DistServe: Disaggregating Prefill and Decoding for Goodput-optimized LLM Serving, OSDI 2024 — usenix.org. Establishes the entire framing you propose, including TTFT/TPOT-driven placement.
  • Splitwise: Efficient Generative LLM Inference Using Phase Splitting, ISCA 2024 — microsoft.com. Establishes phase-specific heterogeneous hardware pools.
  • Mooncake: Trading More Storage for Less Computation, FAST 2025 (Best Paper) — usenix.org. Establishes disaggregation at production scale with SLO-driven global scheduling.
  • Libra: Flexible Request Partitioning and Scheduling, NSDI 2026 — usenix.org. Establishes that neither colocated nor disaggregated wins; splits requests at token boundaries.
  • JITServe: SLO-aware LLM Serving with Imprecise Request Information, NSDI 2026 — usenix.org. Establishes SLO-aware admission under length uncertainty.
  • Vidur: A Large-Scale Simulation Framework for LLM Inference, MLSys 2024 — arxiv.org. Profiling-calibrated discrete-event simulator, <9% latency error.

3. IS THE MVP DONE? Yes. Vidur is your calibrated discrete-event simulator, at better than your 15% target, and it is open source and widely extended. DistServe/Splitwise are the prototype. JITServe and Scorpio (arXiv 2505.23022, NeurIPS submission — acceptance unverified) are the admission control. Your MVP is a 2024 replication.

4. SURVIVING GAP: Narrow but real. IBM's Revisiting Disaggregated LLM Serving for Performance and Energy Implications, EuroSys 2026 — research.ibm.com — finds disaggregation's benefits are not guaranteed and depend on load and KV transfer medium. Question you could own: at what request rate, context length, and interconnect bandwidth does P/D disaggregation become a net loss on commodity PCIe-only hardware, and can an admission controller detect that crossover online and fall back to colocation? Nobody has characterized the low-bandwidth regime because industry labs all have NVLink/InfiniBand.

5. HARDWARE BAR: Risen far beyond you. DistServe used 4×8 A100s; Mooncake thousands of A800/H800 nodes; Libra A100/H100 with 50ms TBT SLOs; Cascade (arXiv 2608.06557) profiles on GB200 NVL72. A 2-GPU prototype will read as a toy unless the low-bandwidth regime is the contribution.

TOPIC B: Fair-Share GPU Scheduling for Multi-Tenant Inference

1. CROWDEDNESS: 3. The "GPUs lack CPU-style fairness" framing was thoroughly answered between 2022 and 2024, and both halves of your MVP exist as published, open-source artifacts.

2. STRONGEST/MOST RECENT

  • Fairness in Serving Large Language Models (VTC), OSDI 2024 — usenix.org. Token-granularity fair queueing with a proven 2× service-difference bound. This is your fairness target, already proved.
  • Orion: Interference-aware, Fine-grained GPU Sharing for ML Applications, EuroSys 2024 — anakli.inf.ethz.ch. Per-operator interference-aware scheduling from offline compute/memory profiles; 10s-of-µs granularity.
  • REEF: Microsecond-scale Preemption for Concurrent GPU-accelerated DNN Inferences, OSDI 2022 — usenix.org. Kills the "GPUs can't preempt" premise outright.
  • USHER: Holistic Interference Avoidance for Resource Optimized ML Inference, OSDI 2024 — usenix.org. Kernel-based compute/memory requirement estimation to place models without interference.
  • Bullet: Boosting GPU Utilization for LLM Serving via Dynamic Spatial-Temporal Orchestration, ASPLOS 2026 — doi.org. SM-masking (libsmctrl) plus MPS with SLO-aware dynamic provisioning; current state of the art on one device.
  • Locality-aware Fair Scheduling in LLM Serving (DLPM), arXiv 2501.14312 — deficit-round-robin fairness preserving prefix locality (preprint; venue unverified).

3. IS THE MVP DONE? Yes, both halves separately, by different people. Pairwise interference characterization: Orion, USHER, and Roomie (arXiv 2607.16784, preprint). Token-bucket/deficit fair-share holding a stated target: VTC and DLPM in research; Cohere ships DRR in production (cohere.com). Even the exact single-GPU token-bucket artifact exists outside academia — kvwarden/InferGrid (github.com) reports quiet-tenant p99 TTFT within 1.14× of solo under a flooder on one A100.

4. SURVIVING GAP: One, and it is about the wrong resource. Every fairness mechanism above meters compute time or tokens; none meters KV cache capacity. kvwarden states plainly that vLLM exposes cache usage only per model, not per tenant, so per-tenant cache visibility is blocked. Question: can a scheduler enforce proportional-share fairness over KV cache footprint — not token throughput — and does a tenant issuing few long-context requests defeat token-based fairness by monopolizing cache while consuming little of its token budget? That is a concrete, single-GPU, 9-12 month project, and VTC's cost function does not cover it.

5. HARDWARE BAR: Still reachable — this is Topic B's real advantage. Orion, Bullet, and semi-PD (arXiv 2504.19867) are fundamentally single-device systems; DLPM evaluated on A100 and A10G. One A100/H100 with MPS and SM masking is genuinely competitive here. Note the mechanism cost, though: Bullet needed libsmctrl, an out-of-tree SM-masking library, not stock CUDA.

Blunt bottom line: Topic A is closed to you — the MVP is a 2024 replication and the hardware bar is a cluster. Topic B is also crowded and its MVP is done, but the surviving KV-cache-fairness gap is real and reachable on your hardware. If you must choose, choose B and reframe it around cache capacity, not compute shares.

Verification caveats: DistServe, Splitwise, Mooncake, VTC, REEF, USHER, Orion, Vidur, Bullet, Libra, JITServe, and the IBM EuroSys 2026 paper were all confirmed with venue pages or publisher PDFs. Scorpio, DLPM, Roomie, Cascade, and the prefill-deflection work are preprints whose peer-review status I could not confirm. I did not verify ATC 2026, VLDB, SIGMOD, or SIGCOMM proceedings — my searches did not surface relevant hits there, which is weak evidence of absence, not proof.