KV-Cache Offload to NVMe vs Recompute: The TTFT Bill Comes Due

Serving a long-context model is a memory problem disguised as a compute problem. Offloading the cache to NVMe and recomputing the prefill are both defensible, and they fail in completely different way

Read Original

Related