KV-Repurpose is introduced for evaluating non-prefix KV-cache reuse in large language model serving. It achieves a 63.9%...

KV-Repurpose is introduced for evaluating non-prefix KV-cache reuse in large language model serving. It achieves a 63.9% token-level cache hit rate under a Zipfian RAG-style workload, compared to 3.3% for prefix caching.Source: arXiv cs.CLhttps://arxiv.org/abs/2610.13060#MachineLearning

Read Original

Related