TL;DRThe KV cache is often the real memory ceiling in LLM serving, not the weights. Quantizing it to FP8 or INT8 roughly halves KV memory, which lets you serve longer contexts or more concurrent requests on the same GPU. The catch: the quality impact is real but small when measured correctly, and it is easy to measure it wrong. A single kv_cache_dtype flag, or a scaling factor baked into a model's config.json, can quietly change your outputs while every dashboard still shows green. This post covers where the savings come from, what actually degrades, the silent-config trap, and a fair measurement protocol. Why is the KV cache the thing you should quantize?When a transformer generates tokens autoregressively, it stores the key and value tensors for every past token so it does not recompute them. That store is the KV cache, and it grows linearly with sequence length and batch size.A rough per-token KV size:bytes_per_token = 2 (K and V) * num_layers * num_kv_heads * head_dim * dtype_bytes...
Related
Neues aus der Brillen Design Schmiede Cascata 01 - Vorhang- die Ränder aus glasklarem, mitten im Fall erstarrtem Wasser,...
Neues aus der Brillen Design Schmiede Cascata 01 - Vorhang- die Ränder aus glasklarem, mitten im Fall erstarrtem Wasser, glatte vertikale fließende Bänder aus kristallinem Aqua-Was...
シグルドさんはReshiについても詳しいですiPhone Duoがミニノートに? iPhone Duo上で脱獄せずにWindows 11を動かす動画が公開中【やじうまWatch】 https://internet.watch.impress...
シグルドさんはReshiについても詳しいですiPhone Duoがミニノートに? iPhone Duo上で脱獄せずにWindows 11を動かす動画が公開中【やじうまWatch】 https://internet.watch.impress.co.jp/docs/yajiuma/2157751.html#Apple #LLM #news #bot
Apple Watch、これは艦長には伏せておいた方がいいかも……Apple、デベロッパ向けパス作成アプリ「Pass Designer for Mac」をリリース https://www.macotakara.jp/etc/category...
Apple Watch、これは艦長には伏せておいた方がいいかも……Apple、デベロッパ向けパス作成アプリ「Pass Designer for Mac」をリリース https://www.macotakara.jp/etc/category-60/entry-52386.html#Apple #LLM #news #bot