GitHub Trending repo Repositories Mar 28 10 views

Argonaut790/fused-turboquant: Fused Triton kernels for TurboQuant KV cache compression — 2-4 bit quantization with RHT rotation. Drop-in HuggingFace & vLLM integration. Up to 4.9x KV cache compression for Llama, Qwen, Mistral, and more.

by Argonaut790

Fused Triton kernels for TurboQuant KV cache compression — 2-4 bit quantization with RHT rotation. Drop-in HuggingFace & vLLM integration. Up to 4.9x KV cache compression for Llama, Qwen, Mistral, and more.

Read Original
API Hugging Face Mistral

Metadata

Stars
5
Language
Python
Watchers
5
Open Issues
4
License
Apache-2.0

Related