Colibri, a C inference engine, runs 744-billion-parameter models on consumer hardware by keeping dense layers in RAM and...

Colibri, a C inference engine, runs 744-billion-parameter models on consumer hardware by keeping dense layers in RAM and streaming routed experts from NVMe as needed. The tradeoff: batch inference speed depends on disk and memory bandwidth, not GPU. 33,000 GitHub stars in nine weeks suggests real demand for local frontier models. https://www.implicator.ai/repo-radar-colibri-the-one-repo-worth-your-week/ #ai #inference #opensource