Just something to keep in mind when you're tuning your local LLM setup. The size of your context can really eat up your RAM!Small Models (~8B Parameters)Context size KV Ram used4K (Default) ~0.5 GB8K ~1.0 GB32K ~4.0 GB64K ~8.0 GB128K ~16.0 GBMedium Models (~14B to 32B Parameters)Context size KV Ram used4K ~0.7 GB8K ~1.5 GB32K ~5.8 GB64K ~11.5 GB#AI #LLM #LocalLLM #vibecoding #coding #softwaredevelopment
Related
Sometimes you catch yourself making the same choice in different situations, and it's weird how often it doesn't work ou...
Sometimes you catch yourself making the same choice in different situations, and it's weird how often it doesn't work out. Like there's a pattern you can't quite see. Your tarot re...
Put every #AI company and CEO in a trash compactor.For society.For the economy.For science.For us.It will have to be don...
Put every #AI company and CEO in a trash compactor.For society.For the economy.For science.For us.It will have to be done, might as well do it now. www.polygon.com/ps5-xbox-gam...#...
AI Has Broken Open Source by ForrestKnighthttps://www.youtube.com/watch?v=qazhWj8kYLo#ai #development #webdev
AI Has Broken Open Source by ForrestKnighthttps://www.youtube.com/watch?v=qazhWj8kYLo#ai #development #webdev