I'm pretty certain I'll remove the stability-matrix from the setup as I primarily used it for research/learning (Note: I wanted to see how it works and what parts were required, how they fit together....)Now it's a resource hog for a piece of tech I don't need. There is enough open-source resources available and frankly am a little uncomfortable running anyway?I also need to refine the specific model param sizes and quant configuration across my cluster, because right now some tasks are being done by models 10x the size required for that level of complexity.Just downscaling the embedder to nomic from qwens already efficient one was a 90% VRAM usage cut for that task 🙃#AI #LLM #tech #selfhost #opensource #openweight
Related
📈 Olud Index at 138. Stars are running 2.14x normal — the AI ecosystem is building far above baseline pace. See what's d...
📈 Olud Index at 138. Stars are running 2.14x normal — the AI ecosystem is building far above baseline pace. See what's driving that surge on olud.ai.https://olud.ai/dashboard.html#...
Rui Ma of TechBuzz China explains why China's AI labs aren’t chasing export markets – they’re fighting a brutal domestic...
Rui Ma of TechBuzz China explains why China's AI labs aren’t chasing export markets – they’re fighting a brutal domestic price war. That competition may be lowering global AI costs...
Firm embroiled in Nvidia scandal linked to Australian newsagent workerBy Jasper WellsThe Australian subsidiary of the fi...
Firm embroiled in Nvidia scandal linked to Australian newsagent workerBy Jasper WellsThe Australian subsidiary of the firm at the centre of the alleged Nvidia spy scandal lists a n...