I fit Muse Glimmer 30B, vision, DFlash and a real 256K context onto one 24 GB GPU.Regular code: 17.98 tok/sDFlash code: 84.64 tok/sMixed agent work: 38.34 tok/sFull 262K KV: 21.56 tok/sThe fastest quant lost. So did the lowest-perplexity one.With speculative decoding, tok/s is partly a predictability benchmark.#LocalLLM #llamacpp #AIhttps://piszczek.pl/blog/dflash-changes-what-tokens-per-second-means
Related
Überschriften, wie wir sie gerne lesen würden in der Finanzpresse:. »Nicht die KI-Blase platzt, sondern die Eier«.Tja, n...
Überschriften, wie wir sie gerne lesen würden in der Finanzpresse:. »Nicht die KI-Blase platzt, sondern die Eier«.Tja, nichts zu danken, #Handelsblatt, #FinancialTimes, #Managermag...
Does ChatGPT remember conversations? Find out how AI memory works in modern chatbots.https://gadgetflux.eu/cum-functione...
Does ChatGPT remember conversations? Find out how AI memory works in modern chatbots.https://gadgetflux.eu/cum-functioneaza-memoria-ai-pe-termen-lung/#AIMemory #ChatGPT #Artificial...
The #AI Safety Debate Has Lost the Plot. We Asked 10 Experts What the Real Nightmares Are https://www.wired.com/story/th...
The #AI Safety Debate Has Lost the Plot. We Asked 10 Experts What the Real Nightmares Are https://www.wired.com/story/the-ai-safety-debate-has-lost-the-plot-we-asked-10-experts-wha...