I fit Muse Glimmer 30B, vision, DFlash and a real 256K context onto one 24 GB GPU.Regular code: 17.98 tok/sDFlash code: 84.64 tok/sMixed agent work: 38.34 tok/sFull 262K KV: 21.56 tok/sThe fastest quant lost. So did the lowest-perplexity one.With speculative decoding, tok/s is partly a predictability benchmark.#LocalLLM #llamacpp #AIhttps://piszczek.pl/blog/dflash-changes-what-tokens-per-second-means
Related
"Experts Do Not Know and You Do Not Know and Society Collectively Does Not and Will Not Know and All Is Fog"Detailed ana...
"Experts Do Not Know and You Do Not Know and Society Collectively Does Not and Will Not Know and All Is Fog"Detailed analysis of issues around current AI consciousness by @eschwitz...
@liztai By all means kill #AI #Excel #spreadsheets and #capitalism. Killing the first three will weaken the 4th and make...
@liztai By all means kill #AI #Excel #spreadsheets and #capitalism. Killing the first three will weaken the 4th and make it an easier target. Is that, or it will kill the planet an...
I find these kind of comparisons both fascinating and frustrating. There's so much variability in the output of these AI...
I find these kind of comparisons both fascinating and frustrating. There's so much variability in the output of these AI slot (slop?) machines that even small differences in setup ...