New post: Laya vs Jev, ten days later 🧠⚖️Ten days ago I benchmarked two "System One" decision models for my LLM router: ...

New post: Laya vs Jev, ten days later 🧠⚖️Ten days ago I benchmarked two "System One" decision models for my LLM router: Jev (TypeSafe, hosted) was usable as-is, Laya (open weights) zero-shot was not. Laya's README says "fine-tune me", so I did, on a 16 GB M4 MacBook.🔁 Laya 0.3.6 → 0.3.24: 18 releases, but the weights are byte-identical. Same numbers.🆕 New open rivals on my 80-prompt bench:• Laya typed-decisions: 71% topic, but never confident → over-provisions everything• Von 1.3: closest to gold out of the box (48%), but over-confident• Kev-0.8B: 81% topic, 100% multilingual, very under-confidentJev 1.13: 89% topic, 72.5% same route as gold, still the only one usable as-is.🎯 Fine-tuning Laya on my own labels (not Jev's: distilling Jev is against TypeSafe's terms):• 480 labelled prompts → 1,920 training items• official Apple Silicon script, MPS• 41.5 min for 4 epochs, 15.4 GB peak memory• rule of thumb: ~6 s per labelled promptResults:• topic 59% → 85%• complexity 45% → 72.5% (= Jev)• ...

Read Original

Related