Your eval library calls the judge once per test case and prints a number. The judge flips its verdict on 5-15% of reruns. That number is noise wearing a suit.
Every LLM Eval Library Has the Same Bug: Stochastic Judges Used as Deterministic Oracles
Related
Ollama /api/embed Silently Dropped 45% of My Document
On a machine without a GPU, Ollama 0.40.14 loaded qwen3-embedding:0.6b with a 4,096-token context,...
The stop conditions I gave Claude Code, and the 8 times it used them in 31 minutes
I use Claude Code for a job that has nothing to do with my repo: submitting my products to launch...
The Claude Code Setup I Wish I Had on Day One
Six small files that turn a coding agent from a clever stranger into a teammate who knows how your...