Your eval library calls the judge once per test case and prints a number. The judge flips its verdict on 5-15% of reruns. That number is noise wearing a suit.
Every LLM Eval Library Has the Same Bug: Stochastic Judges Used as Deterministic Oracles
Related
The Reasoning Ledger: Remembering Decisions, Not Just Data
Part 4 of the Building the AI Memory Stack series After finishing the previous article, I looked at...
[Gemini API in Action] Adding a "Detailed Research Report" Button to a LINE Bot: Using Google Search Grounding to Turn Summaries
Background My LINE Bot has always had a summary feature: you drop a URL in, it crawls...
We caught a SQL injection with an offline AI security scanner — here's the exact query it found
The bug Here's a login endpoint from a small Express demo...