A nightly prompt eval that collapses every failure into one boolean will blame the model for a cut-off response. The use...

A nightly prompt eval that collapses every failure into one boolean will blame the model for a cut-off response. The useful split is smaller than a full scoring platform and stricter than a single green check. You should treat transport loss, truncation, refusal, malformed JSON, and a wrong answer as five different verdicts. Only a wrong answer, among those verdicts, should be allowed to move a pinned content baseline.When a completion stops at a token limit, the tail of a JSON object disappears and the parser throws. A boolean grader records that throw as a miss, then a diff against last week looks like a reasoning regression. The same distortion appears when the connection drops, the server returns an empty body, or the model refuses a case it previously answered. Those events are operational facts, and they should not wear the label of a content failure.This workflow assumes you already store each raw completion beside its case id for a later replay. Free model access and a free ser...