A pass rate without a cost axis is an advertisement, not a measurement, and that distinction decides whether a benchmark...

A pass rate without a cost axis is an advertisement, not a measurement, and that distinction decides whether a benchmark survives review. Teams comparing coding agents publish one percentage, then discover later that the cheapest configuration reached the same number for a fraction of the spend. The repair is rarely a stronger model; it is a harness that records what each solved task actually consumed.Scores compress a messy run into one digestible number, which is precisely why they mislead so easily. Two configurations can converge on identical pass rates while diverging by an order of magnitude in input tokens, retries, and wall-clock time. A reader who sees only the percentage cannot tell whether the result reflects capability or budget, and neither can the team that published it. Freeze the dataset before the first runEvery task should exist as a frozen file with a stable identifier, and the manifest should carry a hash of the prompt bytes so a silent edit cannot slip into a compa...