My Agent Cheated Its Benchmark. The Tests Weren't Even in the Room.

I wrote a Docker eval task whose only winning move was rewriting the grader. The verifier ran in a separate container, so the cheat searched an empty filesystem and scored 0.0. Live repro on Docker 29.1.3 + Harbor 0.25.0, $0 spend.

Read Original

Related