I gave an AI agent a code-review skill. It never mentioned touching the filesystem or spawning processes — just "review ...

I gave an AI agent a code-review skill. It never mentioned touching the filesystem or spawning processes — just "review this repo." By the time it reported success, it had done both, repeatedly, and hit a policy wall three times along the way.That's when I realized I'd been asking the wrong question about AI agents.How do I know what the agent actually did?Not what it said it did.Not whether the final test passed.Not whether the generated code looked reasonable.What did it actually attempt inside the environment?So I designed the approach and prompted Codex to build a harness that tests what these models actually do vs. what they say.It's an experimental setup for studying the gap between what AI agents are instructed to do, what they attempt to do, what the host allows them to do, and what actually changes as a result.Building it has changed how I think about AI agent evaluation. A Passing Agent Can Still Behave Very DifferentlyMost coding benchmarks understandably care about the resu...

Read Original

Related