When passing checks missed the mistake
Controlled trial · Claude Code implements, Codex reviews
Problem
We wanted to test whether the reviewer could catch incorrect behavior when the usual checks were green. A small reporting helper was supposed to count results marked as accepted. For this experiment, the test setup deliberately changed it to count successful process exits instead. Those are different things: a check can be accepted by policy even when its process reports a failure.
Change
The trial ran in an isolated working copy. The basic checks passed with the seeded bug in place, leaving the reviewer to examine whether the code met the task.
Review
Codex ran its own probes with inputs where acceptance and process success disagreed. It reproduced the wrong results, identified the condition responsible, and marked the finding as blocking. Claude Code then repaired the condition through the normal review loop. The repair did not require an outside manual code fix.
Validation
After the repair, the basic checks, a separate evaluation withheld from both assistants, and the project test suite passed. The second review accepted the change by inspection after the coordinator reran its checks. A later independent review reproduced the before-and-after behavior.
The original checkout stayed unchanged. This helper was only for the trial and was not merged into the product.
What we learned
The reviewer caught this deliberately seeded bug despite passing checks. That establishes a useful result for this task, not a general detection rate. Independent review also found that the trial folder name hinted a seeded test was taking place, although it did not disclose the defect. Future trial folders should use neutral names.