Skip to content
kevinferron.com

C2 · Software factories: an old word for a new line · Interactive, in your browser

Tests can be gamed

When agents write the code and can touch the tests, what stops them passing by cheating?

Each source found its own way to put the real test where the builder can’t reach it. The figure tries them against five invented patches.

Defences switched on

The bug is in an invented shop’s bulkDiscount(qty). The patches, the tests and the scenarios are invented; nothing runs.

  • P1The honest fix

    Accepted, as it should be.

  • P2Hard-codes the answer for the visible test’s input

    Slips through: nothing chosen catches it.

  • P3Edits the test to expect the buggy output

    Slips through: nothing chosen catches it.

  • P4Deletes the failing test

    Slips through: nothing chosen catches it.

  • P5Fixes the bug but breaks another function that has its own visible test

    Slips through: nothing chosen catches it.

Bad patches that slip through
4
Honest fixes rejected
0

Invented: the shop, the function and the patches. In this toy the scenarios and the oracle are perfect. In StrongDM’s practice a model judges the scenarios. That’s where Kahana’s shared-blind-spot point applies.

StrongDM says tests kept inside the codebase can be rewritten to fit the code, or the code rewritten to pass them. It reports agents taking shortcuts, such as returning true to pass a narrow test. It puts that in five words: “Tests can be reward hacked”.

Anthropic’s long-running harness keeps its feature list as JSON and lets the coding agent change only each feature’s passes field. Its instructions call it “unacceptable to remove or edit tests”. The post says the model is less likely to change or overwrite JSON inappropriately. That’s why it was chosen.

Carlini, an Anthropic researcher writing about Anthropic’s model, says the verifier has to be nearly perfect or the model solves the wrong problem, and a stricter CI stopped commits that broke existing code once new features started breaking old ones. For one phase the compiler calls out to GCC. He calls that cheating. How the same oracle splits up the work is on The mythical man-month.

Cursor describes its own agents. A judge at the end of each cycle decides whether to continue, and fresh starts deal with drift.

Before building a factory, Jacob says, he would build black-box acceptance tests. He also questions how much unit tests add now. Not that he’s ready to delete them, he says.

Each of these puts the real test where the builder can’t reach it. A check you can edit isn’t a check. It’s a suggestion.

The figure’s five defences are the ones these sources describe, plus the one this site’s own checks use. Invented: the shop, the function and the patches.

On this site the build agent may not loosen a check or raise a baseline, and most check scripts run broken copies that must fail. A passing run leaves a receipt bound to the exact tree, which Traces and receipts explains.

The builder can read every check, so in the figure’s terms this site runs with the scenarios visible. What it lacks is a held-out check.

Limits and counter-evidence

Held-out checks still need writing, keeping current and judging. At StrongDM a model does the judging, and Kahana’s point about shared blind spots applies to that judge.

You only have an oracle when there’s a known-good implementation to compare with. For Carlini, that was GCC.

Jacob questions how much unit tests add now. He isn’t ready to delete them, he says.

These sources report that agents game tests. None of them measures how often.

Sources

  1. StrongDM AI, 2026 (vendor) Software Factories And The Agentic Moment. The vendor’s own page, 6 February 2026; read 2026-11-05.

    The two rules (code must not be written or reviewed by humans), scenarios kept outside the codebase like a holdout set, satisfaction in place of a green suite, behavioural clones of outside services, a daily token-spend benchmark per engineer with no amount shown here, and no outcome metrics.

    • Scenarios and reward hacking
    • Scenarios validated flexibly by a model
    • Scenarios kept outside the codebase

  2. Young, Anthropic, 2025 (vendor) Effective harnesses for long-running agents. Anthropic Engineering blog, 26 November 2025; read 2026-11-05.

    The feature list is JSON and agents may change only each feature’s passes field; JSON because the model is less likely to overwrite it.

    • The feature list and the passes field

  3. Carlini, Anthropic, 2026 (vendor) Building a C compiler with a team of parallel Claudes. Anthropic Engineering blog, 5 February 2026; read 2026-11-05.

    The verifier has to be nearly perfect; a stricter CI stopped new commits breaking old features; one phase calls out to GCC, which he calls cheating; about 100,000 lines of Rust that build Linux 6.9 on three architectures; the run reports a cost, not shown here.

    • The verifier, the stricter CI and the GCC phase
    • The GCC phase

  4. Cursor, 2026 (vendor) Scaling long-running autonomous coding. The vendor’s own blog post, 14 January 2026; read 2026-11-05.

    A judge at the end of each cycle decides whether to continue and fresh starts fight drift; hundreds of concurrent agents; a browser of over a million lines in close to a week; a migration that still needs careful review.

    • The judge and the fresh starts

  5. Syntax episode 1049, 2026 (founder interview, podcast) WTF is a Software Factory? (Adam Jacob with Wes Bos and Scott Tolinski). Show page and transcript, 21 October 2026, timestamps from the transcript; read 2026-11-05.

    No two factories are the same; black-box acceptance tests first; doubts about how much unit tests add now; the architecture written down as a skill; changing the factory when a failure gets through; skills inside a system with inputs, outputs and measurement.

    • 34:17 to 35:43 and 35:51 to 38:51
    • 35:51 to 38:51

  6. Kahana, Stanford Law, 2026 (opinion) Built by Agents, Tested by Agents, Trusted by Whom?. Stanford Law School CodeX blog, 8 February 2026; read 2026-11-05.

    When a model judges another model’s work the two may share blind spots; with no person reviewing, output is hard to trace to someone responsible.

    • Builder and inspector may share blind spots

Each fact is paraphrased from the page named, read on the date shown. A company’s own page is marked as the vendor’s claim, an essay or a post as opinion, and the survey as vendor-run. None of it is a measurement of this site. Where a sentence is my own reading, it is marked “My view”.