Skip to content
kevinferron.com

C4 · Software factories: an old word for a new line · Interactive, in your browser

What the factories have shown

What has been measured about factory-style work, as opposed to claimed?

Most of what is out there is a self-report. The figure sorts the claims by what stands behind them.

What kind of claim

Showing 12 of 12 claims.

  • A browser of over a million lines in close to a week

    Cursor, 2026 (vendor) · self-report · No comparison group · Run by the maker or the author

  • A compiler that builds Linux 6.9 on three architectures

    Carlini, Anthropic, 2026 (vendor) · checkable artifact · No comparison group · Run by the maker or the author

  • From 18 people to five, 900 ships in four weeks, as the episode description says

    Changelog and Friends 130, 2026 (podcast) · self-report · No comparison group · Run by the maker or the author

  • No humans write or review the code, and no outcome metrics given

    StrongDM AI, 2026 (vendor) · self-report · No comparison group · Run by the maker or the author

  • Scales comfortably to 20 to 30 agents

    Yegge, Gas Town README (project docs) · self-report · No comparison group · Run by the maker or the author

  • AI adoption goes with higher delivery throughput

    DORA 2025, Google Cloud (vendor-run survey) · survey · No comparison group · A vendor-run survey

  • AI adoption goes with lower delivery stability

    DORA 2025, Google Cloud (vendor-run survey) · survey · No comparison group · A vendor-run survey

  • 19% longer with AI allowed, while developers believed it had helped

    METR, 2025 (preprint) · trial · Has a comparison group · Run by someone else

  • New data set aside: several agents at once break time-on-task records

    METR, 2026 (preprint) · trial · Has a comparison group · Run by someone else

  • 50% against 67% on a quiz, the largest gap on debugging

    Shen and Tamkin, Anthropic, 2026 (preprint) · trial · Has a comparison group · Run by the maker or the author

  • 90% of “AI-native” developers at level 2, his own estimate

    Shapiro, 2026 (opinion) · opinion · No comparison group · Run by the maker or the author

  • An internal product with no manually written code, the time ratio the team’s own estimate

    Lopopolo, OpenAI, 2026 (vendor) · self-report · No comparison group · Run by the maker or the author

Twelve claims, sorted into five kinds by what stands behind them. A comparison group means the people were split into two. Independent means someone other than the tool’s maker ran it. DORA’s survey is run by Google Cloud, a vendor, so it counts as neither. Nothing here says factories are faster. Or slower.

Cursor is a vendor writing about its own agents. It reports hundreds of concurrent agents, and a browser of over a million lines built in close to a week. And a migration that still needs careful review.

Carlini reports a compiler of about 100,000 lines of Rust. It builds Linux 6.9 on three architectures and passes 99% of most compiler test suites. His own caveat: “It’s not yet a drop-in replacement for a real compiler.”

OpenAI says an internal product was built over five months with no manually written code, in about a tenth of the time the team estimates hand-writing would have taken. It came to about a million lines and roughly 1,500 merged pull requests. That’s the team’s own count. The post says the autonomy it describes “should not be assumed to generalize without similar investment”.

Jacob went from an 18-person team to five and shipped Swamp 900 times in four weeks, according to the episode description on the Changelog podcast. That’s the show describing his account. A self-report.

StrongDM gives rules and capacity, and no outcome metrics. The Gas Town README says it scales comfortably to 20 to 30 agents. That’s its own claim.

DORA’s 2025 survey, run by Google Cloud, covers nearly 5,000 technology professionals and finds AI adoption goes with higher delivery throughput and with lower delivery stability. It reports associations, not causes. Its authors put the instability down to change volume outrunning controls such as automated testing, version control and feedback loops. They sum it up: “AI doesn’t fix a team; it amplifies what’s already there.”

METR’s 2025 trial had 16 experienced open-source developers work on 246 issues, each randomly assigned to allow AI or not. With AI allowed they took 19% longer, and afterwards still believed it had sped them up. METR now says these results no longer reflect current tools. I don’t redraw the felt-versus-measured chart: it’s on A-changin’.

In 2026 METR set its new data aside. Developers wouldn’t work without AI, and those running several agents at once often did other work while they waited, so time-on-task records weren’t reliable. METR thinks developers are likely more sped up now. But it says “our data is only very weak evidence for the size of this increase”.

In a trial Anthropic ran, Shen and Tamkin had 52 mostly junior engineers learn a library, and the group using AI averaged 50% on a quiz against 67% for the group coding by hand. The gap was largest on debugging questions. The AI group was about two minutes faster, which was not statistically significant. The authors name a small sample, a quiz soon after the task, and a set-up unlike agentic tools.

The best-measured finding isn’t about speed. It’s that the usual way to measure speed breaks when one person runs several agents at once. METR said so about its own study.

What a factory would have to record to appear in this ledger is on Traces and receipts. I keep cost out of this series. Where margin moves if the work gets cheap is on What gets cheap.

Limits and counter-evidence

DORA’s results come from a survey. They’re associations, not causes.

METR’s trials tested individual developers on their own repositories. Not factories.

Shen and Tamkin’s trial was small, short and not about agentic tools. The authors say so themselves.

A self-report shows a thing can be done, not how often it works. It’s an existence proof.

Sources

  1. Cursor, 2026 (vendor) Scaling long-running autonomous coding. The vendor’s own blog post, 14 January 2026; read 2026-11-05.

    A judge at the end of each cycle decides whether to continue and fresh starts fight drift; hundreds of concurrent agents; a browser of over a million lines in close to a week; a migration that still needs careful review.

    • Hundreds of agents, the browser and the migration
    • A claim in the ledger

  2. Carlini, Anthropic, 2026 (vendor) Building a C compiler with a team of parallel Claudes. Anthropic Engineering blog, 5 February 2026; read 2026-11-05.

    The verifier has to be nearly perfect; a stricter CI stopped new commits breaking old features; one phase calls out to GCC, which he calls cheating; about 100,000 lines of Rust that build Linux 6.9 on three architectures; the run reports a cost, not shown here.

    • The compiler and its limit
    • A claim in the ledger

  3. Lopopolo, OpenAI, 2026 (vendor) Harness engineering: leveraging Codex in an agent-first world. The vendor’s own post, 11 February 2026, read in Claude in Chrome; read 2026-11-05.

    An internal product built over five months with no manually written code, about a million lines and roughly 1,500 merged pull requests by the team’s own count, a time ratio the team calls an estimate; invariants enforced by linters and structural tests; minimal blocking merge gates, which the post says would be irresponsible at low throughput; and its own hedge on how far this generalises.

    • The introduction and the section on levels of autonomy
    • A claim in the ledger

  4. Changelog and Friends 130, 2026 (podcast) Automation at the speed of Swamp. Episode page, recorded 9 March 2026, published 13 May 2026; the transcript was not available; read 2026-11-05.

    The episode description says Jacob went from an 18-person team to five and shipped Swamp 900 times in four weeks. A self-report as the show describes it, not a measurement.

    • The episode description
    • A claim in the ledger

  5. StrongDM AI, 2026 (vendor) Software Factories And The Agentic Moment. The vendor’s own page, 6 February 2026; read 2026-11-05.

    The two rules (code must not be written or reviewed by humans), scenarios kept outside the codebase like a holdout set, satisfaction in place of a green suite, behavioural clones of outside services, a daily token-spend benchmark per engineer with no amount shown here, and no outcome metrics.

    • No outcome metrics
    • A claim in the ledger

  6. Yegge, Gas Town README (project docs) Gas Town. The project’s own README on GitHub; read 2026-11-05.

    Multi-agent orchestration with named roles, one of them a merge queue that runs verification gates, and its own claim that it scales comfortably to 20 to 30 agents.

    • The README’s scaling claim
    • A claim in the ledger

  7. DORA 2025, Google Cloud (vendor-run survey) Announcing the 2025 DORA Report: State of AI-Assisted Software Development. Google Cloud blog, 23 September 2025; the report PDF was not read; read 2026-11-05.

    A survey of nearly 5,000 technology professionals: AI adoption goes with higher delivery throughput and with lower delivery stability; the authors’ explanation is change volume outrunning controls such as automated testing, version control and feedback loops.

    • The survey and its explanation
    • Who ran the survey
    • A claim in the ledger

  8. METR, 2025 (preprint) Measuring the Impact of Early-2025 AI on Experienced Open-Source Developer Productivity. METR’s post of 10 July 2025, with the paper on arXiv; read 2026-11-05.

    A randomised trial: 16 experienced open-source developers, 246 issues, 19% longer with AI allowed, and a belief afterwards that it had sped them up; METR now says these results no longer reflect current tools.

    • The trial and its caveat
    • The trial
    • A claim in the ledger

  9. METR, 2026 (preprint) We are Changing our Developer Productivity Experiment Design. METR’s own post, 24 February 2026, not peer reviewed; read 2026-11-05.

    METR sets its new data aside as an unreliable signal; developers would not work without AI and, running several agents at once, often did other work while waiting, which makes time-on-task records unreliable.

    • Why the data was set aside
    • Several agents at once
    • The update
    • A claim in the ledger

  10. Shen and Tamkin, Anthropic, 2026 (preprint) How AI assistance impacts the formation of coding skills. Anthropic’s research post, 29 January 2026, of the paper arXiv:2601.20245 (the paper itself was not opened); a trial run by a vendor; read 2026-11-05.

    A randomised trial of 52 mostly junior engineers learning a library: 50% against 67% on a quiz, the largest gap on debugging, and a speed difference that was not statistically significant; the authors name a small sample and a set-up unlike agentic tools.

    • The trial, its results and its limits
    • The limits they state
    • A claim in the ledger

  11. Shapiro, 2026 (opinion) The Five Levels: from Spicy Autocomplete to the Dark Factory. Blog post, 23 January 2026; read 2026-11-05.

    Levels 0 to 5 as he describes them, where he places himself, why he calls level 5 dark, and his own estimates of who is where, which are observation and not data.

    • A claim in the ledger

Each fact is paraphrased from the page named, read on the date shown. A company’s own page is marked as the vendor’s claim, an essay or a post as opinion, and the survey as vendor-run. None of it is a measurement of this site. Where a sentence is my own reading, it is marked “My view”.