C1 · Software factories: an old word for a new line · Interactive, in your browser
How dark is the factory?
At each level of autonomy, where does the person stand, and what has to exist before they can step back?
Shapiro’s levels say where the person stands. StrongDM’s account says what its team built before it stepped back. The figure lets you try a setup against both.
What exists in the setup
Or start from one
This setup supports
Level 3
To go higher: what done means is written down before the run.
Of the 32 possible setups, 24 give level 3, 7 give level 4, 1 gives level 5.
Level 0 · Spicy autocomplete
You write the code. Nothing reaches the disk unless you approve it.
Level 1 · Discrete tasks
You hand off specific, discrete tasks.
Level 2 · A colleague
You pair with the AI like a colleague.
Level 3 · The manager
You’re the manager. Your days go on reviewing code.
Level 4 · Specs, then checks
You write specs, craft skills and review plans. Then you leave and check later if the tests pass. Shapiro puts himself here.
Level 5 · The dark factory
The person isn’t part of the process. Shapiro puts it this way: “It’s a black box that turns specs into software.”
Invented setups, nothing measured. The rule is mine, a simplification of Shapiro’s levels and StrongDM’s account. Level 4 needs a written spec and automated checks. Level 5 needs held-out checks, stand-ins for outside services and no person at the merge. Here, level 5 is one vendor’s answer, not a proof.
Shapiro’s levels aren’t about how good the model is. They’re about where the person stands. At level 3 the person reviews code all day, and at level 4 they write what done means and check later whether the tests pass. Level 5 is the dark factory, where nobody is in the process.
Shapiro puts himself at level 4. He also estimates how many people are at each level, but that’s his own observation, not data. So I leave it out.
StrongDM is a vendor describing its own team. It says it keeps scenarios outside the codebase, like a holdout set in model training, and has a model validate them. A run is scored by satisfaction, not a green test suite, and tested against behavioural clones of the outside services its software depends on.
The loop runs until the holdout scenarios pass and keep passing. That’s from StrongDM’s principles page.
StrongDM publishes a daily token-spend benchmark per engineer, so it reports a cost. Its page gives no outcome metrics, such as defect rates or delivery speed. I don’t show the amount.
Willison, an outside engineer, asks how you prove software works when agents write both the code and the tests. Kahana writes from Stanford Law about StrongDM. His point: a model judging another model’s work can share its blind spots.
Kahana’s other point: with no person reviewing, output is hard to trace to someone responsible. He puts it in one line: “The risk is that when it fails, nobody will know why.”
In my own setup a script ships, not a model, and a hook asks before anything is pushed to main. So a person is at the merge. What done means is written down before the run, and a gate runs the checks. The builder can read every check, though, and nothing stands in for the outside services: in the figure, that’s the preset called My setup.
Limits and counter-evidence
Shapiro’s levels are one person’s framework. He built it from what he’s seen, not from data.
The level 5 recipe belongs to StrongDM, whose page publishes no results. Willison and Kahana question whether software built that way can be trusted.
StrongDM’s stand-ins only matter for software that calls outside services. So the figure’s rule is a simplification, and it says so.
Sources
Shapiro, 2026 (opinion) The Five Levels: from Spicy Autocomplete to the Dark Factory. Blog post, 23 January 2026; read 2026-11-05.
Levels 0 to 5 as he describes them, where he places himself, why he calls level 5 dark, and his own estimates of who is where, which are observation and not data.
- Levels 3 to 5
- Level 4; his estimates
- The levels
- The post
- Levels 0 to 5
StrongDM AI, 2026 (vendor) Software Factories And The Agentic Moment. The vendor’s own page, 6 February 2026; read 2026-11-05.
The two rules (code must not be written or reviewed by humans), scenarios kept outside the codebase like a holdout set, satisfaction in place of a green suite, behavioural clones of outside services, a daily token-spend benchmark per engineer with no amount shown here, and no outcome metrics.
- Scenarios, satisfaction and the clones of outside services
- The benchmark; no outcome metrics
- How its team runs without review
- The page gives no outcome metrics
- Behavioural clones of outside services
- The rule for level 5
StrongDM AI principles, 2026 (vendor) Principles. The vendor’s own subpage; read 2026-11-05.
The loop of seed, validation harness and feedback, which runs until the holdout scenarios pass and keep passing.
- Seed, validation harness, feedback loop
Willison, 2026 (opinion) How StrongDM’s AI team build serious software without even looking at the code. Blog post, 7 February 2026; read 2026-11-05.
An outside engineer’s question: how to prove software works when agents write both the implementation and the tests.
- His question
- The question
Kahana, Stanford Law, 2026 (opinion) Built by Agents, Tested by Agents, Trusted by Whom?. Stanford Law School CodeX blog, 8 February 2026; read 2026-11-05.
When a model judges another model’s work the two may share blind spots; with no person reviewing, output is hard to trace to someone responsible.
- Builder and inspector may share blind spots
- Traceability and the closing risk
- The argument
Each fact is paraphrased from the page named, read on the date shown. A company’s own page is marked as the vendor’s claim, an essay or a post as opinion, and the survey as vendor-run. None of it is a measurement of this site. Where a sentence is my own reading, it is marked “My view”.