C2 · Jev: a model that decides · Interactive, in your browser

What a probability is for

Move a threshold over a set of scripted decisions and see what is accepted, the mistakes on each side and what they cost.

A probability is only useful once you decide where to act on it, and that depends on what each mistake costs.

Act when the probability is at least60%

Cost of running a wrong action10

Cost of holding back a fine one1

Costs are invented units, not money.

Run automatically: 8
Sent to a person: 8
Wrongly run: 1
Wrongly held back: 2
Total cost: 12

On this set, the cheapest threshold at these costs is 82% (total 4).

Proposed actions, most probable first

Read the opening hours97%
Runs automatically. In fact it was fine to run.
Look up an order’s status95%
Runs automatically. In fact it was fine to run.
Draft a reply without sending it92%
Runs automatically. In fact it was fine to run.
Add a note to a job90%
Runs automatically. In fact it was fine to run.
Send an appointment reminder86%
Runs automatically. In fact it was fine to run.
Move a customer’s booking81%
Runs automatically. In fact it was not fine to run.Wrongly run.
Refund a small order72%
Runs automatically. In fact it was fine to run.
Reorder stock66%
Runs automatically. In fact it was fine to run.
Refund an order over the usual limit58%
Goes to a person. In fact it was not fine to run.
Update a customer’s address55%
Goes to a person. In fact it was fine to run.Wrongly held back.
Pay a supplier’s invoice44%
Goes to a person. In fact it was fine to run.Wrongly held back.
Cancel a recurring job40%
Goes to a person. In fact it was not fine to run.
Email every customer31%
Goes to a person. In fact it was not fine to run.
Delete a customer’s record18%
Goes to a person. In fact it was not fine to run.
Change the opening hours on the site12%
Goes to a person. In fact it was not fine to run.
Close the customer’s account6%
Goes to a person. In fact it was not fine to run.

Did the probabilities mean it?

Actions grouped by their probability, with how many were in fact fine
Said	Actions	Fine
0% to under 25%	3	0 of 3
25% to under 50%	3	1 of 3
50% to under 75%	4	3 of 4
75% or more	6	5 of 6

Calibrated means that of the actions given a probability near a band’s level, about that share were fine. The top band is 75% or more and has 5 of 6 fine; the bottom band has 0 of 3. These are invented, and a set this small proves nothing.

My viewA probability does nothing until you decide where to act on it. The figure has sixteen actions an agent might propose, each with an invented probability that it is safe to run without asking a person, and what turned out to be true. Drag the threshold: an action at or above it runs, one below it waits for a person.

My viewThere are two ways to be wrong. Run something that was not fine, and a mistake has happened. Hold back something that was fine, and a person has been bothered for nothing. They rarely cost the same, so the two cost sliders matter more than the threshold: change them and the best place to set the threshold moves. With the starting costs, the lowest threshold with the smallest total on this set is 82%, and it is a hint from a small invented set, not a rule.

TypeSafe’s docs give the same advice in words. They suggest three bands, high confidence to act automatically, medium to proceed cautiously, and low to avoid acting and route elsewhere, and say thresholds should scale with risk, so that different actions in one system are gated at different levels. They say the exact values depend on your domain and recommend starting conservative.

Whether the probabilities mean what they say is calibration, and the table in the figure shows the idea. Calls are grouped into four bands by probability, and each band shows how many were in fact fine. In the top band here 5 of 6 were fine; in the bottom band 0 of 3. TypeSafe’s docs say calibration is measured across groups of predictions, so it does not promise that any single answer is right, and that its probabilities are tuned against outcomes.

TypeSafe’s announcement names its training method, Reinforcement Learning for Calibrated Decisions, and says it targets calibrated probabilities. It does not explain the algorithm there. For a yes or no answer the docs give no separate confidence, and say a value near one half signals doubt.

TechCrunch reports one developer’s caution that a model like this hands part of the hallucination problem back to the user, who must decide what confidence to act on. TypeSafe’s docs say to test with your own data and adjust thresholds from what you observe.

My viewI agree with the framing, and I would add that nobody can decide it from a vendor’s page. Testing on your own data is chapter 6.

Exercise

Set a threshold you could defend.

Set the cost of running a wrong action to its highest and the cost of holding back a fine one to its lowest. Where is the cheapest threshold now? Then swap the two costs.
Pick one action in the list that you would never let an agent run alone, whatever its probability. What does that tell you about a single threshold for everything?
Look at the table. If you had only the top band, would you trust it? What would you want to know about how the examples were chosen?

Sources

TypeSafe docs Confidence. The vendor’s documentation; read 2026-10-30.
Confidence as a statistic of the probabilities, no confidence for a yes/no answer, bands of action, thresholds that scale with risk, testing with your own data.
- Three bands of action; thresholds scale with risk; start conservative
- A yes or no answer has no confidence field; near one half signals doubt
- Test with your own data and adjust thresholds from observed results
- Bands of action and thresholds that scale with risk
TypeSafe docs System One. The vendor’s documentation; read 2026-10-30.
What a System One model is, no free-form output, text-only input, and that calibration is measured across groups of predictions.
- Calibration is measured across groups of predictions, not a guarantee for one answer
TypeSafe AI blog Introducing System One Models and Jev. The vendor’s own post, bylined 15 September 2026; read 2026-10-30.
What a System One model is, the training method’s name, the stated limits, and what TypeSafe says its evaluations are based on.
- The training method and its goal of calibrated probabilities
TechCrunch A new kind of AI model from a ChatGPT inventor is thrilling developers. Press article, 18 September 2026; read 2026-10-30.
Developers’ early uses, one developer’s caution, anecdotal testing, and that outside observers suspect an open-weight base which the article does not confirm.
- A developer’s caution that the user must decide what confidence to act on

Each fact is paraphrased from the page named, read on the date shown. TypeSafe’s own pages are marked as the vendor’s claims: they say what the company reports, and I have not tested any of it. Where a sentence is my own reading, it is marked “My view”.