C3 · Jev: a model that decides · Interactive, in your browser
Routing a request to a model
Using a decision to pick which model handles a request, and what a wrong route costs.
Routing is a decision made before the work starts, so what matters is what a wrong route costs in each direction.
Costs are invented units, not money. The strong model is assumed to be enough for every request.
- Sent to the quick model
- 8
- Sent to the strong model
- 4
- Quick, but not enough
- 2
- Strong, but quick would do
- 1
- Total cost
- 48
- If all went to the strong model
- 48
Requests, most likely to suit the quick model first
- Say when the shop is open on SundayQuick model 98%, Strong model 2%
Goes to the quick model. In fact the quick model would have been enough.
- Turn a pasted list into a tableQuick model 96%, Strong model 4%
Goes to the quick model. In fact the quick model would have been enough.
- Fix the spelling in a short noteQuick model 94%, Strong model 6%
Goes to the quick model. In fact the quick model would have been enough.
- Tag a message with one of five labelsQuick model 91%, Strong model 9%
Goes to the quick model. In fact the quick model would have been enough.
- Summarise a short note in one lineQuick model 87%, Strong model 13%
Goes to the quick model. In fact the quick model would have been enough.
- Draft a calm reply to an angry customerQuick model 79%, Strong model 21%
Goes to the quick model. In fact the quick model was not enough.The quick answer was not enough.
- Pull the dates out of an emailQuick model 74%, Strong model 26%
Goes to the quick model. In fact the quick model would have been enough.
- Compare three offers with different termsQuick model 63%, Strong model 37%
Goes to the quick model. In fact the quick model was not enough.The quick answer was not enough.
- Plan next week’s rota from a listQuick model 45%, Strong model 55%
Goes to the strong model. In fact the quick model would have been enough.Paid for the strong model it did not need.
- Find where two long documents disagreeQuick model 41%, Strong model 59%
Goes to the strong model. In fact the quick model was not enough.
- Work out a multi-step schedule with dependenciesQuick model 30%, Strong model 70%
Goes to the strong model. In fact the quick model was not enough.
- Untangle a long thread of conflicting asksQuick model 22%, Strong model 78%
Goes to the strong model. In fact the quick model was not enough.
My viewRouting is deciding, before the work starts, which model a request goes to. The figure has twelve invented requests, each with an invented probability that the quick model is enough, and what turned out to be true. Drag the threshold: a request at or above it goes to the quick model, and one below it goes to the strong model.
LangChain’s documentation for the TypeSafe integration describes a model-routing middleware. It classifies the latest human message across the models you name, each paired with a criterion for picking it, and uses the chosen model for every model call in that run. You can add guidance; the docs’ example is to choose the least costly model that can complete the task safely. The chosen route is available on the result.
The docs also say this middleware is experimental, needs an extra install, and has an interface that may change without notice. The page I read does not say what the middleware does when the classifier is unsure.
TypeSafe’s docs list routing a ticket to a department as the kind of job a Choice question suits: one answer from a known list with no natural order. They say the question’s criteria define the options, and to add an “other” or “none of the above” option when the list might not cover every input. A Choice returns the selected option, a probability for every option and a confidence that says how concentrated those probabilities are.
TechCrunch reports one developer’s suggestion that predicting whether a given workload needs a particular model would be useful, which is routing. It is a suggestion about where the model might be used, not a reported deployment.
TypeSafe’s docs say a confidence threshold is not one number: an action with bigger consequences should need more confidence. For a low-confidence answer they suggest, as a starting pattern, routing to a human, asking for clarification, or falling back to a different system, and they recommend starting with conservative thresholds. In the figure’s simpler rule, anything below the threshold goes to the strong model.
My viewMy reading of the figure: there are two ways to route wrongly and they cost different amounts. Send a request to the strong model that the quick model would have handled, and you overpaid. Send one to the quick model that it could not handle, and you got a bad answer. Which costs more depends on your work. At the starting costs, the lowest threshold with the smallest total on this set is 80% (total 33), against 48 at the 60% start and 48 for sending everything to the strong model. The figure assumes the strong model is always enough, which real work will not guarantee. All of it is invented.
My viewI would not take a routing rule from a vendor’s page, and this chapter does not show that Jev routes well. It shows how the decision is shaped, and the next steps are to put a cost on your own mistakes and to test the probabilities on your own requests, which chapter 6 is for.
Exercise
Write a routing rule you could defend.
- Raise the cost of a quick answer that was not enough to its highest and lower the strong model’s cost. Where does the cheapest threshold go? Then swap them.
- Pick one request in the list that you would never leave to the quick model whatever its probability. What would you add to the list of routes?
- Write the “other” option and the criteria for each route on one of your own request types, then say what should happen when none of them clearly leads.
Sources
LangChain docs TypeSafe integration. Integration documentation; read 2026-11-01.
The LangChain classifier, the experimental model-routing and tool-gating middleware, what is sent to TypeSafe, and that gating refuses rather than asks. The provider package (langchain-typesafe) may be written by TypeSafe, so I read the page as the integration’s documentation, not as an independent view.
- The model-routing middleware: named models with criteria, optional instructions, the latest human message classified, the route on the result
- The middleware are experimental and their APIs may change without notice; no description of a low-confidence or fallback behaviour
TypeSafe docs Primitives (Questions). The vendor’s documentation; read 2026-11-01.
Choice, Score and Noul in detail, one question per judgment, combining factors in your own code, Choice as the fit for routing a ticket, and an other option when the list may not cover every input.
- Choice for routing a ticket; criteria define the options; an other option when the list might not cover every input; choice, probabilities and confidence returned
- A Choice returns a probability for every option
TechCrunch A new kind of AI model from a ChatGPT inventor is thrilling developers. Press article, 18 September 2026; read 2026-11-01.
Developers’ early uses, one developer’s caution, anecdotal testing, and that outside observers suspect an open-weight base which the article does not confirm.
- A developer suggests predicting which model a workload needs; a suggestion, not a reported use
TypeSafe docs Confidence. The vendor’s documentation; read 2026-11-01.
Confidence as a statistic of the probabilities, no confidence for a yes/no answer, bands of action, thresholds that scale with risk, testing with your own data.
- Thresholds scale with risk; low confidence means route to a human, ask, or fall back to a different system; start conservative
- Thresholds that scale with risk
Each fact is paraphrased from the page named, read on the date shown. TypeSafe’s own pages are marked as the vendor’s claims: they say what the company reports, and I have not tested any of it. Where a sentence is my own reading, it is marked “My view”.