About 25 minutes · the lesson the benchmark was really about
Goal: feel the difference between a probability and a guarantee.
In the community benchmark of 2026-09-17, call 028 was an incomplete German transcript. The expected label was "unclear". Jev labelled it "not sales" with a probability of 0.93. Thirty-nine other calls went well; this one did not, and it did not look uncertain (benchmark brief, section "blind spot").
A calibrated 0.93 is a claim about frequency: across many answers given at 0.93, about 7 in 100 are wrong. It is not a promise about this answer. Call 028 was one of the 7. A workflow that treats 0.93 as "certain" will act on every one of those 7 with full force.
The brief's conclusion became this course's refrain: give uncertainty a route. Two changes fix call 028 without touching the model. First, an unclear option so the honest answer exists (lesson 2). Second, a confidence floor below which no automated action fires, plus a higher bar for actions that are expensive to undo.
The rest of this lesson is about that second change. It rests on one field in every Choice and Score answer: confidence.
Goal: explain why confidence and the winning probability are different numbers.
The docs describe confidence as "a statistic computed from the distribution" over the options or levels, collapsed to a single 0 to 1 number that says how peaked the distribution is (docs: Confidence). The formula is not published, and this course will not invent one. What can be said from the recordings:
| Recorded answer (2026-09-19) | Distribution | Top probability | Confidence |
|---|---|---|---|
| Stripe message → department | billing 0.69 · technical 0.31 · sales 0 | 0.69 | 0.53 |
| Cut-off call → outcome | unclear 0.99 · support 0.01 | 0.99 | 0.98 |
| Cut-off call → mood (Score) | level 0 at 0.89 · level 1 at 0.11 | 0.89 | 0.84 |
| Numbers-only mood ladder | 0.02 · 0.50 · 0.48 | 0.50 | 0.26 |
| Docs: requested_resolution | exchange 0.37 · refund 0.29 · replacement 0.24 · info 0.10 | 0.37 | 0.16 |
Confidence falls faster than the top probability as the mass spreads out. Two options at 0.50 / 0.48 give 0.26, not 0.50. Four options with a 0.37 leader give 0.16. That is the useful property: confidence punishes "several plausible answers" harder than the top probability alone does.
Goal: set two thresholds for a real workflow and see what each setting lets through.
The docs' worked examples use a floor of about 0.5 for anything automated and a bar of 0.85 to 0.9 for actions that move money or cannot be undone (docs: choosing thresholds, pattern: confidence-gated routing). The simulator below routes nine recorded answers through two sliders. The first six are this course's recordings; the last three are from the docs and from the benchmark. One of them is call 028.
unclear option (which the benchmark's question set lacked) or a human sample catches it.Goal: implement act / confirm / human once and reuse it everywhere.
The docs' confidence-gated routing pattern in the shape this course uses. Python first, then the same thing as n8n nodes.
from typesafe_sdk import TypeSafeClient, Choice, Noul
client = TypeSafeClient() # reads TYPESAFE_API_KEY
RISKY = {"sales_lead", "vendor_or_spam"} # actions that are hard to undo
FLOOR, HIGH = 0.5, 0.9
def route(transcript: str) -> tuple[str, str]:
r = client.system_one(
state=transcript,
questions={
"outcome": Choice("What the caller wants from this call", {
"sales_lead": "Wants a quote, purchase or consultation",
"support": "Existing customer with a technical or billing problem",
"vendor_or_spam": "Selling something to the company",
"unclear": "Too little information to tell",
}),
"callback_requested": Noul("The caller asks to be called back or agrees to an appointment"),
},
)
a = r.answers["outcome"]
if a.choice == "unclear" or a.confidence < FLOOR:
return "human", a.choice # review queue
if a.choice in RISKY and a.confidence < HIGH:
return "confirm", a.choice # ask a human to approve, or ask the caller
return "act", a.choice{{$json.answers.outcome.choice}} == "unclear" OR {{$json.answers.outcome.confidence}} < 0.5 → human branch. Second IF: choice in the risky list AND confidence < 0.9 → confirm branch. Everything else → act.The TypeSafe agent skill gives the same advice for code: questions and thresholds in a single file, versioned, so tuning is a diff and not an archaeology project (docs: Agent skill). Log every routed decision with its confidence; lesson 6 uses that log to set the numbers from data instead of from taste.
Goal: gate a Score and a Noul without a confidence field to lean on.
confidence < floor → human, risky and confidence < high → confirm.unclear or other in the options where the model needs somewhere honest to go.Goal: never again treat a probability as a guarantee.
1. A Choice returns 0.50 / 0.48 / 0.02. Roughly what confidence should be expected?
Recorded: 0.26. Confidence punishes a split between plausible answers harder than the top probability does.
2. Benchmark call 028 was labelled wrong at 0.93. Which change would most directly have prevented an automated action?
The honest answer had nowhere to go. Thresholds limit damage from confident misses; an escape option lets the model not miss.
3. Which action deserves the higher confidence bar?
Thresholds scale with the cost of being wrong. A refund is hard to undo; a tag is one click to fix.
4. How is uncertainty read on a Noul?
Nouls have no confidence field. 0.93 and 0.07 are decisive; 0.55 is a review case.
For the decision you have been carrying since lesson 1, write down two numbers and one list: the floor, the high bar, and which actions count as risky. Then add the third outlet to the workflow, even if it is just a Slack message saying "not sure about this one". Lesson 6 shows how to tune the numbers from a shadow run.
Primary source: docs.typesafe.ai/confidence and the community benchmark, in particular the blind-spot section.