Lesson 3 · Writing questions that work

The model is only as good as the question the problem

Goal: see a compound question fail quietly.

On 2026-09-19 the German inverter-fault call from lesson 2 was asked three Nouls in one request:

Noul	Recorded	Problem
"The caller is an existing customer and is angry"	0.81	Which half carried it? Unknowable.
"The caller already owns a system from this company"	0.92	Clear.
"The caller is angry"	0.82	Clear, and "impatient" would have scored differently.

The compound Noul returned a plausible number, and that is the trap. Nothing failed loudly. If the workflow later needs to treat "angry but new caller" differently from "existing customer, calm", the compound question has already thrown the information away. Split questions cost the same (all questions in a request run in parallel) and give code two switches instead of one blurred dial.

The docs' one-line rule

"Decompose questions" is called out as the most important design step in TypeSafe's build guide: one question per fact, and the combining logic lives in code (docs: How to build). The community benchmark reached the same conclusion from the other direction: its one miss came from asking a call to be sorted into three buckets when the honest answer was "not enough call to sort".

The design procedure concept

Goal: run the seven steps from memory on any new decision.

TypeSafe's build guide lays out a sequence. Paraphrased and ordered as the docs order them:

1 Use code when you canRegex, lookups and arithmetic are free, exact and instant. Jev is for judgement, not for things a function already does.

2 Decompose stateSend the part the question is about. A 200-message thread for a "is the last message angry?" question is noise.

3 Structure stateJSON objects with named fields beat a wall of text when the input has parts (metadata, messages, history).

4 Decompose questionsOne fact per question. No "and", no "or", no "if".

5 Structure questionsInstructions and criteria accept JSON. Use fields like what, not_for, examples when prose gets ambiguous.

6 Ask manyParallel evaluation makes ten questions about as fast as one. Ask everything code might need.

7 Combine in codeThresholds, weights, routing tables. The model never decides what happens; the program does.

+ Route on uncertaintyEvery branch needs a "not sure" exit. Lesson 4 is entirely about this.

Step 1 is easy to skip and expensive to skip. Counting items, comparing dates, checking whether a field is empty, matching a product code: all cheaper and more reliable in three lines of code than in one question, and lesson 5 records what happens when you ask the model anyway.

State: give the model the right thing to look at concept + recording

Goal: choose between string, object and array state, and point a question at one field with a backtick path.

StringA message, a transcript, a document. Fine when there is one thing and all questions are about all of it.

ObjectNamed parts: {ticket: {…}, customer: {…}}. Questions can reference parts with backticked paths such as `ticket.messages[0].text`. Use for anything with metadata or history (docs: State).

Array of stringsSeveral texts judged together, for example a batch of reviews or the turns of a conversation.

This request was recorded on 2026-09-19. The state is an object with customer metadata and a three-message thread; two questions point at specific parts:

{
  "state": {
    "ticket": {
      "id": "T-4471",
      "customer": {"plan": "pro", "since": "2024-02-11", "open_tickets": 3},
      "messages": [
        {"from": "customer", "text": "Your inverter app has been logging me out every hour since the update. I have a client demo tomorrow at 9 and I need the dashboard to work."},
        {"from": "agent", "text": "Sorry about that. Which app version are you on?"},
        {"from": "customer", "text": "3.2.1 on Android. Also, honestly, this is the third bug this month."}
      ]
    }
  },
  "model": "jev-latest",
  "questions": {
    "is_bug": {"type": "noul", "instructions": "The customer in `ticket.messages[0].text` reports a software defect rather than a how-to question"},
    "has_deadline": {"type": "noul", "instructions": "The customer mentions a specific upcoming deadline in `ticket.messages`"},
    "platform": {"type": "choice", "instructions": "Operating system named anywhere in `ticket.messages`", "criteria": {"android": null, "ios": null, "web": null, "not_mentioned": "No platform is named"}},
    "churn_risk": {"type": "score", "instructions": "Risk that this customer leaves, judging from `ticket.messages` and `ticket.customer.open_tickets`", "criteria": ["Satisfied, reports one isolated issue", "Irritated, mentions repeated problems", "Explicitly threatens to cancel or switch"]}
  }
}

Reading it

is_bug 0.98 and has_deadline 0.98: the backtick path scoped the question to the right message and the model read "tomorrow at 9" as a deadline without being asked to parse a date.
platform Android at 1.0: a closed-set extraction. The not_mentioned option is the escape hatch for tickets that never name one.
churn_risk exactly 1.0 with confidence 1.0: "third bug this month" is level 1 by its description, and nobody threatened to cancel, so level 2 got nothing. The level descriptions did the work.

Less state is better state

The jaggedness page lists "large irrelevant state" as a known failure mode: accuracy drops when the relevant sentence is buried in pages of unrelated text (docs: Jev 1.13 jaggedness). Trim in code before asking. If a question is about the last message, send the last message (or point at it with a path).

Criteria: where most of the accuracy lives concept

Goal: know when null is enough, when to describe, and when to go structured.

Null is fine for obvious labels. The same German call, asked "classify the call" with {sales: null, support: null, spam: null, unclear: null}, still came back support at 1.0. Labels like german or android need no gloss. Describe when two options could overlap ("billing" vs "technical" for a failing payment integration) or when a label is internal jargon nobody outside the company would decode.

Structured criteria for the hard cases. Instructions and criteria values accept strings, objects and arrays. The field names are yours; the docs use what, not_for and examples (docs: Advanced):

"criteria": {
  "billing": {
    "what": "Charges, invoices, refunds, subscription changes",
    "not_for": "A payment integration that fails technically (that is technical)",
    "examples": ["I was charged twice", "How do I downgrade my plan?"]
  },
  "technical": {
    "what": "Bugs, errors, integrations that do not work",
    "not_for": "Questions about what a feature costs",
    "examples": ["Stripe connection keeps failing", "The app logs me out"]
  }
}

Score levels as situations"Frustrated but civil" beats "medium". The docs measured it: examples in each level lifted confidence from 0.54 to 0.90 on the same bug; an unrelated example dropped it back to 0.57 (docs: Score).

Nouls as positive statements"The caller asks for a callback", not "Did the caller not want a callback?". Add {true, false} criteria when "yes" needs a boundary.

Consistency between instructions and criteriaContradictory instructions and criteria are a documented failure mode. If the instruction says "urgency" and a level says "angry", the model is being asked two things.

Literal readingJev reads literally. "Mentions a competitor" will not fire on "the other guys quoted less" unless the criteria say what counts. Spell out the cases that matter.

Spot the flaw, then fix a real set hands-on

Goal: catch the eight most common question mistakes on sight.

Now the rewrite. Below is a deliberately weak question set for the inverter-fault call. Fix it in the editor: split the compound Noul, describe the score levels as situations, add an unclear option, and phrase the negated Noul positively. The checker validates the shape; the Playground button runs your version live.

A reference rewrite (compare, do not copy)

{
  "is_existing_customer": {"type": "noul", "instructions": "The caller already owns a system from this company"},
  "is_angry": {"type": "noul", "instructions": "The caller is angry", "criteria": {"true": "Raised tone, accusations or strong language", "false": "Worried or impatient but civil"}},
  "caller_mood": {"type": "score", "instructions": "How the caller comes across", "criteria": ["Friendly or neutral", "Impatient or worried", "Angry or hostile"]},
  "outcome": {"type": "choice", "instructions": "What the caller wants from this call", "criteria": {"sales_lead": "Wants a quote, purchase or consultation", "support": "Existing customer with a technical or billing problem", "vendor_or_spam": "Selling something to the company", "unclear": "Too little information to tell"}},
  "needs_attention_today": {"type": "noul", "instructions": "The caller describes a problem that needs attention today"}
}

Recorded answers for this set are in lesson 2, level 5 (call 2): existing customer 0.92, angry 0.82, mood 1.00, outcome support 1.0, attention today 0.93.

Quiz, then rewrite your own set quiz

Goal: the seven steps and the four criteria rules, from memory.

1. Question B needs the answer to question A. What is true?

Questions are evaluated in parallel and never see each other. Order and ids mean nothing to the model.

2. A question about the customer's last message is sent with the full 80-message thread as state. Likely effect?

"Large irrelevant state" is a documented failure mode. Trim in code, or point at `thread.messages[79].text` with a backtick path.

3. Which criteria change did the docs measure lifting Score confidence from 0.54 to 0.90?

Examples that match each level sharpen the boundaries. An unrelated example did the opposite and dropped it to 0.57.

4. "The invoice total is above 1,000 euros." Best implementation?

Step 1: use code when you can. Numbers, dates and counts are exact in code and a documented weak spot in the model.

Ship it

Open the question set you saved at the end of lesson 2 and run the drill's six flaws against it. Fix what you find, run it in the Playground on three real inputs, and note any answer that surprised you. Surprises are the raw material for lesson 4.

Primary source: How to build with TypeSafe, including its complete triage_ticket.py, and Advanced: structuring instructions and criteria.