Typed decisions instead of text

A typed decision is an answer that can only be one of the values you defined, returned with a probability for each of those values. Your code gets something it can switch on, and a number saying how close the call was, instead of a sentence it has to interpret.

Most software that asks a model a question doesn't want prose. It wants to know which queue a message belongs in, whether a document supports a claim, or how bad a review is, and then do something about it. The reply is read by a program, not a person.

Why a written reply is awkward to act on

Ask for one word and you'll usually get one word. Usually is the problem. The reply comes back with a full stop one time in fifty, or as a synonym you didn't list, or wrapped in a sentence explaining itself. So the code grows a parser, then a list of accepted spellings, then a retry when the parse fails, then a fallback for when the retry fails too.

Two costs hide behind that. Every reply is composed word by word, so it takes time proportional to its length and you pay for the words. And whatever comes back is one sample from a distribution you never see: a near-coin-flip and a certainty both arrive as the same confident-looking word.

What a schema fixes, and what it doesn't

Asking for JSON that matches a schema, with the options as a fixed list of allowed values, solves the shape problem. The reply parses, and the value is one you allowed.

What a schema can't do is tell you how sure the model was. The answer is still produced a token at a time, still sampled, and still priced as output. Two answers that look identical in the JSON can come from a distribution that was 0.98 on one option and one that was split 0.4 to 0.35 to 0.25, and nothing in the response distinguishes them. That difference is exactly what you need to decide whether a person should look.

What a typed decision adds

A typed decision starts from the other end. You give the options; the model's probability for each of them is read directly and rescaled to add up to 1. Nothing is composed, so:

• The answer is always one of your options. Not a near miss, not a paraphrase, not a sentence.
• You see the whole distribution. Every option gets a probability, and a certainty score says how concentrated they are.
• The same input gives the same answer. Nothing is sampled, so results don't move between runs.
• There's no output to pay for or wait on. The cost is the material and the questions you sent.

The three types

Options can be plain names, or names with a description when a name alone is ambiguous. The description is for the model; the answer comes back as the name, which is what your code wants.

{
  "state": "The charger you sent stopped working after three weeks, and nobody has replied to my two emails.",
  "questions": {
    "topic": {
      "type": "category",
      "text": "What is this message mainly about?",
      "options": {
        "faulty_product": "Something bought has stopped working or arrived broken",
        "delivery": "Where an order is, or when it will arrive",
        "support_delay": "Nobody has answered an earlier message"
      }
    }
  }
}

{
  "answers": {
    "topic": {
      "type": "category",
      "answer": "faulty_product",
      "probabilities": {
        "faulty_product": 0.82,
        "delivery": 0.02,
        "support_delay": 0.16
      },
      "certainty": 0.5138
    }
  }
}

Note what the probabilities say that the answer alone doesn't: the message is mostly about a faulty product, but a sixth of the weight sits on the complaint about nobody replying. If your support tool can tag a ticket twice, that's the second tag.

Acting on the numbers

Because the distribution comes back with every answer, the decision about whether to trust it belongs in your code, not in a prompt. The usual shape is a threshold per question: act when the leading option clears it, and otherwise queue the item for a person or fall back to a rule.

answer = result["answers"]["topic"]
if answer["probabilities"][answer["answer"]] >= 0.85:
    route_to(answer["answer"])
else:
    queue_for_review()

Choosing those thresholds from labelled examples, rather than by feel, is its own guide.

Where it fits alongside a chat model

Typed decisions don't replace writing; they're the part of a system that has to be predictable. A chat model drafts the reply, and a typed decision decides whether the draft is safe to send. A chat model extracts fields, and a typed decision checks each one against the source. An agent writes and plans, and a typed decision picks which branch it takes next, from the branches you defined.

Using one

SeaCat answers typed decisions over HTTP: one piece of text or JSON, up to 64 questions about it, every answer with its probabilities in a single response. It's $0.20 per million input tokens, with nothing to pay for output.