Probabilities and thresholds: when to let code decide

An answer that arrives with a probability lets you split the work: act automatically when the model is sure, and send the rest to a person. The threshold that divides them is a number you should measure, not guess.

What the probability means

Each answer carries a probability for every option you listed, and they add up to 1. That last part matters more than it looks: the probabilities cover your options and nothing else. If none of them fits the material, the weight is still shared out among them, and you get a confident-looking answer to a question that had no right answer.

Two habits prevent that. Add an option like "none of these" wherever the material might not fit, and ask a separate yes-or-no question such as "does this say enough to tell?" when the input can be too thin to judge.

Alongside the probabilities comes a certainty score, from 0 when they're split evenly to 1 when one option holds all of them. It describes the shape of the distribution. It is not a claim that the answer is right, and since it depends on how many options there are, compare it only between questions with the same number of options.

{
  "state": "Anyone who still uses this framework in 2026 clearly has not written real software.",
  "questions": {
    "personal_attack": {
      "type": "yes_no",
      "text": "Does this post attack a person rather than an idea?"
    },
    "off_topic": {
      "type": "yes_no",
      "text": "Is this post unrelated to the thread it was posted in?"
    }
  }
}

{
  "answers": {
    "personal_attack": {
      "type": "yes_no",
      "answer": "yes",
      "probabilities": {
        "yes": 0.61,
        "no": 0.39
      },
      "certainty": 0.0352
    },
    "off_topic": {
      "type": "yes_no",
      "answer": "no",
      "probabilities": {
        "yes": 0.09,
        "no": 0.91
      },
      "certainty": 0.5635
    }
  }
}

These two answers deserve different treatment. The second is clear: 0.91 that the post is on topic, so nothing to do. The first is nearly a coin flip at 0.61, and a coin flip is exactly the case a person should see. A system that only read the answer field would treat both as settled.

Choosing a threshold from your own data

A threshold trades coverage against accuracy. Raise it and the answers you act on are more often right, but fewer of them qualify and more work lands on people. Lower it and the reverse. The right point depends on what a mistake costs you, which is why nobody can hand you a default.

1. Label 100 to 200 real examples. Use your own data, including the awkward ones. A clean sample flatters the model and misleads you.
2. Run them and keep the whole answer, not just the chosen option: you need the probabilities to test thresholds afterwards.
3. Sweep the threshold. For values from, say, 0.5 to 0.99, measure two numbers: coverage, the share of examples where the leading option clears the threshold, and accuracy, how often those answers match your label.
4. Take the lowest threshold that meets your accuracy target. That gives you the most automation for the standard you set.
5. Check the ones you got wrong. Mistakes above a high threshold usually mean an ambiguous question or an option that overlaps another, and rewording fixes more than raising the bar does.

for threshold in [0.5, 0.6, 0.7, 0.8, 0.9, 0.95]:
    acted = [(a, label) for a, label in results if a["probabilities"][a["answer"]] >= threshold]
    coverage = len(acted) / len(results)
    accuracy = sum(a["answer"] == label for a, label in acted) / len(acted)
    print(f"{threshold}: covers {coverage:.0%} of items, right {accuracy:.0%} of the time")

One threshold per question, not one per system

Thresholds belong to questions, because the cost of being wrong does. Tagging a ticket with the wrong topic is a minor annoyance someone fixes later, so 0.7 may be fine. Auto-closing a ticket, emailing a customer or removing a post deserves 0.95 or higher, plus a way to undo it. In moderation, the same call often wants two: above 0.95 remove, between 0.6 and 0.95 send to a moderator, below 0.6 leave alone.

What to do below the threshold

Abstaining is a feature, not a failure, but only if something happens next. Give every question a path for the unsure cases: a review queue, a rule your team already trusts, a default that's safe to be wrong about, or a second question that asks for the missing piece. When a person does look, record what they decide. Those decisions are the labelled data for your next threshold sweep.

Keep watching it

Probabilities move when the wording moves, so re-check after you edit a question, add an option or change a description. They also drift when your input changes: a new customer segment, a new product line, a new kind of spam. Sampling a handful of automated decisions each week and checking them by hand costs little and catches this early.

One caveat worth keeping in mind: a probability is the model's weight on an option, not a measured success rate. A batch of answers at 0.9 will not necessarily be right 90% of the time. That's what the sweep on your own data tells you, and why it's worth doing rather than trusting the number as an accuracy estimate.

Using one

Every SeaCat answer comes with a probability for each option and a certainty score, so a threshold is something you apply in your own code.