Grading model outputs against a rubric

Scoring generated text with another model is now routine: rating answers in an eval suite, checking support replies before they're sent, ranking candidates for review. Doing it with typed questions gives you the spread across your rubric's levels and an average you can sort by, rather than a number you have to fish out of a paragraph.

Write the rubric as ordered levels

A rubric is a scale with the levels described. Put them in order, lowest first, and describe each one concretely enough that two people reading the same output would land in the same place. Three or four levels is usually right: more invites disagreement between neighbours that no one can resolve.

Avoid bare numbers. "3 out of 5" means whatever the reader assumes; "Answers part of it, or leaves the customer guessing" means one thing. The words are what the model reads, and they're what makes the scores comparable next month.

One criterion per question

"Is this reply accurate and friendly?" folds two judgements into one number, and you can't tell afterwards which one failed. Ask separately: accuracy on its own scale, tone on its own scale, and each policy check as its own yes-or-no. Since every question is answered independently, seeing only the output and itself, one criterion can't drag another along, and adding one more costs only its own words.

{
  "state": {
    "customer_asked": "Can I move my subscription to a different email address?",
    "agent_replied": "Yes. Open Settings, then Account, and change the address there. You will get a confirmation link at the new address; the change takes effect once you open it."
  },
  "questions": {
    "helpfulness": {
      "type": "scale",
      "text": "How well does the reply answer what the customer asked?",
      "options": [
        "Does not answer the question",
        "Answers part of it, or leaves the customer guessing",
        "Answers it fully, with the steps to follow"
      ]
    },
    "tone": {
      "type": "scale",
      "text": "How appropriate is the tone for a support reply?",
      "options": [
        "Rude or dismissive",
        "Flat but acceptable",
        "Warm and professional"
      ]
    },
    "invented_detail": {
      "type": "yes_no",
      "text": "Does the reply state a fact about the product that the customer did not provide and that a support agent could not know from this message alone?"
    }
  }
}

{
  "answers": {
    "helpfulness": {
      "type": "scale",
      "answer": "Answers it fully, with the steps to follow",
      "probabilities": {
        "Does not answer the question": 0.01,
        "Answers part of it, or leaves the customer guessing": 0.07,
        "Answers it fully, with the steps to follow": 0.92
      },
      "certainty": 0.7188,
      "mean": 1.91
    },
    "tone": {
      "type": "scale",
      "answer": "Warm and professional",
      "probabilities": {
        "Rude or dismissive": 0.01,
        "Flat but acceptable": 0.34,
        "Warm and professional": 0.65
      },
      "certainty": 0.3693,
      "mean": 1.64
    },
    "invented_detail": {
      "type": "yes_no",
      "answer": "no",
      "probabilities": {
        "yes": 0.28,
        "no": 0.72
      },
      "certainty": 0.1445
    }
  }
}

Two of these are settled and one isn't. Helpfulness sits at 0.92 on the top level, tone leans warm but keeps a third of its weight on "flat but acceptable", and the invented-detail check is genuinely undecided at 0.28 to 0.72. That last one is the reply mentioning a confirmation link, which may well be how the product works, but isn't in anything the grader was shown. Undecided is the honest answer, and it's the one a person should see.

The average, and why it's useful

A scale answer also comes back with an average: each level's position, counting the lowest as 0, weighted by its probability. It's a single number you can sort by, chart over time, or compare between two versions of a prompt, and it moves smoothly where the chosen level jumps. A run whose average helpfulness slips from 1.9 to 1.6 has got worse in a way that a count of top-level answers might miss entirely.

For a pass rate, add up the levels you count as passing rather than looking only at the leading answer: the probability of "fully" plus the probability of "partly" is the chance the reply was at least partly useful.

Checks that aren't scales

Some rubric items are conditions, not degrees: did it follow the format, did it include the disclaimer, does it state anything the source doesn't support. Those are yes-or-no questions, and they take their own thresholds. A safety check usually wants to escalate at a much lower probability than a style check, because the costs aren't comparable.

What to watch out for

• Grade what you show it. The grader sees only the material you send, so include the original request alongside the output. "Does this answer the question?" needs the question.
• Don't ask it to calculate. There's no working out, so counting words, averaging numbers or checking arithmetic belongs in your code, with the result put into the material.
• Watch wording, not just scores. Rewriting a level's description moves the scores, so freeze the rubric while you're comparing runs, and re-baseline when you change it.
• Spot-check against people. Grade 50 outputs by hand and compare. Where you disagree, the rubric is usually ambiguous, and fixing the words helps both readers.
• Don't let a grade close the loop silently. Sample what gets auto-accepted. A grader that quietly drifts is worse than no grader.

Using one

SeaCat takes the output and the rubric in one request: up to 64 criteria about the same material, each answered with its probabilities, and an average for every scale. The material is billed once however many criteria you attach, at $0.20 per million input tokens.