My Building Library

BAA 101 · Chapter 05

The model's output is untrusted data

1 min read

The principle

What the model returns must cross a boundary before it enters the program: a fixed SchemaThe required shape of a piece of data: its fields, their types, their bounds.See in the glossary →, and a strict ParserThe code that reads the model's output and rejects it if it doesn't match the schema: format, exact fields, types, bounds.See in the glossary → that accepts or rejects it. Otherwise, a RefusalAn explicit "I can't answer": no citation, zero confidence, human review. A state, not a crash.See in the glossary →.

Why it matters

Free text can’t be checked: where’s the source, what confidence? A model can cut its answer short, invent a field, rephrase a CitationThe ID of the document that supports an answer. A correct citation doesn't prove the sentence comes from the document.See in the glossary →, or display certainty that nothing justifies.

The mechanism

  1. A schema fixes the shape of the answer: the text, the cited sources, a bounded confidence, a human-review flag.
  2. A strict parser checks the format, the exact fields (none missing, none extra), the types and the bounds. Its message names the error.
  3. One Corrective retryA single second attempt where the model gets back the error from its first one. Beyond that, a flagged refusal.See in the glossary →: the model gets its error back. If it fails again, a flagged refusal.
  4. The less the model writes, the less there is to check: you can make it choose (passage numbers) and let the code copy. The citations then follow from what’s copied, and can no longer be invented.
  5. A confidence the sources don’t justify is a warning sign, not a detail.

In the field

  • Dev3Pack, Session 3: a numeric citation ([42]) rejected by the parser; a stubborn model still wrapping its JSON in prose on the second attempt, then a flagged refusal; a confidence of 1.0 on a single document.
  • Dev3Pack, Final: after four prompt tweaks with no stable effect, the model stopped writing and started choosing paragraphs, which the code copies. Result: 9/10 in training, then 15/15.

The question to ask

What in this answer was checked by code before being shown?