The final's notebook

You run the course’s reference research assistant, check that it passes four gates, then honestly list what it still does badly. Passing your tests doesn’t mean having no flaws: naming your limits before the demo saves you from discovering them on stage.

The concept

Think of a stock order control tested only with perfect orders you wrote yourself. “Compliant” says nothing about a real trader in a hurry who types the wrong ticker.

The agentAgentA program that uses a language model to decide what to do (search, call a tool, answer, refuse), in a loop, with stopping rules.See in the glossary →Close’s circuit: question → retrievalRetrievalThe step that searches the corpus for passages useful to a question, before calling the model. Nothing found: refuse without a call.See in the glossary →Close → early refusalRefusalAn explicit "I can't answer": no citation, zero confidence, human review. A state, not a crash.See in the glossary →Close if nothing is found → prompt → model → parserParserThe code that reads the model's output and rejects it if it doesn't match the schema: format, exact fields, types, bounds.See in the glossary →Close → citationCitationThe ID of the document that supports an answer. A correct citation doesn't prove the sentence comes from the document.See in the glossary →Close check → result. The four gates:

1. a cited answer, citation verified;
2. a refusal before any model call;
3. a readable traceTraceThe log of a run: every step (search, model call, tool call, decision) with its arguments and result.See in the glossary →Close;
4. the evaluation gate on the golden setGolden setA small set of reference questions, each labeled with the expected behavior: answer, cite, or refuse.See in the glossary →Close.

The fifth step: the list of issues, ranked by impact.

How I worked

The four gates were already written: I ran them. The fifth, the ranked list, is mine.

The grader first rejected it: “every issue needs a sentence, not a placeholder”. I asked the assistant to write the sentences; it refused, since that was still my rule, and gave me a skeleton with blanks. I wrote ranks 2 and 3 myself, over several rewrites. The assistant only fixed one inaccurate word (“outside tools”). Rank 1 keeps the course’s example.

Before starting, I also had to update the course repo without losing my work: a backup branch, a fast-forward update, then a targeted restore of my files.

What the hand-in contains

• Rank 2. The evaluation gate runs on a fake modelFake modelA deterministic model with scripted answers. It tests the circuit, not the judgment: a green test on it proves the plumbing.See in the glossary →Close, so it shows neither hallucinations nor wrong answers. Impact: the report stays green while nobody sees the errors.
• Rank 3. A real model goes through the network, may never answer, and there’s no timeoutTimeoutA maximum waiting time. Past it, you abandon the call and return a readable refusal, instead of waiting forever.See in the glossary →Close. Impact: the user waits with no answer and no error message.

It’s the impact on the user that sets the rank, not the technical difficulty.

What I keep

• A green test on a fake model proves the plumbing, not the quality of the answers.
• A network call without a timeout can block the user forever, with no error message.
• The impact on the user decides an issue’s rank.

The defense question

Your evaluation is green: is your agent reliable?

Not proven yet: that green comes from a scripted fake model, it shows the circuit works when the model answers perfectly. With a real model, you have to measure wrong answers and cap the wait with a timeout, which I ranked 2 and 3.

Result: 500/500, handed in through PR #545.