BAA 101 · Chapter 08
Read the tool call, not the answer
The principle
To understand what an AgentA program that uses a language model to decide what to do (search, call a tool, answer, refuse), in a loop, with stopping rules.See in the glossary → did, you read its TraceThe log of a run: every step (search, model call, tool call, decision) with its arguments and result.See in the glossary →: which tool, with which arguments, for which raw result. The final answer is just plausible text.
Why it matters
A model apologizes convincingly, and a good answer can hide a bad path. Without a trace, you fix things at random. With a trace that holds secrets, you create a leak.
The mechanism
- Trace every step: the search, the model call, the Tool callThe model's request to run a tool with specific arguments. That's what you read in the trace, not the final answer.See in the glossary →, the decision, with their arguments and results.
- Isolate the layer: test the tool alone, then through the agent, before looking further.
- Classify every failure before fixing it (search, tool choice, instruction not followed, format), and handle the most frequent first.
- Mask secrets by replacing them with a stable label, never by deleting them: a hole looks like an absence.
- Be able to say in one sentence which event in the trace decided the result.
In the field
- AWS Scholars, chapter 2: isolate the layer first, then read the logs, which show the tool’s name, its arguments and the raw result, where the model only apologizes.
- AWS Scholars, chapter 3: the trace showed an agent launched before the one it depended on, then, on a correct RefusalAn explicit "I can't answer": no citation, zero confidence, human review. A state, not a crash.See in the glossary →, an OrchestratorIn a multi-agent system, the agent that never answers itself: it routes the work to specialist agents.See in the glossary → that had skipped all its rules.
- Dev3Pack, Session 9: the trace tells a document found but ignored by the model from a document never found, two opposite fixes. And secrets go into it masked.
The question to ask
Which line of the trace decided this result?