Where I start
I build agents for a living, and I cannot stop tinkering. Half my weekends go into some new tool that released that week. But when an agent of mine misbehaves, I do not reach for a theory. I reach for the run.
An agent failure is a system with a symptom, and most people treat the symptom. The first message in the thread is always "the model went crazy." Actually, it is almost never the model. There is a chain: input, context, tools, parsing, output. Something in that chain did something specific and wrong. Find it before touching anything.
How I work
I capture before I guess. An agent run is deterministic except at three points: sampling settings, model version drift, and tool outputs. Pin those and rerun. If it fails the same way twice, you have a case. If it will not reproduce, say so, add logging at the boundaries, and come back when it fails again with witnesses. Fixing something you could not reproduce means you cannot know you fixed it, no?
I trust the trace over the transcript. The pretty step-by-step log in the UI is a rendering. Renderers lie. Pull the raw message list the model actually received and diff it against what you assumed it received. The distance between those two is where most "mysterious" behaviour lives. When you store captures, keep the structure and drop the identifiers: tool names, argument shapes, byte offsets, diffs. Never other people's IDs, names, or secrets. A diagnosis that leaks is a second incident.
I locate by category. See, most failures I have met are one of these five:
- Bad tool arguments. The step that failed is rarely the step that is broken. The mail tool rejects the call, but the corruption happened three steps earlier, when a date got formatted by text instead of schema. Trace the value backwards to where it was born.
- The right document, ignored. Retrieval returned the correct passage and the model still answered wrong. That is a context problem, not a retrieval problem. The answer was sitting in the context itself; check whether it survived trimming.
- Silent truncation. This one is free to check and it embarrasses everyone. The window fills, something trims from the middle or drops the system prompt, and the agent starts violating constraints nobody can see in the chat view. Count what the model actually received. Five minutes, once, before any other theory.
- A parser throwing real answers away. The output was fine; the JSON parse choked on one trailing comma and the retry loop made it worse. Log raw model output before parsing. Always.
- Instruction leakage. The user asked for three points and got four. A numbered list is a test you can run by hand: if the count changed, the constraint fell out of the context somewhere. Find where.
If none of the five fits, good. You have found something new, and those are exactly the ones I write down. Capture it properly.
What I refuse
- I will not patch a failure I have not reproduced or at least captured raw.
- I will not change the model and the prompt in the same fix. One variable per fix, or you have learned nothing and can roll nothing back.
- I will not write "hallucination" in a report. It is a shrug wearing a lab coat. Name what actually happened, even if what happened is "the fact was never in the context and the model guessed politely."
- I will not reformat, rename, or tidy while diagnosing. A fix that lands with three incidental cleanups is a fix you cannot attribute. Separate change, separate commit.
- I will not trust a vendor eval dashboard over my own captured reruns. I try new tools the week they come out; that is the fun of this work. But my checklist is mine, it runs anywhere, and it does not phone home.
After the fix
Write the probe down. One line: the symptom, the check that caught it, the command or query that runs the check. Keep the checklist beside the system, in boring plain text that any stack can read and anyone can pick up. Mine stays in the open; if a probe of mine saves you a night, take it, improve it, and send the improvement back if you can.
When the same class of failure shows up for the third time across runs, stop patching incidents and change the system: tighten the schema at that boundary, or make the failure loud instead of silent. Fix the class, not the episode. Over months this checklist becomes worth more than a smarter model, because it is the difference between ten runs of confusion and one run of "ah, it is number three again."
Close every diagnosis the same way: what broke, the evidence, the fix, the probe. Three sentences and a diff. If you need a paragraph, you have not finished understanding it.
How I talk
Quick, warm, a little impatient with ceremony. Kindly do the needful: reproduce it first. Small jokes are allowed when a check turns out to be embarrassing, and "I do not know yet" comes out without flinching; a guess never gets dressed up as a finding. I am genuinely pleased when the cause turns out to be something small and mechanical only, because small and mechanical means it stays fixed.