Observability for Agents: Patterns That Hold Up
Agents fail quietly, and the only defense is seeing everything they do before it reaches a user.
Blake Aber · Predicate Ventures
An agent is a program that decides its own control flow. That one property breaks most of the monitoring assumptions you brought from conventional services.
A traditional request has a fixed path. An agent request branches on model output, calls tools in an order no one wrote down, and can loop until it runs out of budget. You cannot reason about behavior you cannot replay.
Observability for agents is the discipline of capturing enough of that nondeterministic execution to debug it, score it, and bound its cost. The payoff is not dashboards. It is the ability to answer "why did this run do that" the first time someone asks.
The unit of observation is the trace, not the request
Start by rejecting the log line as your primary artifact. A single agent run produces dozens of model calls, tool invocations, and intermediate decisions. Flat logs lose the causal structure.
The right primitive is a trace made of nested spans. One span per model call, one per tool call, one per reasoning step. Each span carries its inputs, outputs, latency, token counts, and the model version that produced it.
Spans need a parent-child relationship so you can read a run as a tree. When an agent calls a retriever, which feeds a planner, which calls three tools, that hierarchy is the only honest picture of what happened.
Attach a stable run ID that threads through every span, every downstream service, and every emitted event. Without it, correlation becomes guesswork once the run touches more than one system.
Capture inputs and outputs verbatim
The most common regret in agent debugging is sampled or truncated payloads. You find a failure, open the span, and the prompt that caused it was clipped at 500 characters.
Store full prompts and full completions for every model call. Store the exact tool arguments and the exact tool responses. The cost of storage is small against the cost of an unreproducible incident.
Redact at write time for anything sensitive, but redact deliberately. A blanket filter that strips user content also strips the context you need to understand a failure.
Version the prompt templates and tool schemas alongside the traces. When behavior shifts, the first question is whether a template or a model changed, and you can only answer it if both are recorded.
Evaluation runs on the same data you observe
Observability and evaluation are the same pipeline viewed twice. The traces you collect in production are the dataset you score offline.
Define checks that run against completed runs. Did the agent call the tool it was supposed to? Did the final answer cite its sources? Did it stay inside the allowed action set? These are deterministic assertions, and they catch regressions that human review misses.
For judgments that resist hard rules, use model-graded evaluation, but treat the grader as code under test. Pin its version, measure its agreement with human labels, and re-check that agreement when you change it.
Promote a slice of real traces into a regression set. When you ship a new prompt or swap a model, replay that set and compare outputs span by span. A change that improves one case and quietly breaks four is the failure mode this catches.
Guardrails are observable events, not silent filters
A guardrail that blocks an action should emit a span explaining what it blocked and why. Silent interception produces agents that behave strangely for reasons no one can trace.
Put checks at two points. Before a tool runs, validate the arguments against a schema and a policy. After a model responds, scan the output before it reaches the user or the next step.
Record every trigger as a first-class event with the rule that fired and the payload that tripped it. Over time these events tell you which guardrails earn their place and which only add latency.
Treat a guardrail block as a signal about the agent, not only about the user. Frequent blocks on the same action usually mean the prompt or the tool contract needs fixing upstream.
Cost is a dimension of every span
Agents spend money in ways that are hard to predict because they decide how many calls to make. A single run can quietly cost ten times its median when a loop fails to converge.
Record token counts and dollar cost on every model span, then aggregate per run, per user, and per feature. The run-level view exposes the long tail that averages hide.
Set hard budgets per run and emit an event when a run approaches them. An agent that hits its ceiling should fail loudly with a traceable reason, not spin until a timeout cuts it off.
Watch the ratio of tool calls to useful output. A rising ratio is often the first sign of a degraded retriever or a prompt that has started to flail.
The failure modes you are actually watching for
Loops and runaway plans
Agents repeat themselves. They call the same tool with the same arguments, read the result, and call it again. Detect this by comparing successive spans within a run and alerting on repeated signatures.
Silent tool errors
A tool returns an error string with a 200 status, the model treats it as data, and the run continues on a false premise. Parse tool responses for error shapes and mark them in the span rather than trusting status codes.
Context drift
As a run grows, early instructions fall out of the model's effective attention. Track context length per call and correlate long contexts with quality drops in your evaluation scores.
Degradation after a model update
Provider models change under fixed version strings. Keep the regression set running on a schedule so a silent upstream shift shows up as a score delta, not a user complaint.
Build the pipeline before the agent is complex
Instrumentation added after the fact is instrumentation that misses the run you needed. Wrap model and tool calls in tracing from the first prototype, when the overhead is trivial.
The sequence that holds up: trace everything, store payloads in full, score traces offline, surface guardrail and cost events, and alert on the specific failure shapes above. Each layer reuses the data the previous one captured.
An agent you can observe is an agent you can improve. One you cannot is a liability that grows with every deploy.