AI Agent Observability: How to Know Your Agent Is Broken
The single most common failure mode for AI agents in 2026 is silent degradation. The agent shipped, it worked for two months, and then it slowly began to give worse outputs without anyone noticing. By the time the team checks, the agent has been producing wrong reconciliations or bad triage decisions for weeks. The reputational cost is high; the cleanup is expensive.
The fix is observability. Production AI agents need the same monitoring discipline that production microservices have had for a decade. This article is the playbook Semnexus uses to make sure an agent's failures are caught early — what to log, the six signals that indicate failure, the human-review loop that compounds, and the mistakes that produce false confidence.
Why agents fail silently
Traditional software fails loudly. A null pointer exception throws; a database timeout returns an error. Engineers see the failure in logs within minutes.
Agents fail quietly. They return confidently-worded output that is wrong. They make a tool call that succeeds at the API level but achieves the wrong outcome. They drift from their original behavior as inputs distribute differently from what was tested. None of these produce a standard error log.
Observability is the discipline of catching these silent failures before they accumulate.
The 6 signals that an agent is failing
The signals to monitor, ranked by how often they catch real problems:
1. Output schema drift
When the agent's output starts deviating from the expected schema (missing fields, malformed types, unexpected new fields), the underlying model is responding differently than it did at deployment. The drift is usually slow and small individually, but compounds.
How to monitor. Validate every agent output against a strict schema. Log validation failures. Alert when failure rate exceeds 1% over a rolling 24-hour window.
2. Tool call failure rate
The agent makes tool calls (API calls, database queries, web searches). Track the success rate of each tool call type. A rising failure rate on a specific tool indicates a model issue, an API issue, or a context issue — investigate all three.
How to monitor. Log every tool call with input, output, and result. Alert on a 20% degradation week over week.
3. Token consumption drift
If the agent is suddenly consuming 50% more tokens per task, it is either thinking more about each task (often a sign of confusion) or returning more verbose output (often a sign of drift). Either way, investigate.
How to monitor. Track tokens per task type. Alert on 30% rises over a rolling 7-day window.
4. Human override rate
When the agent operates with human review, track the rate at which humans override or correct the agent's output. A rising override rate is the strongest single signal that the agent's reliability is decaying.
How to monitor. Capture override events with the human's correction. Compare week over week. Alert at 20% rise.
5. Latency distribution
Average latency hides problems. Look at p50, p95, and p99. Tail latency rises when the agent is hitting rate limits, when retries are occurring, or when the prompt is generating responses that require multiple LLM calls.
How to monitor. Log latency per run. Alert on p95 above target SLA.
6. Downstream outcome metrics
The truest signal is whether the work the agent produced led to the right downstream outcome. For a sales-triage agent, did the routed leads close at expected rates? For a reconciliation agent, did the reconciled transactions stay reconciled through close?
How to monitor. Tie agent outputs to downstream business metrics. Run weekly retrospectives.
What to log per agent run
Every agent run should produce a structured log with at minimum:
| Field | Description |
|---|---|
| Run ID | Unique identifier |
| Started at | Timestamp |
| Input | Full input (subject to privacy filters) |
| Model used | Exact model and version |
| Prompt version | Version of the system prompt at run time |
| Token in / Token out | Per call and aggregate |
| Tool calls | Each tool call with input, output, latency |
| Final output | The agent's final response |
| Schema validation | Pass / fail with detail on failure |
| Outcome (if known) | Whether the run was correct, when ground truth is available |
Logs should be queryable. The right tool is whatever your team already uses for observability (Datadog, Honeycomb, Grafana, or a 2026 LLM-native equivalent like Langfuse or LangSmith).
The human-review loop that compounds
A production agent needs a sample of its outputs reviewed by humans, indefinitely. The pattern:
Sample 5–10% of runs randomly
Random sampling catches drift across the whole input distribution.
Sample 100% of runs flagged by the schema or downstream signal
Flagged runs deserve review every time.
Have a defined reviewer role
One person whose part-time job is reviewing the samples. Without a named owner, reviews drift to nobody.
Capture the review outcome
Was the output correct? If not, what was wrong? The review database becomes the dataset for future prompt updates.
Run a weekly retrospective
Look at the trends across the week's reviews. Are there patterns in failures? The retrospective is where prompt-update decisions come from.
Alert rules that work
The five alerts every production agent should have:
- Schema validation failure rate above 1% over 24 hours.
- Tool call failure rate above 5% on any tool over 24 hours.
- Token consumption per task type rising 30%+ week over week.
- Human override rate rising 20%+ week over week.
- Downstream outcome metric falling beyond pre-defined threshold.
Alert fatigue is real. Five alerts is the working maximum; more produces ignored alerts. The five above catch most agent failures we see in production.
What good observability looks like at maturity
A mature production agent has:
- Every run logged with structured fields
- A dashboard showing the six signals, refreshed daily
- The five alerts wired into the team's on-call channel
- A defined reviewer for the human-review loop
- A weekly retrospective with at least one action item per week
- A prompt-version-history with the rationale for each change
This is not heavyweight. A small team can stand up the full observability layer in 2 to 4 weeks for a new agent.
Five mistakes that produce false confidence
The mistakes that show up most often in audits:
- Logging only successful runs. The failures are where the learning is; logging only successes guarantees the team is blind.
- Trusting the agent's self-reported confidence. LLMs are unreliable at estimating their own correctness. Trust the schema validation and downstream outcomes instead.
- Reviewing only flagged runs. Random sampling catches problems flagged systems miss.
- No prompt version tracking. When the agent's behavior changes, you need to know whether the prompt changed or the model drifted. Without versioning, you cannot tell.
- Treating the human-review loop as optional. It is not optional. Agents without a review loop in 2026 are agents that will silently degrade until something visible breaks.
Frequently asked questions
When should I add observability to an agent — before or after it ships? Before. An agent without observability should not ship to production. Adding it after the fact is more expensive and produces a window of un-observed runs.
Do I need a separate observability tool, or can I use my existing one? Existing observability tools (Datadog, Honeycomb, etc.) work fine with light LLM extensions. LLM-native tools (Langfuse, LangSmith, Helicone) add value at scale because they understand prompt versioning and token economics natively.
How often should I review the agent's prompt? Re-read the prompt every 30 days; update if the data from the review loop suggests changes. Most prompts stabilize after the first 90 days of operation but should be revisited quarterly.
What if my agent provider does not give me detailed logs? You need to add a logging layer around your provider calls. If you are using a hosted agent platform, evaluate whether their logs meet the criteria above. Many do not.
How does this apply to multi-agent systems? Multiply the observability work by the number of agents, plus add cross-agent communication logs. Multi-agent systems fail in more ways than single agents.
If your AI agent is in production without observability or you are about to ship one, the AI app development team at Semnexus builds observability layers as part of every agent engagement. The business mobile consulting team handles the operational design of the human-review loop and the weekly retrospective discipline.
