Agent Reliability
Beyond Successful Runs: How Do We Actually Measure AI Agent Reliability?
Why execution success isn't enough, how to measure agent behavior, and what we're learning while building reliability engineering for AI agents.
AI agents increasingly search for information, invoke tools, make decisions and coordinate workflows. As they take on more responsibility, product and engineering teams face a deceptively simple question: how do we know whether an agent is reliable?
Traditional monitoring gives us useful signals: availability, latency, error rates, resource use and request success. Those measurements still matter. But an agent can return HTTP 200, complete its execution and make an incorrect decision.
It might recommend a product that does not meet the user's requirements, misunderstand a tool response or omit an important limitation. From the infrastructure's perspective, the request worked. From the user's perspective, the task may have failed.
That gap between execution success and behavioral correctness is the reliability problem we are working on at Gentlity.
Define reliability at the behavior level
One of our first product decisions was to avoid treating every completed run as a successful outcome. Completion tells us that execution reached an endpoint. It does not establish that the agent satisfied the user's requirements.
Consider a shopping assistant asked to find an induction-compatible cooking pan under ₹3,000. It searches, compares products and returns three recommendations. Yet the leading recommendation has no verified evidence of induction compatibility.
The execution completed. Whether the recommendation was correct remains a different question.
We need to distinguish three dimensions:
- Execution reliability: Did the agent and its dependencies execute successfully?
- Behavioral reliability: Did the agent follow the requirements and constraints?
- Outcome reliability: Did the user receive a correct and useful result?
These dimensions are related, but they are not interchangeable. An evaluation also needs a stated boundary: a check of a recommendation cannot establish that a purchase was suitable or that a longer-term customer outcome was achieved.
Gentlity approaches this through explicit reliability indicators. An indicator defines the behavior being measured and the evaluation used to assess it. For example, task-success needs a specific definition of the task and its completion criteria. Budget compliance and recommendation grounding may need separate indicators with their own semantics.
The name of an indicator is not the measurement definition. Teams must be clear about what earns a result and what evidence supports it.
Preserve uncertainty with PASS, FAIL and UNKNOWN
Binary success and failure are not enough when evidence is incomplete. A product recommendation might violate a requirement, or its source might simply provide too little information to establish whether the requirement is satisfied.
Our evaluation model keeps those cases distinct.
| Outcome | Meaning |
|---|---|
| PASS | Available evidence supports that the defined requirement was satisfied. |
| FAIL | Available evidence establishes that the requirement was violated. |
| UNKNOWN | Available evidence is insufficient to determine the result. |
UNKNOWN is an explicit representation of uncertainty. Automatically assigning PASS when evidence is missing manufactures confidence; treating every missing fact as a definite failure can distort the measurement in a different way.
This distinction is particularly useful for agents that depend on search, retrieval, external sources or multiple tools. The measurement should not claim more certainty than its evidence supports.
Turn evaluations into a defined reliability measurement
For a specific indicator and scope, determinate reliability uses PASS and FAIL outcomes:
A measurement with no PASS or FAIL evidence has no determinate reliability value. It should not become 100% merely because no failure was observed.
Even when the denominator is nonzero, the percentage needs context. One PASS is not the same body of evidence as thousands of evaluations. One PASS alongside sixteen UNKNOWN results does not establish comprehensive verification of the agent's behavior.
Teams therefore need the indicator definition, evaluation volume and UNKNOWN count alongside the reliability value. Evaluator methodology and scope matter too: results produced under incompatible measurement definitions should not be silently combined.
We also avoid aggregating unrelated indicators into one attractive percentage without an explicit policy. Budget compliance, tool availability, recommendation grounding and task completion answer different questions. Their combined count is not automatically an agent-wide reliability score.
Apply SLOs and error budgets to agent behavior
Site Reliability Engineering offers useful tools for turning measurements into operational objectives: Service Level Indicators, Service Level Objectives and error budgets. Applying them to agent behavior requires explicit evaluation semantics, just as it does for a conventional service.
We tested that workflow by connecting a registered Healthcare agent identity to hosted Gentlity through the Python SDK.
The first run carried one PASS evaluation for task-success. The hosted platform received the evidence and displayed 100% observed reliability. We created an agent-scoped objective for that indicator with a 90% target and a 24-hour rolling window. Its status was Met.
We then supplied two additional synthetic evaluations: one UNKNOWN and one FAIL. Across the three run records, there was now one outcome of each type. UNKNOWN remained separate, leaving one PASS and one FAIL in the determinate denominator.
Observed reliability changed from 100% to 50%. The 90% objective changed from Met to Breached, and remaining error budget became −400%.
The negative budget follows from the defined arithmetic. A 90% target allows a 10% failure rate. One failure among two determinate evaluations is a 50% observed failure rate: five times the allowance. That consumes 500% of the budget and leaves −400% remaining.
This deliberately small sample verified pipeline behavior and calculations: SDK evidence arrived, uncertainty was preserved, an indicator's reliability was calculated and an SLO breach was reported. It was not a statistically meaningful estimate of Healthcare-agent reliability.
Distinguish corrected mistakes from user-visible failures
Agent applications often contain validators, retries, guardrails and fallback logic. Those mechanisms can correct a mistake before the user sees it.
Suppose a shopping agent calls a product-search tool with an invalid argument. The host application rejects the call, provides corrective context and the agent retries successfully. The final answer may be acceptable, while the rejected call remains useful engineering evidence.
Repeated corrected mistakes can expose a fragile model–tool interface, poor instruction following or a dependence on recovery logic. They may also warrant investigation of extra latency or work, without assuming that either was measured in a particular test.
A host-corrected mistake is different from a user-visible failure. Both deserve visibility.
We explored that distinction with Bharat Shopping: actual Shopping-agent execution using a real local Gemma model, deterministic evaluators, the application's core services and synthetic TEST merchant inventory. The evaluation results came from checks performed during that execution, rather than preassigned outcomes used to exercise a telemetry pipeline.
The local validation covered 41 instrumented runs and 1,087 evaluations:
| Evaluation outcome | Count |
|---|---|
| PASS | 1,030 |
| FAIL | 41 |
| UNKNOWN | 16 |
| Total | 1,087 |
Of the 41 FAIL evaluations, 29 were findings about mistakes corrected by the surrounding application before they reached the shopper. Five were user-visible failure findings across two Shopping conversation turns. Other findings concerned guarded or disclosed conditions, while two remained unclassified.
These are evaluation findings, not necessarily distinct failed runs. A run can contribute more than one evaluation. The two unclassified failure findings are also separate from the sixteen evaluations whose outcome was UNKNOWN.
The checks identified issues such as missing material caveats, rejected tool calls, truncated responses and recommendations with insufficient evidence. These combined counts describe the evaluation activity; they should not be converted into an aggregate reliability claim across unrelated indicators.
Looking only at a final answer would obscure some of the behavior that the surrounding application had to correct. Conversely, counting every recovered mistake as a failed customer interaction would misrepresent what the user experienced.
Instrument execution without redefining business behavior
The Shopping work sharpened several product requirements for reliability instrumentation.
Use execution evidence for behavioral measurements. Synthetic PASS, FAIL and UNKNOWN events are useful when validating the telemetry pipeline. They do not become evidence about an agent's real behavior. The Shopping integration instead connected deterministic checks already performed by the application to the SDK.
Keep monitoring failures separate from application results. Our design goal is that a telemetry destination becoming unavailable should not, by itself, stop the Shopping agent from answering. Delivery problems should remain observable through bounded local diagnostics rather than silently changing the business result. The local validation is not a guarantee that every possible outage or execution failure has been exercised.
Keep telemetry boundaries explicit. The integration uses allowlisted indicator names, evaluator identifiers and reason codes. Raw conversations, merchant secrets, personal information and unrestricted product content are not needed for these structured reliability events. This local evidence is not a claim of validated hosted Shopping delivery or a broader security certification.
Attribute each evaluation to the right execution. During development, we identified and corrected a run-context attribution issue that could associate some Shopping evaluations with another agent's run. The lesson matters beyond that defect: ambiguous execution identity undermines the meaning of the resulting measurements.
Preserve the reason for an outcome. An invalid tool call, an unsupported product attribute and an incomplete response can require different engineering actions. A useful evaluation should retain that distinction instead of producing an unexplained FAIL.
Reliability needs more than one number
A Shopping agent may maintain a strong user-visible completion rate while repeatedly making invalid tool calls that its host corrects. Those signals tell us different things: one concerns the current user experience; the other points to engineering work worth investigating.
A reliability view should help teams ask:
- Which behavior was evaluated, and under what definition?
- What evidence supported PASS, FAIL or UNKNOWN?
- Was a mistake corrected before it reached the user?
- Did a result breach an objective for the relevant agent and indicator?
- How many determinate and uncertain observations support the measurement?
These questions make a percentage useful. Hiding them behind an unexplained aggregate score makes the result harder to act on.
What we are building toward
The hosted Healthcare-labelled validation exercised the evidence, evaluation, measurement and SLO workflow with controlled synthetic outcomes. The Shopping work showed how the SDK can capture evaluations derived from actual agent execution in a local environment.
The next step is to validate those Shopping evaluations against hosted Gentlity with controlled test data and appropriate authorization boundaries. That remains a next step, not a completed hosted integration.
From there, we want reliability evidence to support better operational decisions: recognizing recurring failure patterns, understanding recovery mechanisms and detecting degradation against explicit objectives. Those are directions for development, not claims that the local tests have established production readiness.
The product question is larger than whether an agent is running: can we trust how it behaves, and do we have the evidence to support that trust?
As agents take on more responsibility, execution success will remain necessary. Teams will also need explicit behavioral measurements, visible uncertainty and objectives whose calculations they can explain.
Explore the Gentlity product, the open-source Python SDK on PyPI (opens in a new tab) and its source and documentation on GitHub (opens in a new tab).