Evaluating AI agent reliability before production

Traditional software either works or throws. AI agents fail differently: they produce a confident, well-formatted, plausible answer that is wrong. Nothing crashes, no exception is raised, and the failure only surfaces when a customer reacts to it. That is why 'we tested it and it seemed good' is not a release process.

This is the methodology we use to decide whether an agent is ready to touch production, and what has to be in place around it when it does.

Define reliability before you measure it

Reliability is task-specific. Write down, for your agent, what a correct outcome is and what an unacceptable one is. Without that, evaluation devolves into opinion.

• Task success — did it accomplish the specific job (classified correctly, extracted the right fields, produced a usable draft)?
• Groundedness — is every factual claim supported by retrieved context or tool output, rather than invented?
• Format validity — does the output parse against the schema every single time?
• Safe refusal — does it decline and escalate when it lacks the information to act?
• Tool correctness — did it call the right tool, with the right arguments, at the right time?
• Latency and cost — is it inside the budget the workflow needs?

Build the test set

A test set assembled from happy-path examples tells you nothing useful. Aim for roughly 100–200 cases for a first production agent, deliberately weighted toward the ways it will be attacked and confused.

• Golden cases — real historical inputs with human-verified correct outcomes. Harvest these from actual tickets, emails, or CRM records.
• Edge cases — missing fields, wrong language, huge inputs, empty inputs, contradictory information.
• Adversarial cases — prompt injection in the user content, instructions embedded in a retrieved document, attempts to extract the system prompt.
• Out-of-scope cases — questions the agent must refuse rather than attempt.
• Regression cases — every bug you have ever fixed, kept forever so it cannot come back.

Failure modes to test for explicitly

• Hallucinated facts — invented prices, policies, availability, or capabilities.
• Schema drift — output that parses on Tuesday and fails on Thursday after a model update.
• Silent truncation — long inputs cut off mid-context, producing answers based on half the story.
• Tool misuse — calling a write tool during what should be a read-only turn, or calling the same tool in a loop.
• Prompt injection — instructions inside user content or retrieved documents overriding the system prompt.
• Over-refusal — escalating so often that the automation delivers no leverage.
• Context leakage — surfacing another customer's data because retrieval was not filtered by identity.
• Non-determinism — the same input producing materially different actions across runs.

Evaluation methodology

Run it like a test suite

Evals belong in CI, triggered on every prompt change, model change, tool change, and retrieval change. Any of those four can move behavior; treating only prompt edits as changes is how regressions ship.

Graders

• Deterministic graders first — schema validation, required fields, forbidden strings, tool-call assertions. Cheap, fast, and unarguable.
• Model-as-judge for subjective quality (tone, helpfulness), but calibrate it against human labels on a sample before you trust it.
• Human review on a rotating sample, permanently. Automated graders drift; humans catch what you did not think to grade.

Report the right numbers

Aggregate accuracy hides the failures that matter. Report per-category pass rates, and treat any adversarial or safety category as pass/fail rather than as an average.

Guardrails around the agent

Evals tell you how the agent behaves. Guardrails limit what a bad behavior can do. You need both.

• Structured output validation on every response, with a defined fallback path when parsing fails.
• Least-privilege tools — read-only unless a write is genuinely required, scoped to specific records.
• Allow-list of permitted actions; anything else escalates instead of improvising.
• Human approval for irreversible actions: external messages, payments, deletions, contractual statements.
• Rate and loop limits — max tool calls per run, max runs per record, max spend per day.
• Content filters on both input and output, plus PII redaction before logging.
• Confidence thresholds that route uncertain cases to review rather than guessing.

An agent without guardrails is not fast — it is just failing somewhere you have not looked yet.

Monitoring in production

Your eval set is a snapshot; production is the distribution. Track the signals that reveal drift between them.

• Schema failure rate — should be effectively zero; any increase means a model or prompt change slipped through.
• Escalation rate — rising means the world changed, falling sharply means the agent may have started guessing.
• Human edit rate on drafts — the best available proxy for quality, straight from the people doing the work.
• Tool error rate and retry counts.
• Latency p50/p95 and cost per run.
• Input distribution shift — new topics or languages appearing that your test set does not cover.
• A weekly sample of full traces reviewed by a person who can change the prompt.

Rollback

Assume you will need to reverse a release quickly. That requires three things decided in advance.

• Versioned artifacts — prompt, model id, tool definitions, and retrieval config pinned together and deployable as one unit.
• A kill switch that reverts the agent to draft-only or fully off without a code deploy, usable by an operator.
• Documented reversal for side effects: which writes can be undone, which cannot, and who approves the cleanup.
• Staged rollout — a percentage of traffic first, with automatic rollback triggers on schema failure and escalation spikes.

Production-readiness scorecard

Score each line yes or no. Any no in the first six is a blocker, not a note for later.

• 1. Written definition of success and unacceptable failure for this agent.
• 2. Test set of 100+ cases including edge, adversarial, out-of-scope, and regression cases.
• 3. Schema validity at 100% across the suite, with a defined fallback on parse failure.
• 4. Guardrails: least-privilege tools, action allow-list, human approval on irreversible actions.
• 5. Escalation path to a named human owner, with a response expectation.
• 6. Kill switch reachable by an operator without a deploy.
• 7. Evals wired into CI on prompt, model, tool, and retrieval changes.
• 8. Consistency measured across repeat runs, not just single-shot accuracy.
• 9. Cost and latency budgets defined and enforced per run.
• 10. Structured logging with a correlation id and PII redaction.
• 11. Production dashboards for schema failures, escalation rate, and edit rate, with alerts.
• 12. Staged rollout plan with rollback triggers.
• 13. Versioned prompt/model/tool artifacts deployable as one unit.
• 14. Scheduled human review of a production trace sample.
• 15. Owner named for the agent, with a review cadence in the calendar.

If you are wiring the agent into a system of record, the integration side of this is covered in wiring a CRM to an AI agent with webhooks, and the operational cost side in cost-modelling LLM workflows.

Need this built for you? We scope, build, and hand over automation systems as a done-for-you engagement — see what an AI automation agency actually delivers, the published pricing ranges, or book a 30-minute call.