Building Your First Agent Evaluation Suite: From Zero to Regression Tests

You have an agent in production. It works. Most of the time. But "most of the time" is not a quality metric — it is a hope dressed up as confidence. This guide walks through building an evaluation suite that turns "it seems to work" into "X% of test cases pass, and here is what fails."

Why you need an eval suite before you think you do

The most common objection to building eval suites is "we are too early." This is backwards. You need an eval suite most urgently when you are early, because that is when you are changing things fastest — prompts, models, tools, system messages. Each change can regress behavior, and without a baseline to compare against, you will not know.

The second objection is "we do not have enough data." You do not need much. Ten well-chosen test cases catch most regressions. Fifty catches nearly all. A thousand is for mature systems with subtle edge cases. Start with ten.

The third objection is "we do not have time." An eval suite takes one afternoon to build and saves you from shipping a regression to production. The math is simple.

The five components of an eval suite

1. Test cases

A test case is an input paired with a way to check the output. There are four levels of checking, from cheapest to most expensive:

Check type	Example	Effort	What it catches
Substring match	Output contains "per million tokens"	Trivial	Missing required information
Tool call verification	Agent called web_search before answering	Easy	Wrong tool selection, skipped steps
Rubric scoring	Output is accurate, cited, and not hallucinated (1-5)	Moderate	Quality, tone, completeness
LLM-as-judge	A second model scores the output against criteria	Higher	Subjective quality, safety, alignment

Start with substring match and tool call verification. These are cheap, fast, and catch 80% of regressions. Add rubric scoring when you have a stable agent and want to track quality trends. Add LLM-as-judge last, when you need to catch subtle quality issues that rules cannot detect.

2. Execution harness

This is the code that runs your test cases against the agent. It sends each input, captures the output, tool calls, tokens, cost, and latency, and writes results to a file. See our Agent Eval Runner skill for a minimal implementation.

The harness must capture:

• The full input (including system prompt and context)
• The full output
• Every tool call made (name, arguments, result)
• Token usage (input, output, total)
• Cost per case
• Latency per case
• A trace ID for cross-referencing with observability

3. Metrics

Track these for every run:

• Pass rate: X out of Y test cases passed
• Cost per case: Average and p95 token cost
• Latency: Average and p95 response time
• Failure categories: Why did cases fail? (wrong tool, missing info, over budget, timeout, hallucination)
• Regression rate: Compared to the last run, did any previously-passing cases now fail?

The regression rate is the most important metric. A 90% pass rate that was 95% last week is a regression. A 90% pass rate that was 88% last week is progress.

4. Baseline

Run your eval suite against your current production agent. This is your baseline. Save the results. Every future change — model upgrade, prompt edit, tool addition — gets compared against this baseline. If the new version does not beat or match the baseline on pass rate, do not ship it.

5. CI integration (eventually)

Once your suite is stable, integrate it into your deployment pipeline. Every pull request that touches agent code runs the eval suite. If pass rate drops below the threshold, the build fails. This is the goal — but do not start here. Start with a script you run manually, then automate once you trust the tests.

Designing good test cases

Good test cases share these properties:

Representative. They reflect real inputs, not edge cases you invented. Pull from your actual agent logs (if you have them) or from real user queries.

Diverse. Cover different task types, not variations of the same task. If your agent does research, coding, and summarization, your suite needs cases for all three.

Deterministic in checking, not in output. The check should be deterministic ("contains this string," "called this tool"). The output does not need to be deterministic — LLMs are non-deterministic. Design checks that work across valid output variations.

Budgeted. Each case has a max token and max cost threshold. An agent that passes but costs 10x the budget is a failure. Token cost is a quality metric, not just a financial one — high token usage often means the agent is confused or looping.

Versioned. When you update a test case (because the expected behavior changed), record what changed and why. Otherwise you cannot distinguish "the agent regressed" from "the test changed."

What an eval suite does not replace

An eval suite does not replace:

• Human review of outputs. Tests catch regressions; humans catch quality. Both are needed.
• Observability. Your eval suite runs on test inputs. Production monitoring shows you what real users experience. You need both.
• Red-teaming. Eval suites test known behavior. Red-teaming finds unknown failure modes. An eval suite that passes 100% tells you nothing about inputs you did not think to test.
• User feedback. The ultimate test is whether users find the agent useful. Eval suites are a proxy, not a replacement.

The evaluation loop

1. Define 10 test cases
2. Run against current agent → save baseline
3. Make a change (new model, new prompt, new tool)
4. Run against modified agent → compare to baseline
5. If pass rate dropped → investigate failures, fix or revert
6. If pass rate held or improved → ship the change
7. Add new test cases for any failure mode you discover in production
8. Repeat weekly

This loop is the difference between teams that improve their agents over time and teams that break their agents over time. The eval suite is not a one-time build — it grows as your agent grows, and every production failure should result in a new test case that would have caught it.

Tools that help

• Agent Eval Runner skill — our minimal eval harness for Hermes agents
• Langfuse — tracing and eval platform with built-in scoring
• Pydantic AI evals — type-safe eval framework integrated with Pydantic AI
• Ragas — RAG-specific evaluation pipeline
• pytest — for agent test harnesses using familiar Python tooling

Bottom line

An eval suite is the single highest-leverage investment for any team running agents in production. It costs one afternoon to build, catches regressions before they ship, and compounds in value as your agent evolves. If you are not running evals before every deployment, you are deploying blind. Build the suite. Run it. Ship with confidence.