The reason most AI agent prototypes never reach production is not the model. It is that nobody can prove the agent will not regress on the next prompt change. An eval harness is the gate that turns “the demo worked on my laptop” into “the agent is still passing 94% of its golden tests after the Sonnet swap.” This post covers what a golden-test suite looks like for an agent, how to wire it into CI, how to run a drift-detection cron against production traffic, and how to tell a real regression from a flake.
An eval harness is an automated test suite that runs realistic inputs through the agent and scores the outputs against expected traits. The unit of an eval is not a single LLM call. It is a complete agent trajectory: the user message, the tool calls the planner chose, the tool outputs, the retries, and the final response. A scoring function judges whether that trajectory satisfies the expected behavior.
That definition is more demanding than what most teams ship as their first eval. The first attempt is usually “give the model a prompt, check the response matches a string.” That works for a single completion, not for an agent that calls eight tools per turn and produces a structured response with optional fields. You end up with brittle string matches that break on a comma change and tell you nothing about which tool failed.
A real agent eval suite scores on traits, not exact outputs. Did the agent call lookup_patient exactly once? Did it never call prescribe_medication? Did the structured output validate against the schema? Did every cited fact have a source_span that appeared in the input? Each trait is a separate scoring function. A single trajectory might pass twelve traits and fail one; the failing trait is the actionable signal.
LangSmith’s evaluation docs put the starting point at “5 to 10 examples of what ‘good’ looks like for each critical component,” focused on agent behaviors like tool selection and argument formatting, not exact response strings. (LangChain, “LangSmith evaluation concepts”) Five examples per capability is not a typo. The goal of the first version is to make the gate exist; you expand the dataset later.
A golden-test suite for an agent has three layers: happy paths, edge cases, and adversarial cases. Each answers a different question.
Happy paths answer “does the agent do its core job on clean inputs.” For an intake agent, a clean patient note, a complete history, an unambiguous chief complaint. The expected trajectory calls the canonical tools, validates against the schema, returns the expected fields. If happy paths regress, the system is broken end to end.
Edge cases answer “does the agent handle realistic but uncomfortable inputs.” Long inputs near the context window. Multilingual content. Ambiguous chief complaints. Missing fields. Contradictory medication lists. Realistic noise in production data. Edge cases are where prompt-engineering improvements show up first and where regressions hide longest.
Adversarial cases answer “does the agent fail safely on hostile inputs.” Prompt injection in user input. Tool-output strings that look like instructions. Attempts to call out-of-scope tools. Inputs designed to make the agent fabricate a structured field. Adversarial cases are where compliance reviewers focus and where the most expensive production incidents originate.
For a first version, the target distribution: 5 happy paths, 5 edge cases, 3 adversarial cases per agent capability. An agent with three capabilities (extract, summarize, classify) gets 39 golden tests. Two engineer-days to build, runs in roughly 60 seconds on a CI runner, catches the kinds of regressions a manual review misses for weeks.
A single golden test case has four parts: the input, the expected trajectory, the expected output, and the scoring functions. As code:
{
"id": "intake_happy_001",
"category": "happy_path",
"capability": "extract",
"inputs": {
"transcript": (
"Pt is 47 yr old female. Allergies: PCN, sulfa. "
"On metformin 500mg BID. Reason for visit: f/u DM."
),
},
"expected_trajectory": {
"tools_called": ["record_demographics", "record_allergies",
"record_medications", "record_chief_complaint"],
"tools_forbidden": ["prescribe_medication", "schedule_followup"],
"max_retries": 1,
},
"expected_output": {
"age": 47,
"sex": "F",
"allergies": ["penicillin", "sulfa"],
"medications": [{"name": "metformin", "dose": "500mg",
"freq": "BID"}],
"chief_complaint_keyword": "diabetes",
},
"scoring_functions": [
"schema_conformance",
"allergy_recall",
"medication_recall",
"no_forbidden_tools",
"no_fabricated_fields",
],
}
The case is not a regex on the response. It is a contract on the trajectory. The scoring functions are independent. A run that gets the medications right but misses an allergy fails allergy_recall and passes the rest. You see exactly what regressed and which case revealed it.
A scoring function takes (inputs, outputs, reference_outputs) and returns a score. Two rules.
First, scores are graded, not boolean. allergy_recall returns intersection size divided by expected size, not a 0 or 1. Graded scores let you set a regression threshold (the suite passes if allergy_recall stays above 0.92) instead of demanding perfection. Boolean scores produce a noisy CI that fails on the smallest realistic variation and teaches the team to skip the suite.
Second, the scoring functions you can implement deterministically should be. Schema conformance, tool-call recall and precision, latency budgets, token budgets, forbidden-tool checks, source-span substring validation: all of these are code, not LLM calls. Save LLM-judged scoring for genuinely subjective dimensions like response coherence and tone match.
As code, the pattern is:
from jsonschema import validate, ValidationError
def schema_conformance(inputs, outputs, reference_outputs):
try:
validate(outputs, schema=CHART_SCHEMA)
return {"key": "schema_valid", "score": 1.0}
except ValidationError as e:
return {"key": "schema_valid", "score": 0.0, "comment": str(e)}
def allergy_recall(inputs, outputs, reference_outputs):
expected = set(reference_outputs["allergies"])
actual = set(outputs.get("allergies", []))
score = len(expected & actual) / len(expected) if expected else 1.0
return {"key": "allergy_recall", "score": score}
def no_forbidden_tools(inputs, outputs, reference_outputs, trajectory):
forbidden = set(reference_outputs["expected_trajectory"]["tools_forbidden"])
called = {step["tool_name"] for step in trajectory if step["op"] == "tool"}
score = 0.0 if forbidden & called else 1.0
return {"key": "no_forbidden_tools", "score": score,
"comment": f"called={called & forbidden}"}
def source_span_fidelity(inputs, outputs, reference_outputs):
transcript = inputs["transcript"]
spans = [item.get("source_span", "") for item in outputs.get("allergies", [])]
in_source = sum(1 for s in spans if s and s in transcript)
score = in_source / len(spans) if spans else 1.0
return {"key": "source_span_fidelity", "score": score}
Each function is small, deterministic, and independently inspectable. A regression on source_span_fidelity tells you the agent started making up citations. A regression on no_forbidden_tools tells you the planner started reaching for tools it should not. Specific scoring functions produce specific signals.
The eval harness only earns its keep when it runs on every change. A CI job runs the suite on every pull request and gates merge on results.
# .github/workflows/agent-evals.yml
name: agent-evals
on:
pull_request:
paths:
- "agents/**"
- "prompts/**"
- "tools/**"
schedule:
- cron: "0 8 * * 1" # Monday 08:00 UTC drift sweep
jobs:
golden:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@v4
- uses: actions/setup-python@v5
with: {python-version: "3.12"}
- run: pip install -r requirements.txt
- name: Run eval suite
env:
ANTHROPIC_API_KEY: ${{ secrets.ANTHROPIC_API_KEY }}
LANGSMITH_API_KEY: ${{ secrets.LANGSMITH_API_KEY }}
run: |
python -m agent.evals.run
--dataset golden_v1
--thresholds thresholds.yaml
--emit junit.xml
- name: Publish results
if: always()
uses: dorny/test-reporter@v1
with:
name: agent-evals
path: junit.xml
reporter: java-junit
Two design choices. First, gate on paths. The suite runs when agent code, prompts, or tools change; doc changes do not trigger it. This keeps the budget bounded and the noise low. Second, thresholds live in a config file checked into the repo, not hardcoded in the runner. A threshold change is a deliberate diff reviewed by humans.
The thresholds config is the regression policy:
# thresholds.yaml
golden_v1:
schema_valid: {min: 1.00, allow_below: 0} # never regress schema
allergy_recall: {min: 0.92, allow_below: 1} # one outlier OK
source_span_fidelity: {min: 0.95, allow_below: 0}
no_forbidden_tools: {min: 1.00, allow_below: 0}
no_fabricated_fields: {min: 0.98, allow_below: 1}
latency_p95_ms: {max: 8000}
cost_p95_cents: {max: 200}
A failing PR is now a specific message: “allergy_recall dropped to 0.87 on case intake_happy_004, below threshold 0.92.” That is debuggable. “The eval suite is red” is not.
Golden tests catch regressions you introduce. They do not catch drift you did not introduce. Drift is the slow degradation that happens when the world changes around a static agent: provider model updates, retrieval corpus updates, third-party tool API changes, distribution shifts in production inputs. The eval suite passes every Monday and the agent gets quietly worse anyway.
The pattern we ship: a weekly cron samples 200 recent production traces (anonymized to the same standard as the golden set), re-runs the agent against those inputs in a sandbox, and compares the new outputs to the historical outputs. Differences above a threshold trigger an alert and a triage queue. 200 traces cost a few dollars in re-inference; triage is half a day of an engineer’s time per month. The cost of skipping it is a four-week silent regression you find when a customer complains.
The four signals a drift cron should track:
NIST’s AI Risk Management Framework names “Measure” as one of its four core functions for exactly this reason. (NIST, “AI Risk Management Framework”) You cannot manage what you do not measure, and AI systems drift in ways traditional software does not.
LLM calls are non-deterministic. Even with temperature pinned, a small fraction of runs produce different outputs on identical inputs. A suite that fails on a single case after one run is not a regression; it is a sample of size one.
How to distinguish:
Teams turn off their eval suite when it is too noisy. The fix is almost never “lower the threshold.” The fix is “increase the sample size, pin a baseline, and stop alerting on case-level noise.”
A golden-test suite is not a guarantee. An agent that passes every test can still fail in production on inputs the suite did not cover. The suite is a floor, not a ceiling. It catches the regressions you introduced and the failures you anticipated.
The failures you cannot anticipate (novel adversarial inputs, real-world distribution shifts, downstream tool changes you do not control) are caught by observability, not the eval suite. The 5-layer production architecture this post is part of names observability as Layer 2 for exactly that reason; we cover it in the cornerstone post on production AI agents.
The team that ships an agent with no eval harness is shipping a system nobody can prove is still working. The team that ships one with a 39-case golden suite, CI gating, and a weekly drift cron has the minimum infrastructure to debug, defend, and improve the system over the months that come after launch. From Prompt to Production is the work between those two states.
How do you write an eval harness for an LLM agent?
Define 5 to 10 golden test cases per agent capability. Each case carries an input, an expected trajectory (tools called, tools forbidden, max retries), an expected structured output, and a list of scoring functions. Scoring functions return graded scores, not booleans. Gate CI on per-metric thresholds checked into a config file. Run the suite on every PR that touches agent code, prompts, or tools.
What is the difference between a golden-test suite and a unit test suite?
A unit test asserts a function returns a specific value on a specific input. A golden test asserts that an agent’s trajectory and output satisfy a set of traits on a realistic input. Scores are graded so the suite tolerates the small variation inherent to LLM calls. Unit tests gate on equality; golden tests gate on thresholds.
How do you detect drift in a production AI agent?
Run a weekly cron that samples 200 recent production traces, re-runs the agent on those inputs in a sandbox, and compares current outputs to historical outputs on the same inputs. Track tool-call distribution drift, schema-conformance drift, cost and latency drift, and output-content drift. Alert when any metric moves 3% or more week over week.
How do you tell a real regression from an LLM flake?
Run each case 3 times in CI and use the median score. Pin a baseline of acceptable per-metric scores in a config file. Set alert thresholds high enough that the team trusts the alert. Case-level single-run failures are noise. Distribution-level shifts above the threshold are regressions.
Can you add an eval harness after the agent is in production?
Technically yes, practically expensive. Adding evals retroactively means cataloging the behaviors you already shipped, including the ones you did not consciously decide to ship. It is faster and cheaper to build the smallest possible eval suite (5 cases per capability) before launch and grow it from there. Once the suite exists, every prompt change is verifiable instead of a hope.
If your agent is approaching production and you do not yet have a regression gate, Mirgen and the Motomtech team have shipped eval harnesses for HealthTech, FinTech, and multi-tenant SaaS agents under the same 5-layer pattern. Book a 15-min discovery call to walk through what a golden suite would look like for your workflow: https://cal.com/mirgen-motomtech/quick-intro.
Mirgen Hoxha, CEO, Motomtech.
