agentage.wiki / Articles / Guide
Evaluating AI Agents: Building a Test Set That Means Something
Observation is not validation; build a test set that proves your agent works across real tasks, edge cases, and repeated runs.
Observation Is Not Validation
Watching an agent run is not the same as proving it works. You might see the agent complete a task without errors, yet the output could be factually wrong or hallucinated. Production teams often confuse visibility with quality, assuming that because they can trace every step, the result is correct. This gap leaves systems vulnerable to silent failures that only surface after significant damage occurs.
Data shows that high adoption of observability tools does not guarantee rigorous testing. Many teams track logs and traces but fail to run formal evaluations against a ground truth dataset. An agent can achieve a high score on a generic metric while still producing catastrophic errors in specific, high-stakes scenarios. Without a dedicated validation layer, you are flying blind despite having a full cockpit view.
Validation requires a separate process that compares agent output against known correct answers or strict rubrics. You must treat the evaluation set as a living document that evolves with your system. This approach shifts your focus from simply seeing what happened to understanding whether the outcome was actually right. The goal is to catch systemic errors before they reach real users.
Collecting Real Tasks and Edge Cases
Start your evaluation set with approximately twenty real queries that reflect actual user needs. These initial tasks should cover the most common workflows your agent will handle. Early testing on a small, relevant set yields dramatic improvements in success rates compared to waiting for a massive dataset. You can scale the size of your test suite later once the core logic is stable.
You must also include specific edge cases that test the boundaries of your agent's capabilities. These are scenarios where the agent might hallucinate, misuse a tool, or fail to follow instructions. Human reviewers should curate these difficult examples to ensure the agent handles ambiguity and error conditions correctly. Automated judges often miss subtle failures that a human would spot immediately.
Organize your test cases into categories based on task complexity and risk. Simple fact-checking tasks require different validation criteria than complex research or synthesis jobs. Ensure your dataset includes examples of both successful completions and known failure modes. This mix allows you to measure not just accuracy but also the agent's ability to recover from mistakes or refuse unsafe actions.
Repeated Runs for Non-Determinism
Large language models are inherently non-deterministic, meaning the same prompt can yield different results. Running your agent once is insufficient to gauge true performance. You must execute each test case multiple times to observe the variance in outcomes. This repetition reveals whether the agent is consistently reliable or merely lucky on a single run.
Track metrics across these repeated runs to identify instability. Look for fluctuations in token usage, the number of tool calls, and the final answer quality. High variance indicates that the agent's behavior depends too much on random sampling rather than logical reasoning. You need to stabilize the system before deploying it to production environments where consistency matters.
Use these repeated runs to tune your temperature settings and system prompts. If the agent fails frequently on a specific task type across multiple attempts, the issue lies in the design, not the randomness. Adjust your orchestration logic or tool definitions to reduce this variability. Consistency is a prerequisite for trust, especially when the agent handles sensitive or critical data.
Scoring Outcome, Cost, Steps and Safety
Define a scoring rubric that evaluates the agent on multiple dimensions beyond just success or failure. Measure factual accuracy, citation correctness, completeness, and the efficiency of tool usage. Use an LLM-as-a-judge to automate this scoring, but ensure the judge itself is calibrated against human feedback. A single numeric score can hide critical weaknesses if the rubric is too broad.
Cost and step count are equally important metrics for operational viability. An agent that solves a problem but burns excessive tokens or takes too many steps is not scalable. Track the total token consumption and the number of iterations required for each task. Set hard limits on these values to prevent runaway loops and budget overruns in production.
Safety violations must be treated as automatic failures regardless of the final output quality. If the agent accesses unauthorized data, generates harmful content, or bypasses guardrails, the run is invalid. Implement a verification step that explicitly checks for these safety constraints before accepting any result. This layer acts as a final gatekeeper to protect your organization from liability and reputational damage.
- Factuality: Does the output match known ground truth?
- Efficiency: Did the agent use the minimum necessary tokens and steps?
- Safety: Did the agent adhere to all guardrails and access policies?
- Completeness: Did the agent answer all parts of the user request?
Proving Gains with A/B Runs
You cannot assume that adding a verification step or changing a prompt improves performance without proof. Run A/B tests where you compare the new configuration against the baseline using the same test set. Execute the same queries with and without the proposed change to isolate the effect. This empirical approach prevents you from making decisions based on intuition or anecdotal evidence.
Focus on measuring the specific mechanism of improvement. For instance, if you add a chain-of-verification step, determine if the reduction in errors comes from better reasoning or simply from the agent stating fewer facts. Separate the signal from the noise by analyzing the detailed logs of each run. This granular analysis helps you understand exactly why a change worked or failed.
Scale your evaluation only after you have measured a clear gain. Do not expand the agent's scope or increase its complexity until the current version passes the test set with a statistically significant margin. Use your own data to build the scaling curve, as public benchmarks often fail to predict performance in your specific domain. Empirical validation is the only reliable path to scaling.
Key takeaways
- Observation and logging are not substitutes for formal validation against a ground truth dataset.
- Build your initial test set with twenty real queries and specific edge cases curated by humans.
- Run tests repeatedly to measure non-deterministic variance and ensure consistent behavior.
- Score agents on outcome accuracy, cost efficiency, step count, and safety violations.
- Prove performance gains with A/B tests before scaling or deploying changes to production.
Related questions
How many test cases do I need to start evaluating my AI agent?
Start with approximately twenty real queries that reflect actual user needs. This small, focused set allows you to make rapid improvements early in development. You can scale the dataset size later once the core logic is stable and you have established a reliable evaluation pipeline.
Why is running an agent multiple times necessary for testing?
Large language models are non-deterministic, meaning the same prompt can produce different results. Running tests repeatedly reveals the variance in outcomes and helps you distinguish between consistent performance and lucky guesses. This data is essential for tuning your system to ensure reliability in production.
Can I rely on an LLM judge to score my agent's performance?
You can use an LLM judge, but only if it is calibrated against human feedback and uses a specific rubric. Without calibration, the judge may replicate its own biases or miss subtle errors. Combine automated scoring with regular human spot-checks to ensure the evaluation remains accurate and trustworthy.
AI-assisted: drafted with a language model from the author's research notes, then edited and checked before publication. Author: Zoltán Varga, AI visibility researcher and AI ecosystem architect, Budapest.
