Agent Evaluation
Follow evaluation-driven development
Define evaluation metrics at design-time
Prepare data sets and evaluators
Instrument the system to collect telemetry
Offline evaluation
Multi-faceted methods measuring agent’s success at completing its core functions/behaviors: goal success rate, response quality, safety, finding info, using tools, clear communication, etc.
 |
One evaluator: dataset => agent => evaluation => result: e.g. 95% task-level accuracy Aggregating all evaluators across all instances of the data = final score (think Deep Thought!). Each agent can have multiple evaluation metrics. High-quality custom dataset for a specific task. |
Typical Structure of the Evaluation Process
Create “golden” Eval Dataset to benchmark LLMs & AI agents: Q&A pairs - curated queries and expected answers - org's consensus on quality.
Run the full Eval Dataset (every Q&A pair) across different agents capturing responses for each Q&A pair.
LLM-as-a-judge takes the answers and uses a consistent evaluation prompt or rubric to produce structured metrics (e.g. score each response on a binary or 1-5 scale). If search, run it across different search agents w/same synthesis LLM and prompts - only search quality varied. Judge pitfalls: position bias (swap answer order & average), verbosity bias (length-penalize), self-preference (use a judge from a different model family), score clustering (anchored rubrics w/an example per score level). Pairwise comparison is more reliable than absolute scoring; make the judge reason (CoT) before emitting the score
Aggregate into clear, comparable metrics - powerful for comparing two versions of an agent or different agents. Example - industry benchmarking against competitors.
If custom golden sets aren't feasible, use established benchmarks - SimpleQA, FRAMES, FreshQA, BrowseComp, FinSearchComp, etc. Agent-specific: SWE-bench (real GitHub issues), GAIA (general assistant tasks), WebArena (browser tasks), τ-bench (tool use w/simulated users; reports pass^k — probability that ALL k of k runs succeed (pass to the k, pass power k) — measures reliability/consistency, stricter than pass@k (proba that at least ONE our k passes will be true).
Different task performance: e.g. good at scheduling meeting, but not at sending emails - slicing dimension applied to GSR - breakdown by task type.
1. Retrieval & Grounding (RAG) Quality
Classic RAG triad:
Retrieval quality — did the retriever find the right docs? Measured by relevance, recall@k, precision@k, ranking (mean reciprocal rank (MRR) / Normalized Discounted Cumulative Gain (NDCG).
Groundedness / faithfulness — is the answer supported by the retrieved context? Near-synonyms - Vertex AI calls it groundedness; RAGAS calls it faithfulness. Inverse of groundedness - hallucination rate.
Answer correctness/relevance (accuracy) - answer addresses the Q and matches GT?
Example: Answer can be grounded in retrieved docs, but still be wrong (wrong docs retrieved). If you ask, "What were our company's Q3 revenues?", retrieval quality checks if the agent pulled the Q3 fin. report; faithfulness checks if the answer matches the numbers in that report.
2. Response Quality
Vertex AI uses model-based evaluation: LLM "grades" the agent's output based on a set of criteria that measure subjective qualities:
Fluency: Is the language natural and grammatically correct?
Coherence: Does the response make logical sense?
Helpfulness: Does the answer directly and effectively address the user's query?
Safety: Responsible AI principles observed? (no toxicity, hate speech, PII, or harmful content).
3. Tool-level metrics
Tool selection correct: (e.g., get_weather?).
Correct tool arguments called (e.g., get_weather(location)).
Tool call efficiency (redundant/unnecessary calls - common agent failure mode)
Overall tool use success: tool-calling led to success?
4. Trajectory evaluation (above tools)
Judges the whole reference tool sequence (distinct from single tool-call correctness) - exact match / in-order match / precision-recall (Vertex AI's agent eval).
5. Task outcomes (task-level metrics)
Agent goal success rate (GSR), task completion rate (failure rates), test cases passed (coding), hallucination rate, human intervention rate, user satisfaction, error handling of invalid inputs and API failures.
6. System Performance & Cost (System-level metrics)
Task completion (response) time, Latency per tool call, LLM call error rate, Cost and token usage per task
Task metrics tell you that it failed; component metrics tell you why. These aren’t parallel buckets - it’s a methodology / causal ladder from components to outcomes explaining the diagnostic flow. Groups 1–3 evaluate individual steps in the agent loop (retrieve, generate, act); group 4 evaluates whether the whole trajectory achieved the goal; group 5 evaluates whether it did so efficiently and reliably. When GSR (group 4) drops, you descend the ladder — was it bad retrieval (1), a hallucinated response (2), or a wrong tool call (3)?.
Online evaluation
Definition. Online evaluation = continuous measurement on live production traffic, versus offline eval on curated test sets. Core challenge: no ground truth labels, so you need reference-free scoring.
Signal sources, roughly in order of scale vs. fidelity:
Implicit user signals — thumbs up/down, regeneration rate, copy/accept rate, follow-up rephrasing (indicates the first answer failed), session abandonment, task completion clicks. Cheap, abundant, noisy.
LLM-as-judge on sampled traces — reference-free criteria only: groundedness, answer relevance, coherence, safety, instruction-following. (no answer correctness - no reference). Judge must be periodically calibrated against human labels.
Human evaluation — small samples, calibration layer on top LLM-as-judge and implicit signals (spot-check the judge, adjudicate disagreements), not really an online evaluation.
Guardrails/heuristics — deterministic checks that run on 100% of traffic: PII detection, toxicity classifiers, format validation, citation-presence checks.
Online operational metrics: latency percentiles (p50/p95/p99), error and timeout rates, cost per session, tool-call failure rates, human-escalation rate - Group 6 above.
Drift detection. Distribution shift in incoming queries (new topics, new intents) vs. your offline eval set; degradation in judge scores over time; per-segment slicing (online is where you discover capability gaps by task type).
Methods. A/B testing and interleaving for comparing agent versions on live traffic; canary deployments with automatic rollback on metric regression; shadow mode (new version runs silently alongside prod, scored but not shown).
Offline vs. Online
Labelled online data may be persisted for regression testing. Trace-curation flywheel - online data = unlabeled, but you persist production traces (queries, retrieved context, tool calls, responses) + labels you can attach after the fact - explicit user feedback, LLM-judge, or human annotation of sampled traffic. This feeds back into your offline eval set —production failures become tomorrow's regression tests.
The one-line summary: offline eval => "is this version good enough to ship?", online eval => "is it still good, for real users, right now?", and trace-curation flywheel connects them.
Example of LLM Judge Rubrics (offline & online) - DeepResearch:
Presentation and organization
Fact and logic consistency
Coverage and comprehensiveness
Citation accuracy (does each claim's citation support it - groundedness at claim level).
Instruction adherence (did the report respect scope, format, length constraints).
Validating LLM against humans in the form of an inter-annotator agreement (IAA). Same can be used to calculate agreement among humans:
Cohen's kappa (two raters), Fleiss' kappa (multiple raters) or Krippendorff's alpha (multiple raters + missing data = the most general metric) (for binary or categorical relevance): κ > 0.80 = "almost perfect", 0.60–0.80 = "substantial”, 0.40–0.60 = "moderate," <0.40 = “weak”.
Pearson Correlation (for graded or continuous scores) or Spearman’s for ordinal. Pearson’s r measures how consistently the LLM’s scores track human scores across the full range. r > 0.85: Strong alignment, r > 0.70: Moderate, usable with caution, r < 0.60: Weak alignment.
Threshold-Based Agreement - easily interpretable measure for stakeholders: the percentage of cases where the model’s score falls within a small tolerance of the human score: |LLM_score – Human_score| ≤ 1 on a 0–4 scale. Target: ≥ 75% agreement within ±1 point. Practical usability - is the model “close enough” to downstream decisions.
Human–human IAA = ceiling for LLM–human agreement. Measure human-human first - if it’s κ = 0.65, the judge can’t hit 0.80 against any single human. The practical validation criterion is: LLM-judge agreement with humans ≈ human agreement with each other => the judge is interchangeable with one more annotator, which is the bar for trusting it at scale.
Evaluation Frameworks
Use categories rather than a flat list —"I've heard of these tools" becomes "I understand the space."
Observability / LLMOps platforms (online eval = trace-first, eval added on top)
LangSmith — visibility into agent behavior: tracing, real-time monitoring, alerting, usage insights. LangChain's closed-source product.
Langfuse — the open-source counterweight to LangSmith: traces, prompt management, LLM-as-judge evals, user feedback capture, cost tracking.
Arize Phoenix — open-source tracing + eval from Arize, strong on embedding-drift visualization and retrieval debugging; built natively on OpenTelemetry/OpenInference standards. Its commercial sibling Arize AX - big enterprise ML observability player.
Others in this tier worth name-dropping: W&B Weave (if the org already uses Weights & Biases), Helicone (lightweight proxy-based logging), Braintrust (eval-centric platform popular with product teams), Confident AI (the platform behind DeepEval).
Evaluation libraries (offline eval - metrics-first, you bring the harness)
RAGAS —signature contribution is the reference-free RAG metric suite (RAG triad) - faithfulness, answer relevance, context precision/recall. Computes traditional metrics like BLEU too.
TruLens — built around the "RAG triad" framing explicitly; its distinctive concept is feedback functions attached to any step of the app. Now under Snowflake.
DeepEval — "pytest for LLMs": unit-test style assertions (G-Eval, hallucination, bias, toxicity metrics), CI/CD-friendly. Probably the best answer to "how do you put evals in a CI pipeline."
OpenAI Evals / Evals API and promptfoo — promptfoo especially is popular for declarative, config-driven prompt/model comparison and red-teaming.
Guardrails-adjacent: NeMo Guardrails, Guardrails AI — runtime validators - deterministic checks on 100% of traffic, rather than eval libraries.
Cloud-native eval services
Vertex AI Gen AI Evaluation Service — pointwise and pairwise model-based eval, plus the agent-specific piece we discussed: trajectory evaluation (exact/in-order/any-order match against reference tool sequences) alongside final-response eval.
Azure AI Foundry (formerly AI Studio) evaluations — groundedness, relevance, coherence, safety evaluators; integrates with Azure content safety.
AWS Bedrock Evaluations — automatic and human-based model eval, plus RAG evaluation for Knowledge Bases.
Benchmark harnesses (a fourth mini-category worth one sentence): EleutherAI lm-evaluation-harness and HELM for standardized academic benchmarks, and agent-specific benchmarks like SWE-bench, WebArena, τ-bench, GAIA — these evaluate models/agents in general, versus everything above, which evaluates your application.
Eval libraries (RAGAS, DeepEval) - offline eval in CI; observability platforms (LangSmith, Langfuse, Phoenix) - online tracing and judge-scored production monitoring, and their dataset features close the loop by turning traces into offline test sets. Mature stack = one platform + one library — e.g., Langfuse + RAGAS is a common all-open-source combination, LangSmith alone covers both roles inside the LangChain ecosystem.