AI Evals Explained

How to Evaluate
AI Agents

A beginner-friendly guide to AI evals: what they are, how they differ from benchmarks and classic machine learning evaluation, which metrics to track for which task, and the evaluation loop that separates teams shipping AI agents from teams shipping demos.

Source: youtube.com/watch?v=_Er8Hao_gmQ · Channel: Aishwarya Srinivasan · Full article in this repo

Agenda

What We'll Cover

Foundations

  • What is an AI eval?
  • Evals vs benchmarks vs classic ML evaluation
  • The core metrics and which to track per task
  • Three families of grading metrics

Putting Evals to Work

  • Start with a golden dataset, not a metric list
  • Four ways to grade agent outputs
  • Evals are a continuous loop, not a report card
  • Tools for running and monitoring evals

Section 1 · What Is an AI Eval?

From "Looks Fine" to an Actual Score

Building the agent is the fun part — the hard part is knowing whether it is actually any good. An AI eval is just a structured way of measuring how good your AI outputs really are.

The one question that matters

Is my agent getting better, or did I just break it? Every time you change a prompt, swap a model, or tweak retrieval, the eval tells you which one you did.

An agent is a chain of decisions

It picks a tool, calls it, reads the result, decides what to do next — on and on, with many places where things quietly go sideways. When it breaks, you need to know where and why.

~1/3 gap between lab scores and real deployed work Prompts · models · retrieval — every change Teams that ship measure — no "vibe evals"

Section 2 · Evals vs Benchmarks vs Classic ML

Benchmarks Test the Model; Evals Test Your System

Benchmarks

A generic test — the same test applied to every model.

  • SWE-bench — coding
  • Tau-bench — customer-service agents
  • Terminal-Bench — command-line work
  • Composite indexes — many boards mashed into one score
Great for choosing a starting model

AI evals

A test of your agentic system.

  • Your task
  • Your data
  • Your prompts
  • Your users
Verify the benchmark pick on your own task
A model can sit at the top of every leaderboard and still completely fall apart on your use case — because that benchmark never saw your data.

Section 2 · Evals vs Benchmarks vs Classic ML (cont.)

From Multiple-Choice Test to Essay

Classic ML evaluation

A multiple-choice test with an answer key.

  • Output is clean: a category, a number, a yes or no
  • Test set contains the right answers, so grading is objective
  • Metrics: accuracy, precision, recall, F1

AI evals

An essay — and there is no answer key.

  • Output is open-ended text or whole action sequences
  • No single right answer: two great summaries can look completely different
  • You need a rubric, and somebody has to use judgment
The good news: once the rubric mindset clicks, everything else gets simpler.

Section 3 · The Metrics That Matter

The Core Four Metrics

Relevance

Did the output actually answer what was asked?

Faithfulness

Also called groundedness — is it true and backed by real data, not made up? This is your hallucination check.

Correctness

Did it match the expected answer, when there is one?

Coherence

Does it actually read well?

Top level — but task-dependent

These four are the top-level metrics to reach for again and again. The right metric for any given evaluation depends entirely on your task.

Section 3 · The Metrics That Matter (cont.)

Match the Metric to the Task

TaskWhat to track
SummarizationFaithfulness — stayed true to the source, invented nothing · Coverage — captured the main points · Conciseness — kept it tight
ClassificationOutput is a clean label, so it is old-school ML again: accuracy, precision, recall, F1
TranslationMeaning carries over and it reads naturally — metrics like BLEU and COMET
RAGFaithfulness + answer relevance — together they catch the two ways a RAG system breaks: bad retrieval or bad generation

Section 3 · Three Families of Metrics

Metric Alphabet Soup, Sorted Into Three Families

Word-overlap metrics

BLEU · ROUGE · METEOR

Measure word overlap against a reference answer. Fast, cheap, and dead simple — but blind to meaning: the same idea in different words can score differently.

Fast · cheap Blind to meaning

Semantic metrics

BERTScore · BLEURT · COMET

Compare meaning using embeddings instead of matching words, so they handle paraphrasing far better than overlap scores.

Meaning-aware

Model-based metrics

LLM-as-a-judge · G-Eval

An LLM grades the output against your rubric — the only family flexible enough for open-ended, messy outputs with no reference answer.

Scales to essays

Agent outputs are open-ended and reference-free — so more and more, the thing doing the grading is a model.

Section 4 · Start With a Golden Dataset

The Dataset First, the Metrics Second

The single most important habit in this whole guide: do not start by grabbing a metric off a list. Build a golden dataset — an answer key you write for your own specific task.

A handful of example inputs Each paired with a great output and a bad output Common cases Annoying edge cases Every failure you have watched blow up Start small — about 52 examples is plenty
Work backwards: what would make this output good, and what would make it bad? The quantified ways of judging those answers are your metrics — the dataset comes first, and the metrics fall out of it.

Section 5 · Four Ways to Grade Agent Outputs

Humans, and Your Real Users

Human evals

The gold standard.

A person — ideally a domain expert — grades outputs against your rubric.

Source of truth Slow · expensive · doesn't scale Save it for small, careful samples

User feedback from production

Real signal from real users.

  • Thumbs up or down?
  • Accepted, rejected, or edited?
  • Did they finish the task — or come back with the same question?
Ground truth that matters Noisy — and only after you ship, so it can't be your safety net

Section 5 · Four Ways to Grade Agent Outputs (cont.)

Code Checks, and an LLM Judge

Programmatic evals

Simple checks you write in code.

  • Is the JSON valid?
  • Right tool, right arguments?
  • How long did it take? How much did it cost?
Cheap · instant · run on every change Need clearly checkable answers — pairs beautifully with classification

LLM as a judge

Essay-style grading that scales.

A strong model grades outputs against your rubric — is this helpful? Is it grounded in the source? — across thousands of outputs, the open-ended stuff plain code never could assess.

Biased: prefers longer answers and whatever it sees first
Validate your judge against a human grader on a sample first. Keep it honest, and it becomes your absolute workhorse.

Section 6 · Evals Are a Loop, Not a Report Card

CI/CD for AI Agents

You don't test code once and walk away — you run tests on every change, forever. AI agents get CI/CD plus continuous evaluation and continuous monitoring.

Decide what "good" means Build the golden dataset Pick metrics from the dataset Run a baseline Group the failures Fix: prompts · retrieval · tool definitions Rerun the exact same dataset Ship — and keep watching live traffic
Numbers up and nothing regressed? Catching something you broke without even realizing it is the entire reason the loop exists.

Providers can quietly swap the models underneath you — behavior can shift overnight. Real-world failures flow back into the golden dataset, and you run the whole thing again.

Section 7 · Tools for Running and Monitoring Evals

You Don't Have to Build It From Scratch

Running the evals

  • Promptfoo — test prompts and run evals right from the command line
  • RAGAS — RAG metrics straight out of the box
For the eval runs themselves

Tracing and monitoring

Watch what is happening in production:

  • LangSmith
  • Langfuse
  • Arize
  • Braintrust
Pick the ones that fit your stack, wire them into the loop, and start measuring.

Key Takeaways

What to Remember

An eval is a structured, repeatable score: is my agent better, or did I break it?
Benchmarks test the model; evals test your system — your task, your data, your users
Open-ended outputs have no answer key — you need a rubric and judgment
Core four: relevance, faithfulness (groundedness), correctness, coherence
Task metrics: coverage + conciseness · accuracy/F1 · BLEU/COMET · faithfulness + relevance for RAG
Golden dataset first — metrics fall out of it; about 52 examples is plenty to start
Four graders: humans (gold standard), user feedback, programmatic checks, LLM judges — validated against humans
Evals are a continuous loop — model swaps can shift agent behavior overnight

The End

Measure It, Then Measure It Again

You will never know whether your AI agent actually works by watching it succeed one time. You only know by measuring it — on your data, again and again.

Source: "AI Evals Explained | How to evaluate AI Agents?" — Aishwarya Srinivasan (youtube.com/watch?v=_Er8Hao_gmQ) · Full article in this repo.

← → to navigate · swipe on mobile