Evals

Lock in agent behavior with regression tests.

Next, let’s lock in our agent behavior with evals.

Evals are regression tests for your agents. It’s the same prompts and the same agents, run on a schedule, so you can see when behavior drifts.

When we run /improve-agent, we’re looking for out-of-distribution improvements. Evals make sure in-distribution cases continue to pass. The two work together.

Cases

Cases live in evals/cases.py. Each case sends one input to an agent and (optionally) checks two things:

• judge: AgentAsJudgeEval scores the response against criteria (binary pass/fail) using an LLM.
• reliability: ReliabilityEval checks which tools fired against expected_tool_calls.

A case looks like this:

evals/cases.py

CASES: tuple[Case, ...] = (
    Case(
        name="web_search_recent_anthropic_research",
        agent=web_search,
        input="What did Anthropic publish about agent research recently?",
        tags=("live",),
        timeout_seconds=120,
        criteria=(
            "Answers the question by citing at least one real Anthropic URL "
            "(anthropic.com domain). The response is grounded in fetched content "
            "rather than refusing to answer."
        ),
        expected_tool_calls=(_WEB_SEARCH_TOOL,),
    ),
    # add more cases here
)

A case can use either check or both. If both are set, the agent runs once and feeds the same response into both.

Add tags to group your cases into suites. The template comes with three suites: smoke, release, and live. This case is tagged live because its answer depends on the open web.

Run the suite

1. Create a virtual environment

   The eval suite runs on the host and needs a local virtual environment:

   ./scripts/venv_setup.sh
   

   Activate it:

   source .venv/bin/activate
   
2. Run the eval suite

   python -m evals --tag smoke    # fast suite
   python -m evals                # full suite
   

   Other options:

   python -m evals -v             # stream the agent run with full panels
   python -m evals --name <case>  # single case while iterating
   

Each case prints the response, the judge verdict, and the reliability verdict. The run ends with an Eval Summary table.

Results write to Postgres via eval_db. The eval history shows up on os.agno.com alongside your sessions and traces, so you can see when a case started failing and what changed.

Diagnose failures with Claude Code

Open Claude Code and run:

/eval-and-improve

Claude runs the suite, triages every failure (bad criteria, real regression, flaky LLM judge), and proposes in-scope fixes. It edits the agent or the case and re-runs until the suite is green.

When to run evals

Trigger	Frequency
Before deploying a change to an agent	Every time
As part of CI	Every PR
Against production	On a daily cron
After bumping a model version	Every time

The production cron is the most valuable one, and the template ships it as the run_evals workflow. Set ENABLE_SCHEDULED_EVALS=True to run the smoke-tagged cases daily. See scheduling for the cron API.

What good cases look like

• Specific. “Returns a JSON object with ticker and price” beats “Returns the right answer”.
• Stable. Avoid prompts whose correct answer changes daily. Use phrasing like “describes a real, recent…” instead of locking in a specific result.
• Scoped to one behavior. One case per behavior makes failures easy to read.
• Anchored to tools. expected_tool_calls catches the failure mode where the agent confidently makes things up instead of calling a tool.

Next

Next steps →