---
title: "How to Test a Pydantic AI Agent Without API Calls and Turn Failures Into Evals"
description: "A pytest suite that checks a Pydantic AI agent's tools, structured output and retries with no model calls, plus a pydantic-evals dataset that turns a failed production run into a CI gate."
author: "Jahanzaib Ahmed"
date: 2026-10-14
category: "ai-agents"
readingTime: "22 min read"
tags: ["pydantic-ai", "testing", "evals", "python", "ai-agents"]
canonical: https://www.jahanzaib.ai/blog/pydantic-ai-testing
source: https://www.jahanzaib.ai
---
# How to Test a Pydantic AI Agent Without API Calls and Turn Failures Into Evals

This Pydantic AI testing guide ends with two things you can run today. The first is a pytest suite that checks an agent's tools, structured output and validation retries without a single model call. The second is a pydantic-evals dataset where a failed production run, a double charge the agent filed as "other", becomes a case your CI won't ship past. It costs nothing and needs no API key and no Logfire account.

It's written for Python developers who already have an agent running. If you're still building one, start with my [Pydantic AI tutorial for production agents](https://www.jahanzaib.ai/blog/pydantic-ai-tutorial-production-agents), which covers agents, tools, dependencies and structured output; this page covers how you test all of it.

Every command and block of output below comes from a run I did on October 15, 2026, in a fresh Python 3.13 virtual environment.

## Before you start

-   **Python 3.11 or newer.** `pydantic-ai-slim` 2.56.0 declares `>=3.11`; I used 3.13.12.
-   **Pydantic AI 2.56.0**, released on [PyPI](https://pypi.org/project/pydantic-ai/) on October 13, 2026, and **pydantic-evals 2.56.0** from the same release ([PyPI page](https://pypi.org/project/pydantic-evals/)).
-   **pytest** (I ran 9.1.1). anyio, which runs the async tests, arrives as a dependency of Pydantic AI (4.15.1 here).
-   **The logfire SDK** (5.2.0) installed through the `pydantic-evals[logfire]` extra. You need the SDK for the tool call evaluators. You do not need an account.
-   **Cost:** zero. Every model in this guide is a stub that runs in process.

![Three layers of Pydantic AI testing: TestModel for wiring, FunctionModel for scripted turns, pydantic evals for behaviour](https://cdn.sanity.io/images/qajb7q5q/production/1cfe06482ff2eab7e9db10b9ebfcfebe3fd55385-1376x768.png?w=1200&q=75&auto=format&fit=max)

_TestModel proves the plumbing, FunctionModel proves specific turns, and evals score behaviour across many inputs; none of the three call a provider._

## Step 1: Install pinned versions and block real model calls

Pin the exact versions. Pydantic AI ships often, and a test suite that silently upgrades its own framework is a suite you can't trust when it goes red.

```
uv venv --python 3.13 .venv
uv pip install --python .venv/bin/python "pydantic-ai-slim[openai]==2.56.0" "pydantic-evals[logfire]==2.56.0" pytest
source .venv/bin/activate
```

Every file in this guide sits in one folder:

```
conftest.py       # Step 1: blocks real model calls
triage_app.py     # Step 2: the agent under test
test_triage.py    # Steps 3 to 7: unit tests
stub_models.py    # Step 8: stand in models for the evals
evals_triage.py   # Step 8: dataset, evaluators and task
save_dataset.py   # Step 9: writes the dataset YAML
prod_run.py       # Step 9: simulates and logs a production run
promote_case.py   # Step 9: turns a logged run into a case
run_eval.py       # Step 9: prints the eval report
test_evals.py     # Step 10: the CI gate
```

Then create a `conftest.py` that makes real model requests impossible. The [official unit testing guide](https://pydantic.dev/docs/ai/guides/testing/) recommends setting `ALLOW_MODEL_REQUESTS = False` globally, and I put it in conftest so no test file can forget it.

```
import pytest
from pydantic_ai import models

# Any test that reaches a real provider fails instead of spending money.
models.ALLOW_MODEL_REQUESTS = False


@pytest.fixture(autouse=True)
def placeholder_key(monkeypatch):
    # The OpenAI provider wants a key to exist; this one can never authenticate.
    monkeypatch.setenv('OPENAI_API_KEY', 'placeholder-not-a-key')


@pytest.fixture
def anyio_backend():
    return 'asyncio'
```

The placeholder key needs explaining. When I removed it, an unpatched run never reached the safety switch. The provider refused to build first, with `UserError: Set the OPENAI_API_KEY environment variable`. That still fails the test, but for a confusing reason. With a placeholder present, the failure says exactly what happened: the test tried to call a real model. Tests that override the model pass with or without the key, which I also checked.

**Tip:** Version 2.56.0 prints a large startup banner the first time an agent runs, inviting you to turn on observability. Set `PYDANTIC_AI_NO_BANNER=1` in CI so it stays out of your test logs.

## Step 2: Write the agent you're going to test

The example is a support triage agent. It reads a customer message, looks up an order when one is mentioned, and returns a typed `Ticket`. It has one tool, one dependency and a structured output with constrained values, which is enough to show each pattern once. Save it as `triage_app.py`.

```
from dataclasses import dataclass, field
from typing import Literal

from pydantic import BaseModel
from pydantic_ai import Agent, RunContext


class Ticket(BaseModel):
    category: Literal['refund', 'shipping', 'account', 'other']
    priority: Literal['low', 'normal', 'high']
    order_id: str | None = None
    summary: str


@dataclass
class OrderDB:
    orders: dict[str, str] = field(default_factory=dict)
    lookups: list[str] = field(default_factory=list)

    def status(self, order_id: str) -> str:
        self.lookups.append(order_id)
        return self.orders.get(order_id, 'not found')


triage_agent = Agent(
    'openai:gpt-5.2',
    deps_type=OrderDB,
    output_type=Ticket,
    instructions=(
        'Classify the customer message into a support ticket. '
        'If the message mentions an order number, look it up first.'
    ),
    defer_model_check=True,
)


@triage_agent.tool
def lookup_order(ctx: RunContext[OrderDB], order_id: str) -> str:
    """Return the shipping status of an order."""
    return ctx.deps.status(order_id)


async def triage(message: str, db: OrderDB) -> Ticket:
    result = await triage_agent.run(message, deps=db)
    return result.output
```

Two choices here make testing easier later. `defer_model_check=True` stops the agent from resolving its provider at import time, so test modules import cleanly without credentials. And `OrderDB` records every lookup, which turns "did the tool run, and with what argument?" into a one line assertion.

## Step 3: Check the wiring with TestModel

TestModel tests are the cheapest ones to write. By default it calls every tool on the agent, then produces output that satisfies your output schema. The docs are blunt that there is no AI inside it: it is procedural code that generates schema valid data. Use it to answer one question: is everything connected? Put these tests in `test_triage.py`.

```
import pytest
from pydantic_ai import ToolCallPart, capture_run_messages
from pydantic_ai.models.test import TestModel

from triage_app import OrderDB, Ticket, triage, triage_agent

pytestmark = pytest.mark.anyio


async def test_real_model_is_blocked():
    with pytest.raises(RuntimeError, match='ALLOW_MODEL_REQUESTS'):
        await triage('Where is order 1042?', OrderDB())


async def test_testmodel_calls_every_tool():
    db = OrderDB()
    with capture_run_messages() as messages:
        with triage_agent.override(model=TestModel()):
            ticket = await triage('Where is order 1042?', db)

    assert isinstance(ticket, Ticket)
    assert db.lookups == ['a']  # TestModel invents schema valid arguments
    calls = [p.tool_name for m in messages for p in m.parts if isinstance(p, ToolCallPart)]
    assert calls == ['lookup_order', 'final_result']
```

Here's what TestModel actually returned. It called `lookup_order` with `order_id='a'`, not 1042. It doesn't read the prompt at all. The final output was `Ticket(category='refund', priority='low', order_id=None, summary='a')`: the first allowed value for each literal and a single letter for each string. Usage came back as `RunUsage(input_tokens=110, output_tokens=19, requests=2, tool_calls=1)`, with token counts that TestModel estimates rather than bills.

So assert on structure here, never on content. The `final_result` name in the captured messages is the output tool Pydantic AI registers for structured output, and seeing it confirms the agent took the structured path. `capture_run_messages` is how you see every request and response in the run.

## Step 4: Pin the structured output you need

Sometimes the code under test sits after the agent: a router, a database write, an email. You want the agent to return one specific `Ticket` and nothing else. TestModel takes `custom_output_args` for that, and `call_tools=[]` skips the tools entirely. Append this, and the tests in Steps 5 and 6, to `test_triage.py`.

```
async def test_fixed_structured_output():
    fixed = {'category': 'refund', 'priority': 'high', 'summary': 'Wants money back'}
    with triage_agent.override(model=TestModel(call_tools=[], custom_output_args=fixed)):
        ticket = await triage('I want my money back', OrderDB())
    assert ticket == Ticket(**fixed)
```

Notice the pattern in both tests. `triage()` is application code that knows nothing about tests, and `Agent.override` swaps the model from outside it. You never add a `model=` parameter to production functions just to make them testable.

## Step 5: Script exact turns with FunctionModel

TestModel can't tell you whether your tool result changes the answer, because it ignores both the prompt and the tool output. [FunctionModel](https://pydantic.dev/docs/ai/api/models/function/) can. You write a function that receives the message history plus an `AgentInfo` and returns whatever response the model should give at that point in the conversation.

```
from pydantic_ai import ModelMessage, ModelResponse, ToolCallPart, ToolReturnPart
from pydantic_ai.models.function import AgentInfo, FunctionModel


def lost_parcel_model(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
    last = messages[-1].parts[-1]
    if isinstance(last, ToolReturnPart):
        status = last.content
        args = {
            'category': 'shipping',
            'priority': 'high' if status == 'lost' else 'normal',
            'order_id': '1042',
            'summary': f'Order 1042 is {status}',
        }
        return ModelResponse(parts=[ToolCallPart(info.output_tools[0].name, args)])
    return ModelResponse(parts=[ToolCallPart('lookup_order', {'order_id': '1042'})])


async def test_tool_result_drives_priority():
    db = OrderDB(orders={'1042': 'lost'})
    with triage_agent.override(model=FunctionModel(lost_parcel_model)):
        ticket = await triage('Where is order 1042?', db)
    assert db.lookups == ['1042']
    assert ticket.priority == 'high'


async def test_override_swaps_dependencies():
    fake = OrderDB(orders={'1042': 'delivered'})
    with triage_agent.override(model=FunctionModel(lost_parcel_model), deps=fake):
        ticket = await triage('Where is order 1042?', OrderDB())  # caller passes a real DB
    assert fake.lookups == ['1042']
    assert ticket.priority == 'normal'
```

Note two details. First, `info.output_tools[0].name` gives you the output tool's name, so you're not hardcoding `final_result`. Second, the dependency test proves that `override(deps=...)` wins over the deps the caller passed. That's how you inject a fake database or HTTP client into code that builds its own real one, with zero changes to that code. How to design those deps so they stay cheap to swap is the subject of my [Pydantic AI dependency injection guide](https://www.jahanzaib.ai/blog/pydantic-ai-dependency-injection).

When I write these for an agent going to production, I script the turns that carry business risk: the lookup that should raise priority, the tool that must not run, the refund path.

## Step 6: Test what happens when the model returns bad output

Models sometimes return values your schema rejects. Pydantic AI validates the output and sends a retry prompt back to the model. You can test that loop directly by scripting a model that gets it wrong once.

```
from pydantic_ai import RetryPromptPart


def bad_then_good(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
    retried = any(isinstance(p, RetryPromptPart) for m in messages for p in m.parts)
    priority = 'high' if retried else 'urgent'  # 'urgent' is not an allowed value
    args = {'category': 'refund', 'priority': priority, 'summary': 'Refund request'}
    return ModelResponse(parts=[ToolCallPart(info.output_tools[0].name, args)])


async def test_invalid_output_is_retried():
    with capture_run_messages() as messages:
        with triage_agent.override(model=FunctionModel(bad_then_good)):
            ticket = await triage('Refund me now', OrderDB())
    retries = [p for m in messages for p in m.parts if isinstance(p, RetryPromptPart)]
    assert len(retries) == 1
    assert 'priority' in str(retries[0].content)
    assert ticket.priority == 'high'
```

This is the retry prompt Pydantic AI actually sent back to the stub model, printed with `retries[0].model_response()`:

````
1 validation error:
```json
[
  {
    "type": "literal_error",
    "loc": [
      "priority"
    ],
    "msg": "Input should be 'low', 'normal' or 'high'",
    "input": "urgent"
  }
]
```

Fix the errors and try again.
````

A real model sees that text on its second attempt, so this test checks both your retry budget and whether the error is clear enough to fix. If your `Ticket` has a validator with a vague error, this is where you'll find out.

## Step 7: Run the suite and read the failures

```
PYDANTIC_AI_NO_BANNER=1 python -m pytest test_triage.py -v
```

```
test_triage.py::test_real_model_is_blocked PASSED                        [ 16%]
test_triage.py::test_testmodel_calls_every_tool PASSED                   [ 33%]
test_triage.py::test_fixed_structured_output PASSED                      [ 50%]
test_triage.py::test_tool_result_drives_priority PASSED                  [ 66%]
test_triage.py::test_override_swaps_dependencies PASSED                  [ 83%]
test_triage.py::test_invalid_output_is_retried PASSED                    [100%]

============================== 6 passed in 0.58s ===============================
```

Six tests in 0.58 seconds, no network. Here's what happens when a test calls `triage()` and forgets the override, from a throwaway file that is not part of the suite:

```
import pytest

from triage_app import OrderDB, triage

pytestmark = pytest.mark.anyio


async def test_refund_ticket():
    ticket = await triage('I want my money back for order 1042', OrderDB())
    assert ticket.category == 'refund'
```

```
E   RuntimeError: Model requests are not allowed, since ALLOW_MODEL_REQUESTS is False
FAILED test_forgot_override.py::test_refund_ticket - RuntimeError: Model requ...
1 failed in 0.46s
```

That's the safety switch doing its job.

## Step 8: Build an eval dataset with pydantic-evals

Unit tests answer "is the code right?". They can't answer "does the agent classify well?", because a real model's answers vary and no single case proves much. That's what [Pydantic Evals](https://pydantic.dev/docs/ai/evals/evals/) is for: a dataset of cases, a task that runs your agent on each one, and evaluators that score the results. If you want the general idea first, my glossary entry on an [eval harness](https://www.jahanzaib.ai/glossary/eval-harness) covers it.

```
from dataclasses import dataclass
from typing import Any

import logfire
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import Evaluator, EvaluatorContext, MaxToolCalls, ToolCorrectness

from triage_app import OrderDB, Ticket, triage_agent

# Spans stay on this machine: nothing is sent without a token.
logfire.configure(send_to_logfire=False, console=False)
logfire.instrument_pydantic_ai()


@dataclass
class FieldsMatch(Evaluator[str, Ticket]):
    """Pass when every field in expected_output equals the same field on the Ticket."""

    def evaluate(self, ctx: EvaluatorContext[str, Ticket]) -> dict[str, bool]:
        expected: dict[str, Any] = ctx.expected_output or {}
        return {f'{k}_ok': getattr(ctx.output, k) == v for k, v in expected.items()}


def build_dataset() -> Dataset:
    return Dataset[str, Any, Any](
        name='triage_regressions',
        cases=[
            Case(
                name='lost_parcel',
                inputs='Where is order 1042? It has not arrived.',
                expected_output={'category': 'shipping', 'order_id': '1042'},
                evaluators=(ToolCorrectness(expected_tools=['lookup_order']),),
            ),
            Case(
                name='plain_refund',
                inputs='I want a refund, the mug arrived broken.',
                expected_output={'category': 'refund'},
            ),
            Case(
                name='password_reset',
                inputs='I cannot log in after the password reset.',
                expected_output={'category': 'account'},
                evaluators=(ToolCorrectness(expected_tools=[]),),
            ),
        ],
        evaluators=[FieldsMatch(), MaxToolCalls(max_calls=2)],
    )


def make_task(model):
    async def run_triage(message: str) -> Ticket:
        db = OrderDB(orders={'1042': 'in transit', '2210': 'delivered'})
        with triage_agent.override(model=model):
            result = await triage_agent.run(message, deps=db)
        return result.output

    return run_triage
```

The evaluators split into two kinds. `FieldsMatch` checks the output, and because it returns a dict, each field shows up as its own pass or fail. `ToolCorrectness` and `MaxToolCalls` are from the [agentic evaluators](https://pydantic.dev/docs/ai/evals/evaluators/agentic/) and check the trajectory instead: which tools ran and how many times. None of them call an LLM. For subjective qualities like tone you'd add an [LLM as a judge](https://www.jahanzaib.ai/glossary/llm-as-judge) evaluator, which does spend tokens on every case.

The trajectory evaluators read OpenTelemetry spans, and this is where I hit the one real gap in the docs. Their page says the logfire SDK "must be installed and configured" and that no account is needed, which is true. But its `ToolCorrectness` example shows no setup at all, and `logfire.instrument_pydantic_ai()` only turns up under Troubleshooting on the separate Logfire integration page. I tested both half measures:

-   No `logfire.configure()`: the check fails with "No span tree available", plus a hint to configure instrumentation.
-   `logfire.configure()` but no `logfire.instrument_pydantic_ai()`: it fails with `missing tools: 'lookup_order' (x1)`, which looks exactly like an agent bug.

You need both lines. `send_to_logfire=False` keeps every span on your machine.

For the runs in this guide, the task uses a FunctionModel that classifies by keywords and stands in for the real model, so the whole pipeline runs offline. In your suite, `make_task` runs the agent on its production model, and that's the only line that changes. I kept the stub in its own file, `stub_models.py`, so it can't leak into application code. Version 1 has the bug you'll catch in Step 9; version 2 has the fix.

```
"""Stand ins for the real model, so the eval harness runs with no API key.

In a real suite the eval task calls the agent on its production model.
These two functions only exist so the pipeline can be shown end to end.
"""
import re

from pydantic_ai import ModelMessage, ModelResponse, ToolCallPart, ToolReturnPart, UserPromptPart
from pydantic_ai.models.function import AgentInfo, FunctionModel

REFUND_WORDS_V1 = ('refund', 'money back')
REFUND_WORDS_V2 = REFUND_WORDS_V1 + ('charged twice', 'double charge')


def _prompt(messages: list[ModelMessage]) -> str:
    for part in messages[0].parts:
        if isinstance(part, UserPromptPart):
            return str(part.content)
    return ''


def _classify(text: str, refund_words: tuple[str, ...]) -> str:
    text = text.lower()
    if any(w in text for w in refund_words):
        return 'refund'
    if 'password' in text or 'log in' in text:
        return 'account'
    if 'where is' in text or 'tracking' in text:
        return 'shipping'
    return 'other'


def _make(refund_words: tuple[str, ...]):
    def model(messages: list[ModelMessage], info: AgentInfo) -> ModelResponse:
        text = _prompt(messages)
        order = re.search(r'order (\d+)', text)
        looked_up = any(isinstance(p, ToolReturnPart) for m in messages for p in m.parts)
        if order and not looked_up:
            return ModelResponse(parts=[ToolCallPart('lookup_order', {'order_id': order.group(1)})])
        args = {
            'category': _classify(text, refund_words),
            'priority': 'normal',
            'order_id': order.group(1) if order else None,
            'summary': text[:60],
        }
        return ModelResponse(parts=[ToolCallPart(info.output_tools[0].name, args)])

    return model


model_v1 = FunctionModel(_make(REFUND_WORDS_V1), model_name='triage-v1')
model_v2 = FunctionModel(_make(REFUND_WORDS_V2), model_name='triage-v2')
```

Save the dataset code above as `evals_triage.py`. `make_task(model_v1)` or `make_task(model_v2)` picks which version the eval runs.

## Step 9: Turn a failed production run into an eval case

Here's the loop, end to end, with nothing hosted.

![Loop diagram: a production run with a bad output becomes an eval case, joins the dataset YAML and gates CI](https://cdn.sanity.io/images/qajb7q5q/production/3310a8ee34bad43b99ce2e54ff34bcd378c5ff41-1376x768.png?w=1200&q=75&auto=format&fit=max)

_A bad answer in production becomes a named case with the corrected expectation, so the next version of the agent has to pass it before it ships._

First, log runs the way a production app would. `all_messages_json()` serialises the full conversation, tool calls included. This is `prod_run.py`, using the buggy version 1:

```
"""Simulates one production request and logs the run the way an app would."""
import asyncio
from pathlib import Path

from stub_models import model_v1
from triage_app import OrderDB, triage_agent


async def main():
    db = OrderDB(orders={'2210': 'delivered'})
    with triage_agent.override(model=model_v1):
        result = await triage_agent.run('I was charged twice for order 2210, please fix it.', deps=db)
    print('production output:', result.output)
    Path('prod_logs').mkdir(exist_ok=True)
    Path('prod_logs/run-0001.json').write_bytes(result.all_messages_json())


asyncio.run(main())
```

```
production output: category='other' priority='normal' order_id='2210' summary='I was charged twice for order 2210, please fix it.'
```

A double charge is a refund, and the agent filed it as `other`. Someone spots it. Now the logged run becomes a case: load the messages back with `ModelMessagesTypeAdapter`, pull out the prompt, the wrong answer and the tools that ran, then append a case with the corrected expectation. This is `promote_case.py`:

```
"""Turn a logged production run into a regression case."""
import sys
from pathlib import Path
from typing import Any

from pydantic_ai import ModelMessagesTypeAdapter, ToolCallPart, UserPromptPart
from pydantic_evals import Case, Dataset
from pydantic_evals.evaluators import ToolCorrectness

from evals_triage import FieldsMatch

log_file, case_name, expected_category = sys.argv[1:4]
messages = ModelMessagesTypeAdapter.validate_json(Path(log_file).read_bytes())

prompt = next(p.content for m in messages for p in m.parts if isinstance(p, UserPromptPart))
tools = [p for m in messages for p in m.parts if isinstance(p, ToolCallPart)]
bad_output = tools[-1].args_as_dict()  # the final_result call is the agent's answer
used_tools = [t.tool_name for t in tools[:-1]]

dataset = Dataset[str, Any, Any].from_file(
    'triage_regressions.yaml', custom_evaluator_types=[FieldsMatch]
)
dataset.add_case(
    name=case_name,
    inputs=prompt,
    expected_output={'category': expected_category, 'order_id': bad_output.get('order_id')},
    metadata={'source': log_file, 'bad_output': bad_output},
    evaluators=[ToolCorrectness(expected_tools=used_tools)],
)
dataset.to_file('triage_regressions.yaml')
print(f'added {case_name}: {prompt!r} was {bad_output["category"]!r}, expected {expected_category!r}')
```

`save_dataset.py` writes the starting YAML:

```
from evals_triage import build_dataset

build_dataset().to_file('triage_regressions.yaml')
```

Create the YAML, log the bad run, then promote it:

```
python save_dataset.py
python prod_run.py
python promote_case.py prod_logs/run-0001.json double_charge refund
```

```
added double_charge: 'I was charged twice for order 2210, please fix it.' was 'other', expected 'refund'
```

Store the bad output in `metadata`. Whoever reads the YAML later sees what the agent used to say. `to_file` also writes a JSON schema next to the YAML, so editors autocomplete the case format, as the [dataset management guide](https://pydantic.dev/docs/ai/evals/how-to/dataset-management/) describes.

Run the dataset against the version that failed with `python run_eval.py v1`, where `run_eval.py` is:

```
import sys
from typing import Any

from pydantic_evals import Dataset

from evals_triage import FieldsMatch, make_task
import stub_models

model = getattr(stub_models, f'model_{sys.argv[1]}')
dataset = Dataset[str, Any, Any].from_file('triage_regressions.yaml', custom_evaluator_types=[FieldsMatch])
report = dataset.evaluate_sync(make_task(model), name=f'triage-{sys.argv[1]}', progress=False)
report.print(include_input=False, include_output=False, include_durations=False)

for case in report.cases:
    failed = [name for name, result in case.assertions.items() if result.value is not True]
    if failed:
        print(f'FAILED {case.name}: {failed}')
```

```
            Evaluation Summary: triage-v1
┏━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━━━━━━━━━━┳━━━━━━━━━━━━┓
┃ Case ID        ┃ Metrics             ┃ Assertions ┃
┡━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━━━━━━━━━━╇━━━━━━━━━━━━┩
│ lost_parcel    │ requests: 2         │ ✔✔✔✔       │
│ plain_refund   │ requests: 1         │ ✔✔         │
│ password_reset │ requests: 1         │ ✔✔✔        │
│ double_charge  │ requests: 2         │ ✔✗✔✔       │
│ Averages       │ requests: 1.50      │ 92.3% ✔    │
└────────────────┴─────────────────────┴────────────┘
FAILED double_charge: ['category_ok']
```

I trimmed the token rows. The new case fails on category and passes everything else, including the check that it looked up order 2210. After the fix (in the stub, a wider refund rule; in your app, a prompt or model change), the same dataset reports `double_charge ✔✔✔✔` and `Averages 100.0% ✔`.

### Doing the same thing in Logfire

If you already send traces to [Logfire](https://pydantic.dev/docs/logfire/evaluate/manage-datasets/), the hosted version of this step takes a few clicks. Open **Live view**, select the span, select the **database icon (+)** in the span details, choose or create a hosted dataset, then review the extracted input, expected output and metadata before saving. The case keeps a link to its source trace. There's also an experimental [Datasets SDK](https://pydantic.dev/docs/logfire/evaluate/datasets-sdk/) (`LogfireAPIClient` with `push_dataset`, `add_cases` and `get_dataset`) that needs an API key with the dataset read or write scopes. One limit to know: a hosted dataset holds at most 10,000 cases. And experiments still run from your code; the web UI prepares datasets and analyses results but doesn't execute the task.

## Step 10: Make the eval a CI gate

Save this as `test_evals.py`. It fails the build when any case assertion fails.

```
import os
from typing import Any

import pytest
from pydantic_evals import Dataset

import stub_models
from evals_triage import FieldsMatch, make_task

MODEL = getattr(stub_models, os.environ.get('TRIAGE_MODEL', 'model_v2'))


@pytest.mark.anyio
async def test_regression_set_passes():
    dataset = Dataset[str, Any, Any].from_file(
        'triage_regressions.yaml', custom_evaluator_types=[FieldsMatch]
    )
    report = await dataset.evaluate(make_task(MODEL), progress=False)
    failures = {
        case.name: [n for n, r in case.assertions.items() if r.value is not True]
        for case in report.cases
    }
    failures = {name: names for name, names in failures.items() if names}
    assert not failures, f'eval cases failing: {failures}'
```

Pointed at the old version with `TRIAGE_MODEL=model_v1 python -m pytest test_evals.py`, the gate fails with the exact case and check:

```
E   AssertionError: eval cases failing: {'double_charge': ['category_ok']}
FAILED test_evals.py::test_regression_set_passes - AssertionError: eval cases...
1 failed in 0.28s
```

Pointed at the fixed version, together with the unit tests:

```
.......                                                                  [100%]
7 passed in 0.71s
```

With a real model behind the task, I wouldn't gate on 100 percent. Model output varies run to run, so I gate on the cases that came from production incidents (those must pass) and track the average for the rest. The hard gate belongs on regressions, not on the whole set.

## Troubleshooting Pydantic AI testing

-   **`RuntimeError: Model requests are not allowed, since ALLOW_MODEL_REQUESTS is False`.** A test reached the real model. Override the model, or use a fixture that does it for every test.
-   **`UserError: Set the OPENAI_API_KEY environment variable`.** Provider setup ran before the safety switch. Add the placeholder key from Step 1.
-   **`Failed: async def functions are not natively supported`.** pytest 9 fails async tests that have no async plugin marker. Add `pytestmark = pytest.mark.anyio` and the `anyio_backend` fixture.
-   **`ToolCorrectness` fails with "No span tree available" or `missing tools`.** Call both Logfire lines from Step 8 first.
-   **`UserWarning: Could not determine the generic parameters`.** Create and load datasets as `Dataset[str, Any, Any]` (or your real input, output and metadata types) when saving to or reading from files.
-   **`TypeError: EvaluationReport.print() got an unexpected keyword argument 'include_metrics'`.** That argument doesn't exist in 2.56.0. The options are things like `include_input`, `include_output`, `include_durations` and `include_reasons`.
-   **Your agent uses provider native tools.** TestModel can't emulate them. The testing guide says to override with `native_tools=[]`, and the agentic evaluators can't see them either, because they produce no local span.

## What Logfire adds, and what it costs

Everything above runs locally for free. Logfire adds a shared UI for datasets and experiments, which matters once more than one person curates cases. Here's what each piece costs, according to the [Logfire pricing page](https://pydantic.dev/pricing) on October 15, 2026:

| What you want | Where it runs | Plan needed |
| --- | --- | --- |
| Unit tests with TestModel and FunctionModel | Your machine or CI | None |
| pydantic-evals datasets in YAML, tool call evaluators | Your machine or CI, logfire SDK capturing spans locally | None, no account |
| Traces and eval results in the Logfire UI | Logfire | Personal is free: 10 million records a month, hard capped, 30 day retention, 1 seat |
| "AI output evaluations + human review" | Logfire | Listed as a Team plan highlight at $49 a month (5 seats, 10 million records then $2 per million); not in the Personal feature list |
| Longer retention, unlimited seats | Logfire | Growth at $249 a month, up to 90 day retention |

The pricing page has no row that says outright whether hosted datasets work on Personal. So if your plan depends on adding production traces to a hosted dataset, test that on your own account before you design around it. The local loop in Step 9 needs no plan at all.

Logfire also has a prompt optimizer. In a [July 17, 2026 post](https://pydantic.dev/articles/logfire-prompt-optimization), Pydantic describes an Optimize tab that reads up to a hundred conversations from the last five days, weights failures highest, and proposes a single prompt edit with the supporting traces quoted next to the diff. I haven't run it, because it needs a hosted project with real traffic. My take: treat every proposed edit like a pull request, and run it against your regression dataset before you accept it.

Some guides record real provider responses as VCR cassettes and replay them. I skip that for agents: a replayed answer hides the variation evals exist to catch, and the recording goes stale the day the prompt changes. [Online evaluation](https://pydantic.dev/docs/ai/evals/online-evaluation/) attaches evaluators to live traffic with a per evaluator sample rate, so a cheap check can run on every request and an LLM judge on 1 percent. If you're comparing frameworks, the same no token approach works in TypeScript, as I showed in [testing a LangGraph JS agent without tokens](https://www.jahanzaib.ai/blog/langgraph-js). For how testing fits next to the rest of a self hosted stack, see [how to build your own AI agent](https://www.jahanzaib.ai/blog/how-to-build-your-own-ai-agent), and for what to trace once it ships, my note on [observability](https://www.jahanzaib.ai/glossary/observability).

When not to use evals: if a rule is deterministic, like "an order number always triggers a lookup", it belongs in a unit test with FunctionModel. Evals are for behaviour that depends on the model's judgement, and they cost real tokens once a real model sits behind the task.

[Video: Agent Optimization with Pydantic AI: GEPA, Evals, Feedback Loops](https://www.youtube.com/watch?v=A48uhxfxbsM)

_Samuel Colvin's workshop at AI Engineer. Skip to 16:55 for running evals against a golden dataset, and 25:16 for comparing prompt variants._

If you'd rather have an agent built with this test suite and eval loop already in place, that's the work I do: [see how I build AI agents](https://www.jahanzaib.ai/agents).

## Frequently asked questions

### What is TestModel in Pydantic AI?

TestModel is a stub model that ships with Pydantic AI. By default it calls every tool on your agent with arguments generated from each tool's schema, then returns output that passes your output type's validation. It doesn't read the prompt and makes no network calls, so it's ideal for checking wiring and useless for checking answers.

### When should I use FunctionModel instead of TestModel?

Use FunctionModel when the test depends on what the model says at a specific turn: calling a tool with a particular argument, reacting to a tool result, or returning invalid output to exercise retries. You write a Python function that receives the message history and returns the response, so every turn is under your control.

### How do I stop tests from calling a real LLM by accident?

Set `pydantic_ai.models.ALLOW_MODEL_REQUESTS = False` in your conftest. Any test that reaches a real model then raises a RuntimeError instead of sending a request. Combine it with `Agent.override(model=...)` so application code runs against a stub, and with `defer_model_check=True` so modules import without credentials.

### Do I need a Logfire account to use pydantic-evals?

No. pydantic-evals runs datasets and prints reports locally. The tool call evaluators need the logfire SDK installed and configured to capture spans, but the docs confirm no account is needed. Use `logfire.configure(send_to_logfire=False)` plus `logfire.instrument_pydantic_ai()` and everything stays on your machine.

### How do I turn a failed production run into an evaluation case in Logfire?

Select the span in Live view and use the database icon (+) to add it to a hosted dataset, then review the extracted fields and save; the case links back to its trace. Without Logfire, log `all_messages_json()` and convert it with `Dataset.add_case`, as in Step 9.

### Can Logfire optimize my agent's prompts from production traces?

Pydantic says yes, through an Optimize tab that proposes one prompt edit backed by quoted traces. It needs a hosted project with real traffic. I'd run any proposed edit against a regression dataset before accepting it.

## Related

- [How to Use Pydantic AI Output Types Now That result_type Is Gone](https://www.jahanzaib.ai/blog/pydantic-ai-output-types)
- [How to Build Pydantic AI Multi Agent Apps Without Losing Track of Usage](https://www.jahanzaib.ai/blog/pydantic-ai-multi-agent)
- [How to Use Pydantic AI Dependency Injection With deps_type and RunContext](https://www.jahanzaib.ai/blog/pydantic-ai-dependency-injection)

---

Canonical HTML version: https://www.jahanzaib.ai/blog/pydantic-ai-testing