AI agents · 8 min read

How to test an LLM agent: traces, budgets and an eval gate in CI (Python)

An agent you cannot measure is an agent you cannot change safely. Three pieces make it measurable: traces, budgets and an eval gate.

By Homingo Learn · 2 November 2026

Unit tests cover the deterministic parts of an agent: parsers, validators, tool wrappers. They say nothing about whether the agent still does its job after you change a prompt. For that you need three things working together: traces that record what each step did, budgets that stop a run before it burns tokens in a loop, and an eval gate that scores the agent on fixed tasks and fails the build when the score drops. This guide builds all three in plain Python.

1. Trace every step as one JSON line

Write one line per model call or tool call: which agent, which task, what it did, how many tokens, how long, and how it ended. JSON Lines files are easy to grep, diff and load into a notebook.

import json
import time
from contextlib import contextmanager

class Tracer:
    def __init__(self, path: str, run_id: str):
        self.path, self.run_id = path, run_id

    @contextmanager
    def step(self, agent: str, task: str, kind: str):
        rec = {"run": self.run_id, "ts": time.time(), "agent": agent, "task": task, "kind": kind, "tokens": 0}
        start = time.perf_counter()
        try:
            yield rec  # the caller fills rec["tokens"], rec["attempt"], rec["input_chars"], ...
            rec.setdefault("status", "ok")
        except Exception as e:
            rec["status"], rec["error"] = "error", str(e)
            raise
        finally:
            rec["ms"] = round((time.perf_counter() - start) * 1000)
            with open(self.path, "a", encoding="utf-8") as f:
                f.write(json.dumps(rec, default=str) + "\n")

When a report comes out wrong, the trace tells you which agent produced the bad step, how many tokens and how long it took, and (if you fill those fields) its input size and attempt number, instead of a guess.

2. Budgets: check before every call

Agents loop. A tool returns an error, the agent tries again, and again. A budget object checked before each model call turns that into a clean stop with a clear reason:

class BudgetExceeded(RuntimeError):
    pass

class Budget:
    def __init__(self, total_tokens: int, per_agent_tokens: int, max_calls: int):
        self.total, self.per_agent, self.max_calls = total_tokens, per_agent_tokens, max_calls
        self.used, self.by_agent, self.calls = 0, {}, 0

    def check(self, agent: str, estimate: int) -> None:
        """Call before every model call; counts the attempt even if the call then fails."""
        if self.calls + 1 > self.max_calls:
            raise BudgetExceeded(f"call cap {self.max_calls} reached")
        if self.used + estimate > self.total:
            raise BudgetExceeded(f"run budget {self.total} tokens would be exceeded")
        if self.by_agent.get(agent, 0) + estimate > self.per_agent:
            raise BudgetExceeded(f"{agent} would exceed {self.per_agent} tokens")
        self.calls += 1

    def charge(self, agent: str, tokens: int) -> None:
        """Call after a model call with the real token count from the API response."""
        self.used += tokens
        self.by_agent[agent] = self.by_agent.get(agent, 0) + tokens

Charge the real token count from the API response after each call, not the estimate. The estimate (for example, prompt characters divided by four plus the maximum output tokens) only decides whether to make the call at all.

def call_model(agent, task, prompt, budget, tracer, client, max_out=800):
    budget.check(agent, estimate=len(prompt) // 4 + max_out)
    with tracer.step(agent, task, "model") as rec:
        rec["input_chars"] = len(prompt)
        reply = client.complete(prompt, max_tokens=max_out)  # your model client
        rec["tokens"] = reply.input_tokens + reply.output_tokens
    budget.charge(agent, rec["tokens"])
    return reply.text

One model call with both pieces wired in.

3. An eval set: fixed tasks, a score, a threshold

Pick a small, fixed set of tasks that represent the job: ten to twenty is enough to start. For each run, compute a score you can defend. For a research agent that writes cited reports, a good one is the share of report sentences that a separate checker marks as supported by their cited sources; add the run’s token count and failure count next to it.

import json
import statistics
import sys
import traceback

from agents.team import run_agent  # returns an object with .sentences, .supported, .tokens

def load_tasks(path: str) -> list[dict]:
    with open(path, encoding="utf-8") as f:
        return json.load(f)  # e.g. [{"topic": "..."}, ...]

def run_evals(tasks, min_score: float = 0.85) -> int:
    if not tasks:
        print("eval: no tasks")
        return 1
    scores, failures, tokens = [], 0, 0
    for task in tasks:
        try:
            result = run_agent(task)
            scores.append(result.supported / max(1, result.sentences))
            tokens += result.tokens
        except Exception:
            failures += 1
            scores.append(0.0)
            traceback.print_exc()
    mean = statistics.mean(scores)
    print(f"eval: mean {mean:.3f} over {len(tasks)} tasks, {failures} failed, {tokens} tokens, threshold {min_score}")
    return 0 if mean >= min_score and failures == 0 else 1

if __name__ == "__main__":
    sys.exit(run_evals(load_tasks("evals/tasks.json")))

evals/run.py (run with python -m evals.run). run_agent is your agent entry point.

4. Run agent evals in GitHub Actions

Run the eval script on every pull request that touches prompts, agent code or dependencies. The exit code is the gate. Keep the API key in the repository’s secrets (secrets are not passed to pull requests from forks, so the if: line skips those instead of failing them):

name: agent evals
on:
  pull_request:
    paths: ["agents/**", "prompts/**", "evals/**", "requirements.txt", ".github/workflows/evals.yml"]
jobs:
  evals:
    # secrets are empty on pull requests from forks: run only for branches of this repository
    if: github.event.pull_request.head.repo.full_name == github.repository
    runs-on: ubuntu-latest
    steps:
      - uses: actions/checkout@v4
      - uses: actions/setup-python@v5
        with:
          python-version: "3.12"
      - run: pip install -r requirements.txt
      - run: python -m evals.run
        env:
          MODEL_API_KEY: ${{ secrets.MODEL_API_KEY }}

.github/workflows/evals.yml

What to watch over time

Score and cost together. A prompt change that raises the score but doubles the tokens is a trade-off, not a win.
Failure rate. A run that crashes on one task in twenty will likely crash on a similar share of real tasks.
Variance. Model output varies between runs, so a single run near the threshold can flap. Run the set two or three times and gate on the mean, or keep a small margin above the threshold.

These pieces slot into a full agent team: see our multi-agent research system guide for the planner, researchers, shared memory and source checker they measure.