Guides

AI agent evaluation: a test plan for small teams

Salesforce just announced Agent Evals and OpenAI's Evals platform is shutting down. Here's how a small team can test AI agents without either product.

S

SelfAgentic Team

7 min read
A hand holding a fountain pen ticks boxes on a handwritten checklist in a notebook.

On October 27, Salesforce announced Agent Evals for its Agent Fabric platform, which generates tests for AI agents before deployment and keeps scoring them once they're live. Four days later, on October 31, OpenAI's Evals platform goes read-only, and it shuts down on November 30, so anyone who kept agent tests there has a month to move them.

A small team can do AI agent evaluation well without either product. You need 20 to 30 real tasks with known good outcomes, a pass or fail rule for each one, a check on the steps the agent took as well as what it produced, and the habit of rerunning the whole set every time you change a prompt, a tool or a model. After launch, read a sample of live runs every week, because test sets go stale.

What changed in agent evals this week

Salesforce's Agent Evals and a companion feature, API Evals, enter beta in November 2026, with general availability planned for January 2027. When an agent's scores fall below a quality threshold, Agent Evals can trigger a predefined response, including Salesforce's Agent Kill Switch, with a human approval step where a team's policy requires one. The same announcement added hard spending limits that stop requests before a budget is exceeded.

The product targets large companies, but the design is worth copying at any size. Tests run before release, scoring continues in production, and the team decides in advance what happens when a score drops.

The OpenAI change is more practical. OpenAI announced the deprecation on June 3, 2026, and its deprecations page points to a migration guide for moving tests to Promptfoo. If your test cases live in the Evals dashboard, export them before the November 30 shutdown.

What is AI agent evaluation, in plain terms?

An eval is a test with three parts: an input, the outcome you expect, and a rule that decides whether the agent passed. AI agent evaluation is running a set of those tests against an agent and tracking how often it passes.

There are two timings. Offline evals run against a fixed test set before you ship a change. Online evals score real runs after the agent is live.

The rule that decides pass or fail is called a grader. OpenAI's evaluation best practices guide describes three kinds:

  • Metric-based graders, such as an exact match on a field. They're cheap and good for catching regressions, but they miss nuance.
  • Human graders, which give the best judgement but are slow and expensive.
  • LLM-as-a-judge, where a second model grades the output against a rubric. It scales well, but it can favour longer answers or whichever option it saw first, so check it against human labels before you rely on it.

The guide recommends pass or fail, or a choice between two outputs, over open-ended 1 to 10 scores.

How to test AI agents: check the steps as well as the answer

A chatbot gives you one answer to grade. An agent takes a series of actions to reach its result, and a good-looking final document can hide a bad route: the wrong tool, the wrong data passed to the right tool, or forty extra calls you paid for.

The OpenAI guide names what to test. For a single agent, check tool selection (did it pick the right tool?) and data precision (did it pass the correct arguments?). For several agents working together, check handoff accuracy, meaning whether each agent recognises when a task belongs to another. It also lists edge cases worth including: vague requests, typos, long conversations, circular handoffs, and conflicts between what the user asks and what the system instructions allow.

In a setup where a manager agent splits work across specialist agents, which is how SelfAgentic structures delegation, handoffs are the first place to look. Did the manager send the pricing question to the market analyst, or try to answer it itself from stale context?

Step-level checks only work if each run leaves a record of what the agent did. We covered what that record should contain in our post on AI agent audit trails.

A worked example: evaluating a lead research agent

Say a three-person team runs an agent overnight. It researches each new inbound lead, scores it from 1 to 5 against their ideal customer profile, and drafts a first email for anything scored 4 or higher. Here's a test plan they could set up in an afternoon:

  1. Pull 25 past leads where the founder already knows the right answer: 12 good fits and 13 poor ones.
  2. Write the pass rules. The agent's score must be within one point of the founder's. It must never draft an email for a lead scored below 4. Every claim in a draft must come from a page the agent actually opened. The run must stay under the token budget they set.
  3. Pick a grader for each rule. A short script checks the first two and the budget. An LLM judge checks the sourcing rule as pass or fail, and the founder reads five of its verdicts to confirm she agrees with them.
  4. Run all 25 and record the failures. Suppose 21 pass. Two drafts cite a company page the agent never opened, and two scores are off by two points on companies that had recently rebranded.
  5. Fix the prompt, then rerun all 25 cases, not just the four that failed. Prompt fixes often break something that used to pass.
  6. Every Monday, read five live runs from the previous week. Any new kind of failure becomes test case 26, 27 and so on.

The set grows from real mistakes, so it ends up describing this team's work far better than a public benchmark would.

Agent evals in production: the part most teams skip

LangChain's State of Agent Engineering report, based on 1,340 survey responses collected in late 2025, found that 52.4% of organisations run offline evaluations on test sets, while 37.3% run online evaluations on live traffic. Human review was the most common method at 59.8%, and 53.3% used LLM-as-a-judge. Quality was the top barrier to putting agents into production, cited by 32%.

So most teams test before launch, and fewer keep checking afterwards. The gap hurts most with agents that run on a schedule. Leads change, a connected tool changes its output, or the model gets updated, and an August test set stops matching November's work.

Production checks need two things: a score per agent over time, and a trail behind each run. SelfAgentic has built-in scoring that tracks each agent's performance, and each run records which tools it called and how many tokens it used, so a dropped score can be traced to specific runs.

Evals also have limits you should plan around:

  • They only catch failures you thought to write a test for.
  • With 25 cases, each failure moves the pass rate by four points, so one bad night can look like a trend.
  • LLM judges carry their own biases, and checking them against human labels takes time.

Because evals miss things, they work best alongside other limits. In SelfAgentic, posts and ad changes wait in an approvals inbox by default, and that catches an outward-facing mistake that no test predicted.

AI agent evaluation FAQ

How many test cases do I need to evaluate an AI agent?

Start with 20 to 30 cases drawn from real work, including a few edge cases and a few you expect it to get wrong. Add a case each time a live run fails in a new way.

What is LLM-as-a-judge?

It's using a second model to grade an agent's output against a written rubric. It's cheaper and faster than human review, but OpenAI's guide notes it can be biased by answer length and position. Check its verdicts against a human on a sample, and prefer pass or fail over numeric scores.

What's the difference between offline and online evals?

Offline evals run a fixed test set before you ship a change, so you catch regressions early. Online evals score real runs after launch, so you catch drift and failure types you didn't anticipate. You need both.

What happens to tests stored on the OpenAI Evals platform?

Existing evals become read-only on October 31, 2026, and the dashboard and API shut down on November 30, 2026. OpenAI's deprecations page links to a guide for moving to Promptfoo. Export anything you want to keep before the shutdown.

If you run agents today, write down ten tasks you already know the right answer to, plus a pass rule for each, before you change another prompt. Rerun them after every change, and add a case every time a live run surprises you.

If you'd like to try a team of agents with per-agent scoring, run records and approval gates built in, see how SelfAgentic works.

Keep reading

Ready to build your AI team?

See how a team of agents handles real work, and checks with you before anything goes out.

Free to start. No credit card required.