Evals

Scenario tests and @b4run/testing answer "is it correct?" for a handful of cases. Evals answer "how good is it?". They run your agent over a whole dataset, score each output and aggregate the scores into a single verdict you can gate on. Use them when you want a quality bar that survives prompt edits, model swaps and refactors.

By default, each case replays pre-written aimock fixtures, just like @b4run/testing, so evals are safe to run in CI. b4 eval --live measures the real model locally.

InstallCopy link to section: Install

Install both packages as dev dependencies:

bash

npm install -D @b4run/evals @b4run/testing

@b4run/evals provides defineEval and the scorers. @b4run/testing provides script() and the harness the b4 eval command drives.

The `.eval.ts` conventionCopy link to section: The .eval.ts convention

Evals live next to the route they exercise, in an evals/ directory:

text

src/app/chat/
  index.ts
  evals/
    quality.eval.ts

b4 eval discovers src/app/<route>/evals/*.eval.ts the same way b4 test discovers run.test.ts. An eval under a route directory binds to that route automatically. You only need the route field when the file lives elsewhere.

`defineEval`Copy link to section: defineEval

Here is a one-case eval:

src/app/chat/evals/quality.eval.ts

import { contains, defineEval } from "@b4run/evals"
import { script } from "@b4run/testing"
 
export default defineEval({
  name: "chat quality",
  // route: "/chat#agent",  // optional: inferred from the file location
  dataset: [
    {
      name: "greets the user",
      input: "hello",
      fixtures: script().user("hello").replies("Hi! How can I help?"),
    },
  ],
  scorers: [contains("help", { threshold: 1 })],
  threshold: 1,
})

The fields:

name: required label for the eval, shown in the report.
route: the Agent Protocol route key ("/chat#agent"). Optional when the file lives under a route, required otherwise.
dataset: the cases to run (see below).
scorers: one or more scorers. Each runs against every case.
middlewareContext: the context your middleware.ts would have produced, as a value or a (run) => context function evaluated per case. b4 eval runs the route in-process and bypasses middleware, so tools that read ctx.middleware need it supplied here. See Supply the middleware context.
responseSchema: the JSON Schema your client sends as hashbrown.responseSchema. b4 eval binds it on the root model exactly as the server does for an AG-UI run, so replayed, live and recorded cases send the model the request production sends. See Constrain the final message.
threshold: sugar for gate.mean(threshold) (see Gating).
gate: a composable gate policy. It takes precedence over threshold.

A dataset case is { input, expected?, name?, fixtures?, metadata? }. input is the user message for an agent route. fixtures is the per-case aimock script used in replay mode (--live ignores it).

DatasetsCopy link to section: Datasets

dataset accepts three shapes:

// 1. Inline array of cases
dataset: [{ input: "hello", expected: "Hi!" }]
 
// 2. A path to a committed file (.json or .jsonl), relative to the eval file
dataset: "./cases.json"
dataset: "./cases.jsonl"
 
// 3. A sync or async function returning cases
dataset: async () => loadCasesFromSomewhere()

A .json file must contain a JSON array of cases. A .jsonl file holds one case object per line. Relative paths resolve against the eval file's directory.

ScorersCopy link to section: Scorers

A scorer reads the run result and the case and returns a score: a number in 0..1, a boolean (true = 1, false = 0), or a rich verdict { score, label?, reason? }. Built-in scorers:

exactMatch(): final message equals case.expected.
contains(substring): final message includes the substring.
regex(re): the regex matches the final message.
jsonEquals(): case.expected deep-equals the parsed final message (or a value you select).
toolCalled(name, { withArgs? }): a tool was called (optionally with matching args).
tokensUnder(budget): fewer than budget tokens were streamed.
memoryRecalled(expectedIds): every expected memory id appears in a recall tool result.
memoryFresh(expectedValue): the final message contains the expected newer value.
memoryIsolated(forbidden): the forbidden value appears in neither recall results nor the final message.

Each built-in takes an optional { threshold }. That's its own pass bar, used by gate.perScorer and per-case pass logic.

For anything else, write your own:

import { custom, llmJudge } from "@b4run/evals"
 
// Arbitrary scoring logic
custom((run, testCase) => (run.finalMessage.length < 200 ? 1 : 0), { name: "concise" })
 
// Grade output quality with a model
llmJudge({
  criteria: "Does the answer fully address {{input}}? Output: {{output}}",
  model: "gpt-5-mini", // optional
  threshold: 0.7,       // optional
})

custom((run, testCase) => Score | Promise<Score>) gives you the full run result and case. llmJudge asks a model to grade the output against criteria (which interpolates {{input}}, {{expected}}, {{output}}) and returns { score, reason }.

The judge sends a chat-completions request whenever it runs. In B4.run CLI replay, that request goes through aimock, and fixtures can answer it. In live, record or unmocked programmatic runs, it needs model credentials unless you inject a mocked fetchImpl.

GatingCopy link to section: Gating

After every case is scored, a gate turns the aggregated scores into pass/fail. Gate policies compose via the gate helper:

import { gate } from "@b4run/evals"
 
gate: gate.mean(0.8)        // dataset-wide mean ≥ 0.8
gate: gate.passRate(0.9)    // ≥ 90% of cases pass (every scorer met its bar)
gate: gate.everyCase(0.6)   // every case's mean ≥ 0.6
gate: gate.perScorer()      // each scorer's mean ≥ that scorer's own threshold
 
// Combine them:
gate: gate.all(gate.mean(0.8), gate.perScorer())   // all must pass
gate: gate.any(gate.passRate(0.9), gate.mean(0.95)) // any one passing is enough

There are two shorthands:

threshold: on the eval is sugar for gate.mean(threshold). If gate is set, threshold is ignored.
A per-scorer threshold sets that scorer's own bar. gate.perScorer() uses it, and so does the check for whether a case "passed" (every scorer must meet its bar, and the default bar is 0.5).

If you set neither gate nor threshold, the eval is informational. It always passes and just reports scores, so you can land an eval and watch the numbers before you commit to a bar.

Execution: replay vs liveCopy link to section: Execution: replay vs live

Evals use the same aimock fixture system as agent tests. See Fixtures and Recording for how fixtures, script() and replay-vs-live mode work, and why live runs must never touch CI.

By default b4 eval runs in replay mode. Each case replays its fixtures (a script() builder inline on the case, or a committed fixture file) through aimock, so the agent's model call is replaced. This is the mode to run in CI. If the eval includes llmJudge, the scorer still runs during replay, so include a fixture for its judge request or inject a mocked fetchImpl.

bash

b4 eval

Pass --live to run the real model locally:

bash

b4 eval --live

Live mode ignores per-case fixtures and calls the actual provider. It requires OPENAI_API_KEY and is meant for local prompt tuning. Keep both --live and --record out of CI.

Recording fixtures with `--record`Copy link to section: Recording fixtures with --record

b4 eval --record runs the suite against the real model (it requires OPENAI_API_KEY) and writes agent-run request fixtures you can replay later. For each case that has no inline script() fixtures, it writes a sibling file <evalBasename>.<caseSlug>.fixtures.json next to the .eval.ts. A plain b4 eval then auto-loads those files, so the agent run stays deterministic in CI without any code changes.

Recording snapshots only the agent run. Scorer model calls stay out of that sibling file. If the eval uses llmJudge, separately add and commit a fixture that matches the judge request, or inject a mocked fetchImpl into the scorer before replaying in CI.

--record skips cases that already have inline script() fixtures (logging skipped record (inline fixtures)) and never overwrites them. The gate still applies during --record, and scores are checked against the threshold after all cases run. Fixture files are written per case before the verdict, so a gate failure never discards captured responses.

A recording is checked against the same rules replay loads it with, so --record never writes a file that plain b4 eval would reject. The usual cause is a model turn that came back empty: the model refused, ran out of output tokens, or had nothing left to say after a tool did the work. In that case the command exits with code 2 and names the case and turn, and that case's file is not written. If the answer is a tool call, export returnDirect from that tool so the run ends on its result.

bash

# Record real-model responses into sibling fixture files
b4 eval --record
 
# Replay the committed agent fixtures; supply scorer fixtures separately
b4 eval

Command referenceCopy link to section: Command reference

text

b4 eval [path] [--live] [--record] [--json [file]] [--cwd <path>]

[path]: narrow discovery to a subdirectory.
--live: run the real model (requires OPENAI_API_KEY).
--record: record real-model responses into sibling fixture files (requires OPENAI_API_KEY). Mutually exclusive with --live.
--json [file]: also write a JSON report. Defaults to .b4/eval-report.json.
--cwd <path>: operate on a different app root.

A gated eval that fails causes a non-zero exit, so CI fails when quality drops below the bar. Informational evals never affect the exit code. See the CLI reference for the full command surface.

A full exampleCopy link to section: A full example

Let's put a tool check and a text check behind a per-scorer gate:

src/app/chat/evals/quality.eval.ts

import { contains, defineEval, gate, toolCalled } from "@b4run/evals"
import { script } from "@b4run/testing"
 
export default defineEval({
  name: "chat quality",
  dataset: [
    {
      name: "filters open items",
      input: "Filter open items",
      fixtures: script()
        .user("Filter open items")
        .callsTool("applyFilter", { status: "open" })
        .replies("Found 2 open items. Let me know if I can help further."),
    },
  ],
  scorers: [
    contains("help", { threshold: 0.5 }),
    toolCalled("applyFilter", { threshold: 0.5 }),
  ],
  gate: gate.perScorer(),
})

Run it:

bash

b4 eval

The single case calls applyFilter and its reply contains "help", so both scorers score 1.00. gate.perScorer() checks each scorer's mean against its own bar. Both are 1.00 ≥ 0.5, so the eval passes:

text

PASS chat quality › filters open items mean=1.00 [contains(help)=1.00 toolCalled(applyFilter)=1.00]
PASS chat quality mean=1.00

Commit the eval and its fixture files alongside your route. CI replays them, including any llmJudge calls.

Your scaffolded app already has an evalCopy link to section: Your scaffolded app already has an eval

The default basic template generates src/app/hello/evals/smoke.eval.ts, a one-case replay eval for the /hello agent, and an eval npm script. The navlog template (npm create b4-app@latest my-agent -- --template navlog) generates server/src/app/navlog/evals/navlog-quality.eval.ts next to its flight-planning route, adds @b4run/evals as a dev dependency of the server package, and wires an eval npm script at the workspace root. In either app, run it right after scaffolding:

bash

npm run eval

This wraps b4 eval and runs the agent cases in replay mode by default. The navlog eval (server/src/app/navlog/evals/navlog-quality.eval.ts) has three dataset cases, each with inline script() fixtures, and these scorers:

toolCalled("computeNavlog", { threshold: 1 }), and toolCalled("task", { withArgs: { subagent: "weather" } }) and its performance twin, assert the plan used the navlog tool and both subagents.
Custom navlog-shape and totals-add-up-and-reserve scorers parse computeNavlog's result and check its legs, totals and legal reserve.
A custom cites-poh scorer asserts the brief cites a POH file that exists in the workspace, and files-only-when-asked that no flight plan was filed unasked.
llmJudge({ criteria: "The brief names the flight category at every airport, …", model: "gpt-5-mini", threshold: 0.7 }).

The gate is gate.all(gate.passRate(1), gate.perScorer()), so every case must pass and every scorer must meet its threshold. The scripted fixtures include the judge turn used by llmJudge, so the default npm run eval needs no key; the tools still run, so the weather tools reach aviationweather.gov.

To run the agent cases against the real model locally:

bash

npm run eval -- --live

Grow it by adding cases to the dataset and more scorers.

Programmatic APICopy link to section: Programmatic API

runEval from @b4run/evals lets you drive an eval definition from your own script, for example in custom CI tooling or a reporting pipeline:

scripts/run-chat-eval.ts

import { fileURLToPath } from "node:url"
import { contains, defineEval, runEval } from "@b4run/evals"
import { createAgentHarness, script } from "@b4run/testing"
 
const appRoot = fileURLToPath(new URL("..", import.meta.url))
const evalDir = fileURLToPath(new URL("../src/app/chat/evals", import.meta.url))
 
const myEval = defineEval({
  name: "chat quality",
  dataset: [{ input: "hello", fixtures: script().user("hello").replies("Hi!") }],
  scorers: [contains("Hi", { threshold: 1 })],
  threshold: 1,
})
 
const harness = await createAgentHarness({ appRoot, route: "/chat#agent" })
try {
  const report = await runEval(myEval, {
    runCase: async (testCase) => {
      harness.reset()
      if (typeof testCase.input !== "string") {
        throw new TypeError("Agent eval input must be a string")
      }
      return harness.run({
        input: testCase.input,
        ...(testCase.fixtures !== undefined ? { fixtures: testCase.fixtures } : {}),
      })
    },
    baseDir: evalDir, // for resolving dataset paths
  })
 
  console.log(report.passed, report.mean)
} finally {
  await harness.close()
}