Fixtures and Recording

Fixtures are the model responses the agent test harness replays. Use an inline fixture when a scenario is short enough to read beside the test. Use a committed fixture file when several tests share an exchange, a tool chain is long, or you want to review the model traffic itself. Both replay deterministically. Recording and live mode are local opt-ins that call a real provider.

Choose inline fixtures or a committed fixture file#Copy link to section: Choose inline fixtures or a committed fixture file

Start inline with script(). A short intent → tool → answer flow is easier to maintain next to its assertions:

const fixtures = script()
  .user("Filter open items")
  .callsTool("applyFilter", { status: "open" })
  .replies("Found 2 open items.")

Move the exchange to a committed fixture file when the script obscures the test or must be reused. Fixture replay replaces model HTTP traffic only. Authored tools, capabilities, state, and any external I/O they perform still run.

How script matching works#Copy link to section: How script matching works

aimock selects each response using three match fields:

• userMessage compares against the latest user message that contains text. A string from .user() matches when that latest text contains the fixture value. A trailing attachment-only user message is skipped in favor of the nearest text-bearing one.
• turnIndex is the number of assistant messages already in the thread before that model call. It keeps counting in a continuing thread.
• hasToolResult examines only messages after the latest user message. It's false for the current turn's initial model call and true after that turn produces a tool-role result. Earlier turns' tool results do not make the next turn's initial model call match hasToolResult: true.

Within one .user() group, script() emits turnIndex: 0 for the first response, then increments it for each tool call or reply. Post-tool responses also set hasToolResult: true. Each new .user() group starts again at zero, so groups suit fresh-thread scenarios. For a same-thread follow-up, supply explicit fixtures with the cumulative turnIndex from the existing thread.

Fixture files: author, commit, replay#Copy link to section: Fixture files: author, commit, replay

Resolve a fixture beside the test once, then pass that absolute path to the file helpers:

import { fileURLToPath } from "node:url"
import { loadFixtures, script, writeFixtures } from "@b4run/testing"
 
const fixturesPath = fileURLToPath(
  new URL("fixtures/filter-open.fixture.json", import.meta.url),
)

new URL(..., import.meta.url) is relative to the test file. loadFixtures(path) and writeFixtures(path, fixtures) use the path as given, so a relative string passed directly to either helper is relative to process.cwd().

Author inline and snapshot to a file#Copy link to section: Author inline and snapshot to a file

writeFixtures accepts a script() builder or a fixture array. It creates parent directories and writes formatted { "fixtures": [...] } JSON:

writeFixtures(
  fixturesPath,
  script()
    .user("Filter open items")
    .callsTool("applyFilter", { status: "open" })
    .replies("Found 2 open items."),
)

Run this authoring step on purpose, inspect the JSON and commit it. Then take it out of the normal test path so tests only replay.

Replay a fixture file in tests#Copy link to section: Replay a fixture file in tests

loadFixtures checks only the supported top-level container (a bare array or an object whose fixtures property is an array) and returns that array. It does not validate every fixture entry. Replay also does not fall back to a provider. If no fixture matches, the model request fails.

Choose one registration scope. Put a shared file in createAgentHarness({ fixtures }), or supply it before a run as below. Do not register the same fixture file at both scopes:

import { fileURLToPath } from "node:url"
import { afterAll, it } from "vitest"
import { createAgentHarness, expectFinalMessage, loadFixtures } from "@b4run/testing"
 
const appRoot = fileURLToPath(new URL("..", import.meta.url))
const fixturesPath = fileURLToPath(
  new URL("fixtures/filter-open.fixture.json", import.meta.url),
)
const h = await createAgentHarness({ appRoot, route: "/chat#agent" })
 
afterAll(async () => {
  await h.close()
})
 
it("replays a committed exchange", async () => {
  const run = await h.run({
    input: "Filter open items",
    fixtures: loadFixtures(fixturesPath),
  })
  expectFinalMessage(run).toContain("Found 2")
})

Fixtures supplied to h.run() are appended before that run, and they stay. They persist across later h.run() calls until h.reset() clears them. Because aimock selects the first registered matching fixture, overlapping additions can leave an older fixture shadowing a newer one. Use h.reset() between independent scenarios, and omit it between turns that intentionally share one thread.

A turnIndex mismatch is nonfatal by default. aimock can select a content-matching fixture at a different assistant-message count. Set AIMOCK_STRICT_TURN_INDEX=1 in the test process when any turnIndex mismatch must reject the fixture.

Record from a real model (local only)#Copy link to section: Record from a real model (local only)

Harness recording exercises one B4.run route and turns the most recent run's captured model traffic into replay keys. Use it only for one fresh-thread first run. Construct a new harness, run one representative input right away, inspect the result, then write it. Set OPENAI_API_KEY for the default upstream:

import { fileURLToPath } from "node:url"
import { createAgentHarness, writeFixtures } from "@b4run/testing"
 
const appRoot = fileURLToPath(new URL("..", import.meta.url))
const fixturesPath = fileURLToPath(
  new URL("fixtures/filter-open.fixture.json", import.meta.url),
)
const h = await createAgentHarness({
  appRoot,
  route: "/chat#agent",
  record: true,
})
 
try {
  await h.run({ input: "Filter open items" })
  const fixtures = h.getRecordedFixtures()
  writeFixtures(fixturesPath, fixtures)
} finally {
  await h.close()
}

getRecordedFixtures() returns only traffic captured for the most recent run(). It builds each fixture's match fields from that call's own request, read the way aimock reads it on replay:

• userMessage: the latest user message with text in the request.
• hasToolResult: whether a tool-role message follows that user message.
• turnIndex: the number of assistant messages already in the request.

So each call is keyed by its position within its own agent run. A route that dispatches subagents in parallel records correctly: their calls interleave with the parent's in any order, but each subagent's request carries only its own history.

Every call in record mode goes to the upstream, including a call that an earlier recording in the same run would match. After building the fixtures, getRecordedFixtures() replays each recorded request against them. A subagent's input often contains its parent's message, and userMessage matches by substring, so one fixture can match another call's request. That fixture gets a toolName key (a tool only its own request offers) or a toolCallId key (the tool call its request answers). If nothing in the two requests tells them apart, getRecordedFixtures() throws and names both turns. It won't return fixtures that replay would serve to the wrong call.

The standalone record({ out, provider? }) helper is a different API. It launches the aimock recorder in a separate process and returns nothing. There's no harness and no route involved:

import { record } from "@b4run/testing"
 
record({ out: "test/fixtures/filter-open.fixture.json" })
// Optional upstream override:
record({
  out: "test/fixtures/filter-open.fixture.json",
  provider: "https://api.openai.com",
})

The child process resolves a relative out path against the inherited process.cwd(). Use an absolute path when the command may run from different directories. The call is synchronous and throws if the recorder exits with an error.

Live mode (real model)#Copy link to section: Live mode (real model)

Use live: true only for local prompt validation. It proxies model calls to the real OpenAI endpoint, requires OPENAI_API_KEY, and registers no fixtures. The harness still captures the system prompt, but model output is nondeterministic:

import { fileURLToPath } from "node:url"
import { it } from "vitest"
import { createAgentHarness, expectFinalMessage, expectToolCalled } from "@b4run/testing"
 
const appRoot = fileURLToPath(new URL("..", import.meta.url))
 
it.skipIf(process.env.CI || !process.env.OPENAI_API_KEY)(
  "checks the prompt against a real model",
  async () => {
    const h = await createAgentHarness({ appRoot, route: "/chat#agent", live: true })
    try {
      const run = await h.run({ input: "Filter open items" })
      expectToolCalled(run, "applyFilter")
      expectFinalMessage(run).toMatch(/open/i)
    } finally {
      await h.close()
    }
  },
  120_000,
)

Construct and close the live harness inside the skipped test, so importing the module can't start a real-model harness. Keep assertions loose. Check that a tool was called, an answer has the expected shape, or a prompt contains a stable instruction. Avoid asserting on literal arguments or response text. Never run live mode in CI, even when a provider key is present there.

CI rules and fixture drift#Copy link to section: CI rules and fixture drift

CI should only replay fixtures. Keep record and live mode local. Commit every fixture a test loads, keep recording scripts out of the test command, and fail when a fixture update is tracked or untracked:

- run: npx vitest --run
- run: test -z "$(git status --porcelain -- test/fixtures/)"

Unlike git diff, the status-based fixture drift check catches both tracked edits and untracked fixture files. Code review still decides whether a changed exchange is correct.

Process-global constraints and cleanup#Copy link to section: Process-global constraints and cleanup

One harness changes process-global aimock and environment state: OPENAI_BASE_URL, OPENAI_API_KEY, and materialized-model caches. So keep only one harness alive per process. Run harness suites sequentially, or isolate them in subprocesses. Always await h.close() in afterAll or finally. Cleanup stops aimock, releases sandboxes, restores the prior environment, and clears runtime caches.

Record and live runs can send prompts, tool schemas, and thread content to the configured provider. Treat captured files as reviewable source, and remove secrets and irrelevant turns before committing them.

Update fixtures and troubleshoot#Copy link to section: Update fixtures and troubleshoot

When prompts, tools, or model behavior intentionally change:

1. Re-run the authoring script or one integrated recording locally.
2. Inspect the fixture diff for sensitive data and unexpected calls.
3. Replay the focused test without provider credentials.
4. Commit the fixture and test changes together.

If replay reports no matching fixture, compare the actual user text, cumulative turnIndex, and hasToolResult with the JSON. If a later scenario matches an earlier wildcard, call h.reset() between them. If a following suite points at a stopped aimock port or inherits a temporary key, confirm every harness cleanup is awaited and that harnesses don't overlap.

Related#Copy link to section: Related