Release · v0.4.0

B4.run 0.4: Start with an agent test

Learn how the offline agent test added to the starter in 0.4 works, and what a scripted model response can verify.

June 2, 2026

Brian Love

releases

Testing an agent can feel strange at first. We're used to calling a function with an input and asserting its output. A model can give different answers to the same question, and calling it in every test adds network calls and cost.

The 0.4 release added an offline agent test to the starter application. A new project included the test harness, Vitest and a fixture describing the model's response. Let's look at why that's useful and how the test works.

Version note: The code below uses the current @b4run package names and basic starter, which arrived after 0.4. Use the current starter to follow along.

Goals#Copy link to section: Goals

We want to:

1. Run an agent route without calling a model provider.
2. Supply a predictable model response.
3. Assert that the application returns the expected final message.
4. Understand which behavior this test verifies and which behavior needs a live model.

What are we testing?#Copy link to section: What are we testing?

An agent run includes more than the model request. The application loads a route, builds its instructions, makes tools available, processes model responses and maintains state.

A fixture supplies a known response in place of the live model, and the rest of the application runs with that response. If you have mocked an HTTP API in a TypeScript test, this should sound familiar. We control the external dependency so that a failed test points to a repeatable scenario.

B4.run's agent harness intercepts the model's HTTP endpoint with a local mock. Tools, prompts, capabilities and state still run. Only the model response is replaced.

Create a project#Copy link to section: Create a project

Use Node.js 24 or later and select the basic template:

pnpm create b4-app my-agents --template basic
cd my-agents
pnpm install
pnpm test

The basic template includes the /hello route and test/agent.test.ts. Its fixture-backed test runs without a provider API key. You'll only need provider configuration when you run the agent against a real model.

Read the test#Copy link to section: Read the test

Here is the test included in the current basic starter:

import { fileURLToPath } from "node:url"
import { afterAll, it } from "vitest"
import { createAgentHarness, expectFinalMessage, script } from "@b4run/testing"
 
const appRoot = fileURLToPath(new URL("..", import.meta.url))
const h = await createAgentHarness({ appRoot, route: "/hello#agent" })
afterAll(() => h.close())
 
it("greets by name", async () => {
  const run = await h.run({
    input: "Say hello to Ada",
    fixtures: script().user("Say hello to Ada").replies("Hello, Ada!"),
  })
  expectFinalMessage(run).toContain("Hello")
}, 60_000)

Let's break this down:

1. Locate the application#Copy link to section: 1. Locate the application

fileURLToPath(new URL("..", import.meta.url)) resolves the application root relative to the test file. The harness uses that directory to find the application's configuration and routes, so the test doesn't depend on a hard-coded path on one developer's computer.

2. Create the harness#Copy link to section: 2. Create the harness

createAgentHarness() prepares the application for a test run. The route value selects /hello, and #agent identifies the agent entry.

The harness starts the local model mock, runs type generation, and resolves the route. We register h.close() with Vitest's afterAll() hook so the test releases those resources when it finishes.

3. Describe the model response#Copy link to section: 3. Describe the model response

The script() builder describes the exchange:

script().user("Say hello to Ada").replies("Hello, Ada!")

The user message selects the fixture, and replies() supplies the model's response. When the route makes its model request, the local mock returns this text without contacting the real provider.

4. Assert the result#Copy link to section: 4. Assert the result

Finally, expectFinalMessage(run).toContain("Hello") checks the final message returned by the run.

The 60_000 argument is Vitest's timeout in milliseconds. It gives the scenario up to a minute to finish.

Test a tool call#Copy link to section: Test a tool call

As the application grows, we can script a tool request as well. For a route with an applyFilter tool, a fixture could look like this:

script()
  .user("Filter open items")
  .callsTool("applyFilter", { status: "open" })
  .replies("Found 2 open items.")

First, the mock returns a request to call applyFilter. The runtime executes the actual tool. On the next model turn, the mock supplies the final message.

This tests how the application handles a known tool call. We can assert the tool name and arguments with expectToolCalled, and test the tool's own output separately. The Agent Test Harness guide includes a complete example.

What does a passing test tell us?#Copy link to section: What does a passing test tell us?

The greeting test tells us the application can run the selected route with the supplied fixture and return the expected message. A tool scenario checks the wiring and behavior around a scripted call.

It can't tell us whether a live model will choose the correct tool, because we supplied that choice in the fixture. For the same reason, it can't show that a changed prompt improves the model's answers.

Use fixture-backed tests for repeatable application behavior. Use live runs and representative evaluation datasets to investigate model behavior.

Conclusion#Copy link to section: Conclusion

In my opinion, the best part of this release was giving a new application a test from day one. We can run the route, control the model response and make an assertion before adding our own features.

Start with that small test. As you add tools and state, add scenarios for the behavior you expect and the failures you need to handle. The Testing agents guide and Evals guide explain how to extend that coverage.