Scenario Testing

Scenario tests check what a route returns for a given input. They live next to each route as run.test.ts files. Each file default-exports a route-scoped suite built with scenarios("/route"), and b4 test runs every scenario through the route runtime. (Plain default-exported scenario arrays are not supported.)

A minimal test#Copy link to section: A minimal test

A scenario file has no describe() or test() wrapper. Pass the route ID to scenarios(), add cases with .scenario(), then set the input and expected status explicitly:

import { scenarios } from "@b4run/sdk/testing"
 
export default scenarios("/support/[tenant]").scenario("greets a tenant", (s) =>
  s
    .input({ tenant: "acme" })
    .expectPassed()
    .expectOutput({ tenant: "acme", greeting: "Hello, acme!" }),
)

Then run all scenarios:

b4 test

B4.run discovers every run.test.ts under src/app/, checks that the declared route matches the file's directory, invokes the route, and evaluates the expectations. b4 typegen writes the route and application-tool types the builder uses, so you'll get IntelliSense for route names, .mockTool(), and .expectTool().

Builder API#Copy link to section: Builder API

Every .scenario(name, configure) callback must return a builder that has called .input() exactly once and selected either .expectPassed() or .expectFailed(). The remaining methods add expectations or choose where the scenario runs:

• .input(value) sets the route input.
• .expectPassed() and .expectFailed() select the required result status.
• .expectOutput(value) matches a passed route's returned state.
• .expectError(value) matches a failed route's modeled error.
• .expectMeta(value) matches { mode, routeId, routePath, executionSource }.
• .assert(callback) runs a synchronous or asynchronous custom assertion after declarative expectations.
• .server(url) sends the scenario through a running B4.run-compatible server instead of invoking in-process.
• .mockTool(name, implementation) replaces one application tool for this in-process invocation.
• .expectTool(name, configure) asserts calls made to a tool mocked earlier in the same scenario.

Passing scenarios can use .expectOutput(). Failing scenarios can use .expectError(). The builder's types hide combinations that can't succeed, and the loader checks the same rules at runtime.

For programmatic assertions, import the existing helpers from @b4run/sdk/testing:

import { expectMeta, expectOutput, scenarios } from "@b4run/sdk/testing"
 
export default scenarios("/support/[tenant]").scenario("custom assert", (s) =>
  s
    .input({ tenant: "acme" })
    .expectPassed()
    .assert((result) => {
      expectOutput(result, { greeting: "Hello, acme!" })
      expectMeta(result, { mode: "agent", routeId: "/support/[tenant]" })
    }),
)

expectOutput, expectMeta, and expectError use the same result model as declarative expectations and produce focused mismatch messages.

Mocking application tools#Copy link to section: Mocking application tools

Use .mockTool() when an in-process scenario should replace an external or nondeterministic application tool while keeping the rest of the route real. Tool names, parameters, and awaited return values come from the generated route types:

import { scenarios } from "@b4run/sdk/testing"
 
export default scenarios("/navlog").scenario("uses a controlled search result", (s) =>
  s
    .input({ messages: [{ role: "user", content: "Research B4.run" }] })
    .mockTool("searchWeb", async ({ query }) => ({
      results: [{ title: "B4.run", url: "https://example.test/b4", query }],
    }))
    .expectPassed()
    .expectTool("searchWeb", (call) =>
      call.calledOnce().withArgs({ query: "B4.run" }),
    ),
)

Mocks are partial. Application tools you don't name in .mockTool() keep their real implementations. B4.run resolves shared and route-local tool precedence first. Then it swaps only the selected definition's implementation and keeps its schema, description, scope, and source metadata.

.expectTool() is available only for tools mocked earlier in the same scenario. Its call builder supports:

• .called() for one or more calls.
• .calledOnce() for exactly one call.
• .calledTimes(n) for exactly n calls.
• .notCalled() for zero calls.
• .withArgs(partial) for at least one call containing the supplied deep-partial object. Primitive values and arrays match exactly.

Count and argument assertions are checked separately when combined. Multiple .withArgs() assertions each need a matching invocation, and one compatible invocation may satisfy more than one matcher. This API has no assertions for call order or return values.

Mocks apply only to the root route's generated application-tool set: route-local tools and shared tools. They can't shadow built-in planning, workspace, memory, skill, or subagent capability tools. A parent route's mocks never reach a child subagent route either.

Each scenario invocation gets a fresh override set and call journal. B4.run never mutates cached route modules, so mocks can't leak into another scenario, a later b4 run, a server request, or a concurrent invocation.

Against a live dev server#Copy link to section: Against a live dev server

Use .server(url) to exercise a scenario through a running b4 dev, built B4.run server, or staging deployment. You set the URL per scenario, because b4 test has no --url flag:

import { scenarios } from "@b4run/sdk/testing"
 
export default scenarios("/support/[tenant]").scenario(
  "greets a tenant via dev server",
  (s) =>
    s
      .input({ tenant: "acme" })
      .server("http://127.0.0.1:3001")
      .expectPassed()
      .expectOutput({ tenant: "acme", greeting: "Hello, acme!" }),
)

Start the dev server first, then run b4 test:

b4 dev --port 3001 &
b4 test

When to use a server-backed scenario#Copy link to section: When to use a server-backed scenario

Use a server-backed scenario when the claim depends on:

• JSON request and response serialization
• request middleware and request-scoped context
• runtime boot and route lookup wiring
• packaged static modules in a built runtime
• real staging infrastructure and deployment configuration

Server-backed scenarios execute in another process and exchange JSON. JavaScript mock functions can't cross a process, so .server() and .mockTool() are mutually exclusive in the builder and in runtime validation. The server has no test backdoor for mocks.

So control server dependencies from outside. Start a local fake HTTP service, point the server at a model proxy, or use a dedicated staging dependency before running b4 test.

Agents, retries, and middleware#Copy link to section: Agents, retries, and middleware

Two cross-cutting features shape what scenarios assert:

• Agent retries: agent({ retry: { maxAttempts, baseDelay } }) retries on transient errors. To assert the exhausted-retry path, call .expectFailed().expectError(...) or use .assert() with expectError. See Retry.
• Middleware: src/middleware.ts runs before local Agent Protocol run and stream requests. Server-backed scenarios targeting a B4.run runtime exercise middleware. In-process scenarios bypass it. A scenario whose middleware calls reject(...) should use .expectFailed().expectError(...). See Middleware. The in-process agent harness also bypasses middleware. createAgentHarness({ middlewareContext }) supplies the context its tools would otherwise read from it.

Rules#Copy link to section: Rules

1. 1

   File location

   run.test.ts must live in the route's own directory. B4.run matches tests to routes by directory, so a sibling or nested folder won't match.

2. 2

   Route-scoped suite

   Default-export scenarios("/route").scenario(...). The declared route must match the file location, and plain scenario arrays are rejected.

3. 3

   Complete every scenario

   Call .input() once and select .expectPassed() or .expectFailed(). Add declarative expectations or .assert() for the behavior the scenario owns.

4. 4

   Use typed application-tool mocks narrowly

   Mock only the external or nondeterministic application tools needed for the claim. Leave deterministic tools real, and use a server-backed scenario when the transport itself is under test.

5. 5

   Keep scenarios focused

   One scenario makes one claim about the route's behavior. If you're adding branches, add more scenarios.

CI#Copy link to section: CI

Run b4 verify first (it covers app, routes, typegen, and deps in one call), then b4 test:

- run: npx b4 verify
- run: npx b4 test

b4 verify runs typegen and check, plus the deps check (missing packages, missing env vars) that bare b4 check && b4 typegen skips. Typegen also refreshes .b4/scenarios.generated.d.ts, which powers route and application-tool completion in run.test.ts. b4 test exits non-zero on any failure and outputs a diff per mismatched scenario.

Unit-testing tools and middleware#Copy link to section: Unit-testing tools and middleware

Sometimes you want to test one unit (a single route tool, a FilesystemMiddleware, or code that uses ctx.fs) without standing up an agent. @b4run/testing ships three harnesses for this. They run against the real WorkspaceFs and filesystem backend over a temp directory, so you get real permission gating, realpath resolution and parent-directory creation.

All three are async create*Harness factories that return a handle with .close() and [Symbol.asyncDispose], the same convention as createAgentHarness and every other handle in @b4run/testing.

A tool that uses ctx.fs#Copy link to section: A tool that uses ctx.fs

createToolHarness(tool) builds the B4ToolContext and gives you a reusable invoke(). Assert both the return value and what landed on disk via h.workspace.read(...):

import { afterEach } from "vitest"
import { createToolHarness } from "@b4run/testing"
import type { B4ToolContext } from "@b4run/sdk"
 
const saveNote = async (input: { name: string; body: string }, ctx: B4ToolContext) => {
  const { bytesWritten } = await ctx.fs.writeFile(`notes/${input.name}.md`, input.body)
  return { bytesWritten }
}
 
let h: Awaited<ReturnType<typeof createToolHarness>>
afterEach(() => h.close())
 
test("saveNote writes into the workspace", async () => {
  h = await createToolHarness(saveNote)
 
  const result = await h.invoke({ name: "todo", body: "ship it" })
 
  expect(result.bytesWritten).toBe(7)
  expect(await h.workspace.read("notes/todo.md")).toBe("ship it")
})

invoke() is reusable and shares one workspace across calls, so you can assert cumulative state across several invocations. Pass { workspace } to share a fixture you already own (the harness won't close it), or { permissions } to exercise allow/deny gating instead of the permissive default.

Testing ctx.fs code directly#Copy link to section: Testing ctx.fs code directly

createWorkspaceHarness() is the shared fixture the tool harness builds on. Use it directly when the code under test takes a WorkspaceFs. Seed the workspace with write, run your code against h.fs, and assert with read:

import { createWorkspaceHarness } from "@b4run/testing"
import type { WorkspaceFs } from "@b4run/sdk"
 
const appendLine = async (fs: WorkspaceFs, path: string, line: string) => {
  const prev = await fs.readFile(path)
  await fs.writeFile(path, `${prev}\n${line}`)
}
 
test("appendLine round-trips through the real WorkspaceFs", async () => {
  await using h = await createWorkspaceHarness()
  await h.write("log.txt", "first")
 
  await appendLine(h.fs, "log.txt", "second")
 
  expect(await h.read("log.txt")).toBe("first\nsecond")
})

The await using form disposes the harness (and cleans up the temp dir) at the end of the block, in place of afterEach(() => h.close()). It works on all three harnesses.

A filesystem middleware#Copy link to section: A filesystem middleware

createMiddlewareHarness(mw) composes a FilesystemMiddleware over a temp localFilesystem and exposes the wrapped backend plus a ctx to call it with. assertForwardsAll() catches the most common middleware bug: silently dropping a backend method (required or optional) that the middleware doesn't intercept:

import { createMiddlewareHarness } from "@b4run/testing"
import type { FilesystemMiddleware } from "@b4run/workspace"
 
const uppercaseReads: FilesystemMiddleware = (next) => ({
  ...next,
  readFile: async (path, ctx) => (await next.readFile(path, ctx)).toUpperCase(),
})
 
test("uppercaseReads forwards every other backend method", async () => {
  await using h = await createMiddlewareHarness(uppercaseReads)
 
  await h.backend.writeFile("a.txt", "hello", h.ctx)
  expect(await h.backend.readFile("a.txt", h.ctx)).toBe("HELLO")
 
  // Fails loudly if the middleware forgot to spread a method like realPath.
  h.assertForwardsAll()
})

Related#Copy link to section: Related