Nexus
Sign inStart building
Guides / Brain

BrainBench

The test suite that checks whether memory carries over between sessions and between agents.

BrainBench is a test suite for agent memory. It replays scripted conversations against nexusbrain and checks four things: whether memory speaks up when it should, whether what it volunteers is relevant, whether facts from a conversation get stored correctly, and whether a decision made in one agent tool is remembered in another.

It runs on your machine in an in-memory database, with no API keys and no model calls, so every run of the same code gives the same numbers. Use it to check that a change did not make memory worse, or to grade your own conversations.

Time
10 minutes
For
Developers changing nexusbrain, and anyone who wants to measure memory quality

Before you start

  • nexusbrain is installed.
  • A copy of the nexusbrain source tree. The default test data is read from evals/brainbench/fixtures and evals/brainbench/gold, relative to the folder you run the command in. Otherwise pass --fixtures and --gold.
  • No brain setup is needed. BrainBench never opens your own brain.

01What BrainBench measures

A fixture is a scripted conversation, plus any pages and facts that should already be in memory. Its gold file, kept separately, records what should happen on each turn. Keeping gold separate means the code being tested never sees the answers.

Every fixture is replayed through each harness seam: openclaw, claude-code and codex. A seam is the point where memory is injected into an agent tool. openclaw rows are marked production because they use the shipped integration. claude-code and codex rows are marked contract because they test the same nexusbrain pieces through the shape that integration will use.

SuiteQuestion it answersMain metrics
know-to-askDoes memory surface on its own when it should, and stay quiet when it should not?know_to_ask_failure_rate, false_fire_rate
pushWhen memory volunteers context, is it the right context?push_precision, push_recall
write-backDo facts from a conversation get stored, with correct provenance?write_back_fidelity, provenance_accuracy
continuityIs a decision recorded in one harness recalled in another?continuity_rate
Note

About 15% of fixtures are marked holdout: true. They are left out of normal runs so nobody can tune the code against them, and scored only when you pass --include-holdout.

02Run BrainBench

  1. Open a terminal in the root of the nexusbrain source tree.

  2. Run the full suite.

    nexusbrain eval brainbench

    You should seeA markdown scoreboard starting # BrainBench scoreboard, a line with the fixtures hash, holdout excluded (gate mode) and deterministic (hermetic), then one table row per harness and suite. The exit code is 0.

  3. Narrow the run while you work on one area.

    --harness and --suite take a comma-separated list or all.

    nexusbrain eval brainbench --harness openclaw --suite know-to-ask,push

    You should seeThe same scoreboard with only the rows you asked for.

  4. Save a machine-readable result.

    --out writes the full JSON result to a file. --json prints JSON on stdout instead of the scoreboard.

    nexusbrain eval brainbench --json --out results/bb.json

    You should see[eval brainbench] result written: results/bb.json on stderr, and a JSON object with receipt, cells, turn_rows, seed_failures and _meta.metric_glossary.

FlagWhat it does
--fixtures DIRFixture folder. Default evals/brainbench/fixtures.
--gold DIRGold folder. Default evals/brainbench/gold.
--harness a,b | allHarness seams to grade. Default all.
--suite a,b | allSuites to run. Default all.
--include-holdoutScore holdout fixtures too.
--jsonPrint the JSON result instead of the scoreboard.
--out FILEWrite the full JSON result to a file.
--compare BASE [CURRENT]Gate against a saved baseline. With two files, compares them without running.
--update-baseline [FILE]Save this run as the baseline. Default evals/brainbench/baselines/main.json.
--justification "reason"Recorded in the baseline you write. Required when you accept a regression.
--allow-regression "reason"Let a single run pass despite a regression. The reason is recorded.
--llmUse the real model extractor in the write-back suite. Needs a configured chat model.
--budget-usd NSpend cap for --llm. Default 5.
--seed NRecorded in the receipt. Default 42. The run is deterministic, so nothing uses it yet.

03Read the scoreboard

Each row is one harness and one suite. failed/gold shows how many gold items failed out of how many were checked. A metric that does not apply to that suite shows a dash. Rates run from 0 to 1.

MetricMeaningGood direction
know_to_ask_failure_rateOf the turns where memory should have surfaced something unprompted, the share where nothing relevant was injected.Lower
false_fire_rateOf the turns where memory should have stayed silent, the share where it injected anyway.Lower
push_precisionOf everything memory volunteered, the share that was relevant. Counted per injected pointer.Higher
push_recallOf everything that should have been volunteered, the share that was. Pointer limits cap this on purpose.Higher
write_back_fidelityOf the facts stated in a conversation, the share that survived the real storage pipeline and can be found with the right entity.Higher
provenance_accuracyOf the stored facts, the share with the correct source tag, session id and origin page.Higher
continuity_rateOf decisions recorded in one session, the share recalled when a different harness asks later.Higher
extraction_recallWith --llm only: the share of gold facts the real extractor found and stored.Higher
extraction_precisionWith --llm only: the share of extracted facts that match a gold fact.Higher
source_isolation_violationsCount of injected pointers that belong to a source other than the active one. This is a data leak.Must be 0
avg_injected_tokensEstimated tokens of volunteered context per turn (characters divided by 4). Reported, not gated.Judgment call
Warning

Any source_isolation_violations above 0 fails the gate, whatever the baseline says.

04Catch regressions with a baseline

A baseline is a saved copy of the metrics from a run. When you compare against it with the same fixtures, any gold item that newly fails, any gated metric that moves the wrong way, or any source isolation violation counts as a regression.

  1. Compare your current code against the committed baseline.

    nexusbrain eval brainbench --compare evals/brainbench/baselines/main.json

    You should seeThe scoreboard followed by ## Gate: PASS (same-hash), or ## Gate: REGRESSION with a table of each breach: cell, metric, baseline value, current value and detail.

  2. Check the exit code.

    0 means pass, 1 means regression, 2 means an error, an inconclusive result or a usage mistake.

  3. If the change in numbers is intended, save a new baseline with a reason.

    nexusbrain eval brainbench --update-baseline --justification "push recall traded for precision"

    You should see[eval brainbench] baseline written: evals/brainbench/baselines/main.json. Commit the file so reviewers see the change.

  • If the fixtures changed, the gate switches to corpus-bless mode: your committed baseline must exactly match this run, and any metric that got worse needs a justification.
  • A run with seed failures never writes a baseline.
  • To diff two saved result files without running anything: nexusbrain eval brainbench --compare base.json current.json.

05Use your own fixtures and gold

  1. Write fixtures and gold files that follow the published schemas.

    Fixture files end in .fixture.json and gold files end in .gold.json. They are matched by fixture_id, which must be kebab-case. The schemas are in evals/brainbench/schema/.

    A fixture needs schema_version: 1, fixture_id, suites and turns. Each turn needs turn_id, role (user or assistant) and text. Write-back fixtures and continuity writers need a ts timestamp on every turn. A continuity pair has one writer and one reader sharing a pair_id.

    Never put a gold key inside a fixture turn. That is a validation error.

  2. Run BrainBench on your folders.

    nexusbrain eval brainbench --fixtures my-bench/fixtures --gold my-bench/gold --json --out result.json

    You should see[eval brainbench] result written: result.json. The result follows evals/brainbench/schema/result.schema.json.

Tip

Use invented names, dates within 2024 to 2026, and placeholder amounts like $X. Do not base a scenario on a real situation with the names swapped.

06Troubleshooting

eval brainbench: run failed: brainbench: cannot read fixtures dir evals/brainbench/fixtures.

You are not in the root of the nexusbrain source tree. Change to that folder, or pass --fixtures DIR and --gold DIR.

eval brainbench: no fixtures found in <dir>.

The folder has no files ending in .fixture.json. Check the path.

the harness/suite filters matched zero fixtures.

No fixture belongs to the suites you picked. Widen --suite or --harness, or check the suites field in your fixtures.

eval brainbench: fixture validation: ....

A fixture or gold file breaks the schema. The message names the file and the problem, such as fixture_id must be a kebab-case string or unknown suite.

SEED FAILURES and exit code 2.

Some fixtures could not be loaded into the test database, so the whole run is invalid. Fix the fixture ids listed under the message and run again.

--llm requires a configured chat model.

Configure a chat model (for example with nexusbrain models), or drop --llm to use the default deterministic mode.