How search modes are compared

01What is compared

Each mode is run over the same set of test questions from an eval suite. The suites that depend on the search mode are longmemeval (a public long-term memory benchmark) and replay (your own captured queries replayed against the current brain). brainbench does not depend on the search mode, so it is run once and recorded with mode n/a.

Because every mode answers the same questions, results can be compared question by question. That pairing is what makes the statistics below work.

The published method covers 3 modes and 4 metrics, which makes 12 comparisons.

02Paired bootstrap, in plain terms

Suppose balanced scores 0.81 recall and conservative scores 0.78. Is that a real improvement, or would a different set of questions have flipped it? A bootstrap answers this by pretending to rerun the benchmark many times on new question sets, using only the questions you already have.

Paired means each resample keeps both modes' results for the same question together. Hard questions stay hard for both modes, so the comparison measures the difference between modes, not the difference between easy and hard questions.

1. 1

   Record the difference per question.

   For each question, take mode A's score minus mode B's score.

2. 2

   Resample the questions 10,000 times.

   Each resample draws the same number of questions at random, with replacement, so some questions appear twice and some not at all. Compute the average difference for that resample.

3. 3

   Look at the spread of those 10,000 averages.

   The middle 95% of them is the 95% confidence interval: the range the true difference probably falls in.

   The share of resamples that land on the other side of zero gives the p-value: how likely it is that the gap is just noise.

03Why the Bonferroni correction is applied

Every comparison has a small chance of looking significant by luck. With a 0.05 threshold, that chance is 1 in 20 for each one. Run 12 comparisons and a false win somewhere becomes likely.

The Bonferroni correction fixes this by sharing the 0.05 across all 12 comparisons: a single comparison counts as significant only when its p-value is below 0.05 divided by 12, about 0.0042. Equivalently, multiply each p-value by 12 and compare it with 0.05.

04How to read a result

• Start with the difference and its confidence interval. If the interval includes 0, treat the modes as tied on that metric.
• If the interval excludes 0, check the p-value against the corrected threshold (about 0.0042), not 0.05.
• Look at the size of the difference, not only whether it is significant. A significant gain of 0.005 recall may not be worth the extra cost of tokenmax.
• Weigh quality against cost. The operational metrics avg_tokens, cost_per_query_usd and p99_latency_ms show what each mode charges for its results.
• Only compare runs on the same data and commit. A result is tied to the questions it was measured on.

05Record and view a comparison yourself

Comparison results live in .nexusbrain-evals/eval-results.jsonl at the root of your git repo (or the current folder when you are not in a repo). Your personal brain is never touched.

1. 1

   Run the sweep to create the run records.

   It first prints a cost estimate and the budget caps (default $5 for retrieval and $20 for answer generation). A sweep over the cap is refused unless you pass --yes, and, when not in a terminal, an explicit --budget-usd-retrieval or --budget-usd-answer.

   nexusbrain eval run-all --modes conservative,balanced,tokenmax --suites longmemeval,replay --seed 42

   Copy

   You should see[eval run-all] cost estimate: and budget caps: lines, then one line per suite and mode such as [eval run-all] <commit>-longmemeval-balanced-42: skipped, then complete. Audit trail: and the results file path.

2. 2

   Run each suite once per mode.

   In this version run-all records longmemeval and replay as skipped placeholders and does not run them for you. Run the suite commands yourself with --mode.

   nexusbrain eval longmemeval longmemeval.jsonl --mode conservative --output lme-conservative.jsonl
   nexusbrain eval longmemeval longmemeval.jsonl --mode balanced --output lme-balanced.jsonl
   nexusbrain eval longmemeval longmemeval.jsonl --mode tokenmax --output lme-tokenmax.jsonl

   Copy

   You should seeOne output file per mode. For replay, use nexusbrain eval replay --against captured.ndjson --mode <mode>.

3. 3

   Add a completed line per mode to the results file.

   eval compare reads every line that has run_id, suite and mode, and shows status, ran_at and the numbers in a metrics object. When the same suite and mode appear more than once, the most recent ran_at wins.

   {"run_id":"abc1234-longmemeval-balanced-42","ran_at":"2026-09-17T10:00:00Z","suite":"longmemeval","mode":"balanced","commit":"abc1234","seed":42,"status":"completed","metrics":{"recall@10":0.81}}

   Copy
4. 4

   View the comparison.

   nexusbrain eval compare

   Copy

   You should seeA markdown report starting # Search Mode Comparison, with a section per suite, a table of mode, status, run id and time, then each metric listed per mode to 4 decimals with a Plain English: explanation.

06Troubleshooting