Improving a skill automatically

01How the optimization loop works

Think of SKILL.md as the thing being trained, and three models as the trainers. The target model runs the skill on each task. The judge model scores each output. The optimizer model reads the successes and failures and proposes edits.

Your test tasks are split, sorted by task_id, into three groups using --split TRAIN:SEL:TEST (default 4:1:5). Training tasks feed the optimizer. Selection tasks decide whether an edit is kept. Test tasks are only scored at the end, to show whether the improvement holds up.

Each step runs a batch of training tasks, asks the optimizer (two reflect calls) for edits, applies up to the current edit budget, then scores the edited skill on every selection task three times and takes the median. The edit is accepted only if the average beats the best score so far by more than 0.05. Rejected edits are remembered so they are not proposed again.

The edit budget starts at --lr (default 4 edits per step) and shrinks over the run following --lr-schedule (default cosine). There are --epochs passes (default 4), each with training tasks divided by --batch-size (default 8) steps.

02Build the test set

The benchmark is a JSONL file: one JSON object per line with a unique task_id, a task prompt, and a judge. By default it is read from skills/<skill>/skillopt-benchmark.jsonl; pass --benchmark <path> to use another file.

1. 1

   Generate a starter benchmark from the skill itself.

   This reads SKILL.md, asks the optimizer model for about 15 tasks with rule checks, and writes them to skills/<skill>/skillopt-benchmark.jsonl. Use --bootstrap-tasks N for a different count, up to 50.

   nexusbrain skillopt meeting-prep --bootstrap-from-skill

   Copy

   You should seeThe file is created and its last line is # BOOTSTRAP_PENDING_REVIEW. If a benchmark already exists you get Benchmark already exists at <path>.; pass --force to overwrite.

2. 2

   Review and strengthen every task.

   Read each task and its checks. Remove vague checks, tighten limits, and make sure headings and tool names are real. The optimizer only chases what the judges reward, so weak judges produce a weak skill.

3. 3

   Delete the last line.

   Remove the # BOOTSTRAP_PENDING_REVIEW line. Until you do, the run refuses the file.

4. 4

   Pick a split that leaves at least 5 selection tasks.

   With 15 tasks the default 4:1:5 gives only 1 selection task, so use --split 1:1:1 (5 train, 5 selection, 5 test) or write more tasks.

• Rule op values: contains, regex, section_present, tool_called, tool_not_called (string arg), and max_chars, min_citations (number arg).
• A full line looks like {"task_id":"t01","task":"Prep me for my 2pm with the design team","judge":{"kind":"rule","checks":[{"op":"section_present","arg":"Agenda"},{"op":"max_chars","arg":4000}]}}.
• If the skill already has a routing-eval.jsonl, --bootstrap-from-routing builds the starter from that instead.

03Preview the cost with a dry run

1. 1

   Run a dry run.

   A dry run loads and validates the benchmark, checks the safety gates, and estimates cost. It makes no model calls and changes no files.

   nexusbrain skillopt meeting-prep --bootstrap-reviewed --split 1:1:1 --dry-run

   Copy

   You should seeA block starting [skillopt] Cost estimate for with rollout, reflect and judge call counts, estimated tokens, and Est. cost: $X (cap: $5.00), then [skillopt] Outcome: aborted. The exit code is 2, which is normal for a dry run.

2. 2

   Adjust the cap or knobs if needed.

   If the estimate is over --max-cost-usd (default 5.00), the run is refused before anything is spent. Raise the cap or lower --epochs or --batch-size.

04Run and review the result

1. 1

   Start the run.

   Add --json for one machine-readable line on stdout that includes the receipt and its run_id.

   nexusbrain skillopt meeting-prep --bootstrap-reviewed --split 1:1:1

   Copy

   You should seeThe cost estimate, then at the end [skillopt] Outcome:, Best sel-score:, Final cost: and either SKILL.md rewritten with N optimization steps. or Proposed improvements written to <path>. Review + copy manually.

2. 2

   Check the exit code.

   0 means an improvement was accepted (or proposed.md was written). 1 means no improvement, so SKILL.md is unchanged. 2 means a gate stopped the run.

3. 3

   Review the change before you commit it.

   For a user-owned skill, SKILL.md is replaced in place and each accepted version is saved under skills/<skill>/skillopt/versions/. Use git diff to read the change.

   To keep SKILL.md untouched and get skills/<skill>/skillopt/proposed.md instead, pass --no-mutate.

   git diff skills/meeting-prep/SKILL.md

   Copy

• --all optimizes every skill that has a benchmark, with --max-cost-usd per skill and --brain-wide-max-cost-usd (default 10.00) for the whole batch.
• --target-models a,b,c runs once per target model in no-mutate mode and reports the best model.
• Every run writes events to ~/.nexusbrain/audit/skillopt-YYYY-Www.jsonl.

05Resume a run that stopped

1. 1

   Find the run id.

   While a run is going, its state is saved as skills/<skill>/skillopt/checkpoint-<run-id>.json. The file is deleted when a run finishes with an improvement or with no improvement, and kept when it is aborted or errors. Stale checkpoints older than 7 days are cleaned up by the dream cycle.

2. 2

   Resume with the same flags as the original run.

   nexusbrain skillopt meeting-prep --bootstrap-reviewed --split 1:1:1 --resume <run-id>

   Copy

   You should seeThe run skips the epochs and steps already completed and continues from the best skill text saved in the checkpoint.

06Improving a skill that ships with nexusbrain

A bundled skill is one that lives in the skills/ folder of your nexusbrain install. By default skillopt never overwrites it: an accepted improvement goes to proposed.md for you to review.

To change a bundled skill in place you need both --allow-mutate-bundled and a held-out file with at least 5 tasks. The held-out file uses exactly the same JSONL format as the benchmark, and none of its task_id values may appear in the benchmark. Before any edit is accepted, the edited skill and the original are both scored on the held-out tasks, and the edit is refused if it scores lower than the original.

1. 1

   Write an independent held-out file.

   Use real tasks you did not put in the benchmark, with at least 5 lines.

2. 2

   Preview the run.

   nexusbrain skillopt brain-ops --allow-mutate-bundled --held-out skills/brain-ops/held-out.jsonl --dry-run

   Copy

   You should seeThe cost estimate now includes held-out scoring, then Outcome: aborted for the dry run.

3. 3

   Run it.

   nexusbrain skillopt brain-ops --allow-mutate-bundled --held-out skills/brain-ops/held-out.jsonl

   Copy

   You should seeSKILL.md rewritten if an edit passed both the selection gate and the held-out gate.

07Troubleshooting