ckb/tests/evals
2026-11-11 18:00:20 +01:00
..
ckb-decide Template 1.28.8: backport 2.0.7 evals and reset phrase; ckb-upgrade removes files upstream dropped 2026-11-11 18:00:20 +01:00
ckb-feynman/session Template 1.27.0: reference run for ckb-feynman/session (6/6) 2026-11-02 21:43:16 +01:00
ckb-ingest Template 1.28.8: backport 2.0.7 evals and reset phrase; ckb-upgrade removes files upstream dropped 2026-11-11 18:00:20 +01:00
ckb-module Template 1.28.8: backport 2.0.7 evals and reset phrase; ckb-upgrade removes files upstream dropped 2026-11-11 18:00:20 +01:00
ckb-reset Template 1.28.8: backport 2.0.7 evals and reset phrase; ckb-upgrade removes files upstream dropped 2026-11-11 18:00:20 +01:00
ckb-retract/source Template 1.26.0: release 3 batch 3, capture module, skill evals, multi-agent docs 2026-11-02 20:44:05 +01:00
ckb-retrieve Template 1.26.0: release 3 batch 3, capture module, skill evals, multi-agent docs 2026-11-02 20:44:05 +01:00
ckb-upgrade Template 1.28.8: backport 2.0.7 evals and reset phrase; ckb-upgrade removes files upstream dropped 2026-11-11 18:00:20 +01:00
README.md Template 1.28.8: backport 2.0.7 evals and reset phrase; ckb-upgrade removes files upstream dropped 2026-11-11 18:00:20 +01:00

Skill evals

Structural checks of what an agent run leaves behind: files created or moved, frontmatter values, log entries, pages left alone, lint checks clean. They never check wording, because model output varies between runs.

python3 tests/run_evals.py list
python3 tests/run_evals.py prepare ckb-decide/record /tmp/eval-decide   # scratch KB + the prompt
# open /tmp/eval-decide in the agent, give it the prompt, let it finish
python3 tests/run_evals.py check ckb-decide/record /tmp/eval-decide     # PASS/FAIL per assertion

Each case is tests/evals/<skill>/<case>/case.json: the skill, a fixture under tests/fixtures/, optional setup files, the prompt, and a list of assertions (kinds in run_evals.py). tests/smoke.sh validates every case and checks the runner itself; it does not run an agent, since that costs tokens. Run the agent part by hand before a release, or in CI with a token budget, and say which cases you ran in the release notes.

A case may list replies: what the operator types, in order, each time the agent stops and waits (a check-in, a typed confirmation). Setup can copy template files into the scratch KB (template, optionally from an older git ref) and create a local upstream for ckb-upgrade (upstream); see the docstring of run_evals.py.

A case may ship a reference/ folder: the files a correct run added or changed, plus reference/.deleted listing what it removed, recorded from a real run with run_evals.py record CASE DIR. run_evals.py replay rebuilds that result on a fresh KB and checks it; the smoke test replays every reference, so a change to a case or to the runner that breaks a known-good run fails there.

Case Recorded Reference
ckb-decide/record 2026-11-02 yes
ckb-reset/no-confirm, ckb-reset/confirmed 2026-11-11 confirmed only
ckb-module/install, ckb-module/round-trip 2026-11-11 install only
ckb-upgrade/major-guard, ckb-upgrade/take-major 2026-11-11 no (a reference would copy the template's docs)
ckb-decide/supersede 2026-11-11 yes
ckb-ingest/check-in-waits, ckb-ingest/reviewed-page 2026-11-11 reviewed-page only

The cases without a reference end where they started (a run that must stop and ask) or would need a large one; they are run by hand before a release.