|
|
||
|---|---|---|
| .. | ||
| ckb-decide | ||
| ckb-feynman/session | ||
| ckb-ingest | ||
| ckb-module | ||
| ckb-reset | ||
| ckb-retract/source | ||
| ckb-retrieve | ||
| ckb-upgrade | ||
| README.md | ||
Skill evals
Structural checks of what an agent run leaves behind: files created or moved, frontmatter values, log entries, pages left alone, lint checks clean. They never check wording, because model output varies between runs.
python3 tests/run_evals.py list
python3 tests/run_evals.py prepare ckb-decide/record /tmp/eval-decide # scratch KB + the prompt
# open /tmp/eval-decide in the agent, give it the prompt, let it finish
python3 tests/run_evals.py check ckb-decide/record /tmp/eval-decide # PASS/FAIL per assertion
Each case is tests/evals/<skill>/<case>/case.json: the skill, a fixture
under tests/fixtures/, optional setup files, the prompt, and a list of
assertions (kinds in run_evals.py). tests/smoke.sh validates every case
and checks the runner itself; it does not run an agent, since that costs
tokens. Run the agent part by hand before a release, or in CI with a token
budget, and say which cases you ran in the release notes.
A case may list replies: what the operator types, in order, each time the
agent stops and waits (a check-in, a typed confirmation). Setup can copy
template files into the scratch KB (template, optionally from an older git
ref) and create a local upstream for ckb-upgrade (upstream); see the
docstring of run_evals.py.
A case may ship a reference/ folder: the files a correct run added or
changed, plus reference/.deleted listing what it removed, recorded from a
real run with run_evals.py record CASE DIR. run_evals.py replay rebuilds
that result on a fresh KB and checks it; the smoke test replays every
reference, so a change to a case or to the runner that breaks a known-good
run fails there.
| Case | Recorded | Reference |
|---|---|---|
ckb-decide/record |
2026-11-02 | yes |
ckb-reset/no-confirm, ckb-reset/confirmed |
2026-11-11 | confirmed only |
ckb-module/install, ckb-module/round-trip |
2026-11-11 | install only |
ckb-upgrade/major-guard, ckb-upgrade/take-major |
2026-11-11 | no (a reference would copy the template's docs) |
ckb-decide/supersede |
2026-11-11 | yes |
ckb-ingest/check-in-waits, ckb-ingest/reviewed-page |
2026-11-11 | reviewed-page only |
The cases without a reference end where they started (a run that must stop and ask) or would need a large one; they are run by hand before a release.