- 2.1.0 new opt-in `voice` module (D-0020): skill ckb-voice rewrites every prose file written under apply_to (outputs/answers, documents, reports, PROJECT-OVERVIEW) in a voice whose profile is a wiki page, wiki/voice/<name>.md (type voice-profile); settings in ckb.yaml (mode, profile, writer, apply_to, skip, chat); frozen spans never change, vague claims flagged not cut; `writer:` delegates to an external rewriting skill such as a personal ghost-writer. ckb-module writes a manifest's default_config into the entry's config:. Eval ckb-module/install-voice. - 2.2.0 new opt-in `convert` module (D-0021): skill ckb-convert and convert.py turn Markdown into Word/ODT/PDF/HTML through pandoc with a house style in templates/styles/; unit tests and eval ckb-module/install-convert. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> |
||
|---|---|---|
| .. | ||
| ckb-decide | ||
| ckb-feynman/session | ||
| ckb-ingest | ||
| ckb-module | ||
| ckb-reset | ||
| ckb-retract/source | ||
| ckb-retrieve | ||
| ckb-upgrade | ||
| README.md | ||
Skill evals
Structural checks of what an agent run leaves behind: files created or moved, frontmatter values, log entries, pages left alone, lint checks clean. They never check wording, because model output varies between runs.
python3 tests/run_evals.py list
python3 tests/run_evals.py prepare ckb-decide/record /tmp/eval-decide # scratch KB + the prompt
# open /tmp/eval-decide in the agent, give it the prompt, let it finish
python3 tests/run_evals.py check ckb-decide/record /tmp/eval-decide # PASS/FAIL per assertion
Each case is tests/evals/<skill>/<case>/case.json: the skill, a fixture
under tests/fixtures/, optional setup files, the prompt, and a list of
assertions (kinds in run_evals.py). tests/smoke.sh validates every case
and checks the runner itself; it does not run an agent, since that costs
tokens. Run the agent part by hand before a release, or in CI with a token
budget, and say which cases you ran in the release notes.
A case may list replies: what the operator types, in order, each time the
agent stops and waits (a check-in, a typed confirmation). Setup can copy
template files into the scratch KB (template, optionally from an older git
ref) and create a local upstream for ckb-upgrade (upstream); see the
docstring of run_evals.py.
A case may ship a reference/ folder: the files a correct run added or
changed, plus reference/.deleted listing what it removed, recorded from a
real run with run_evals.py record CASE DIR. run_evals.py replay rebuilds
that result on a fresh KB and checks it; the smoke test replays every
reference, so a change to a case or to the runner that breaks a known-good
run fails there.
| Case | Recorded | Reference |
|---|---|---|
ckb-decide/record |
2026-11-02 | yes |
ckb-reset/no-confirm, ckb-reset/confirmed |
2026-11-11 | confirmed only |
ckb-module/install, ckb-module/round-trip |
2026-11-11 | install only |
ckb-module/install-voice |
not run yet | no |
ckb-module/install-convert |
not run yet | no |
ckb-upgrade/major-guard, ckb-upgrade/take-major |
2026-11-11 | no (a reference would copy the template's docs) |
ckb-decide/supersede |
2026-11-11 | yes |
ckb-ingest/check-in-waits, ckb-ingest/reviewed-page |
2026-11-11 | reviewed-page only |
The cases without a reference end where they started (a run that must stop and ask) or would need a large one; they are run by hand before a release.