ckb/tests/evals
Michał Kopeć c57ddceffd Template 2.2.0: voice and convert modules
- 2.1.0 new opt-in `voice` module (D-0020): skill ckb-voice rewrites every
  prose file written under apply_to (outputs/answers, documents, reports,
  PROJECT-OVERVIEW) in a voice whose profile is a wiki page,
  wiki/voice/<name>.md (type voice-profile); settings in ckb.yaml (mode,
  profile, writer, apply_to, skip, chat); frozen spans never change, vague
  claims flagged not cut; `writer:` delegates to an external rewriting
  skill such as a personal ghost-writer. ckb-module writes a manifest's
  default_config into the entry's config:. Eval ckb-module/install-voice.
- 2.2.0 new opt-in `convert` module (D-0021): skill ckb-convert and
  convert.py turn Markdown into Word/ODT/PDF/HTML through pandoc with a
  house style in templates/styles/; unit tests and eval
  ckb-module/install-convert.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-11 20:57:39 +01:00
..
ckb-decide Template 2.0.7: evals for reset, module, upgrade, decide supersede, ingest check-in 2026-11-11 17:58:44 +01:00
ckb-feynman/session Template 1.27.0: reference run for ckb-feynman/session (6/6) 2026-11-02 21:43:16 +01:00
ckb-ingest Template 2.0.7: evals for reset, module, upgrade, decide supersede, ingest check-in 2026-11-11 17:58:44 +01:00
ckb-module Template 2.2.0: voice and convert modules 2026-11-11 20:57:39 +01:00
ckb-reset Template 2.0.7: evals for reset, module, upgrade, decide supersede, ingest check-in 2026-11-11 17:58:44 +01:00
ckb-retract/source Template 1.26.0: release 3 batch 3, capture module, skill evals, multi-agent docs 2026-11-02 20:44:05 +01:00
ckb-retrieve Template 1.26.0: release 3 batch 3, capture module, skill evals, multi-agent docs 2026-11-02 20:44:05 +01:00
ckb-upgrade Template 2.0.7: evals for reset, module, upgrade, decide supersede, ingest check-in 2026-11-11 17:58:44 +01:00
README.md Template 2.2.0: voice and convert modules 2026-11-11 20:57:39 +01:00

Skill evals

Structural checks of what an agent run leaves behind: files created or moved, frontmatter values, log entries, pages left alone, lint checks clean. They never check wording, because model output varies between runs.

python3 tests/run_evals.py list
python3 tests/run_evals.py prepare ckb-decide/record /tmp/eval-decide   # scratch KB + the prompt
# open /tmp/eval-decide in the agent, give it the prompt, let it finish
python3 tests/run_evals.py check ckb-decide/record /tmp/eval-decide     # PASS/FAIL per assertion

Each case is tests/evals/<skill>/<case>/case.json: the skill, a fixture under tests/fixtures/, optional setup files, the prompt, and a list of assertions (kinds in run_evals.py). tests/smoke.sh validates every case and checks the runner itself; it does not run an agent, since that costs tokens. Run the agent part by hand before a release, or in CI with a token budget, and say which cases you ran in the release notes.

A case may list replies: what the operator types, in order, each time the agent stops and waits (a check-in, a typed confirmation). Setup can copy template files into the scratch KB (template, optionally from an older git ref) and create a local upstream for ckb-upgrade (upstream); see the docstring of run_evals.py.

A case may ship a reference/ folder: the files a correct run added or changed, plus reference/.deleted listing what it removed, recorded from a real run with run_evals.py record CASE DIR. run_evals.py replay rebuilds that result on a fresh KB and checks it; the smoke test replays every reference, so a change to a case or to the runner that breaks a known-good run fails there.

Case Recorded Reference
ckb-decide/record 2026-11-02 yes
ckb-reset/no-confirm, ckb-reset/confirmed 2026-11-11 confirmed only
ckb-module/install, ckb-module/round-trip 2026-11-11 install only
ckb-module/install-voice not run yet no
ckb-module/install-convert not run yet no
ckb-upgrade/major-guard, ckb-upgrade/take-major 2026-11-11 no (a reference would copy the template's docs)
ckb-decide/supersede 2026-11-11 yes
ckb-ingest/check-in-waits, ckb-ingest/reviewed-page 2026-11-11 reviewed-page only

The cases without a reference end where they started (a run that must stop and ask) or would need a large one; they are run by hand before a release.