ckb/docs/decisions/0017-capture-module-and-skill-evals.md
Michał Kopeć dad2264a65 Template 1.26.0: release 3 batch 3, capture module, skill evals, multi-agent docs
- F11 new opt-in module `capture` (skill ckb-capture, capture_url.py):
  stdlib fetch with readable-text extraction, Wayback only over a failed
  or thinner live page, --from-file for the agent's own fetch, PDFs saved
  as files, --media via yt-dlp with consent, untrusted: true marking,
  dedup at capture time; dedup_inbox.py now compares source_url (D-0017)
- F22 tests/run_evals.py (list/prepare/check/validate) and five
  structural eval cases; reference run for ckb-decide/record
- F18 manual section with untested setups for Codex CLI, Cursor,
  OpenCode and Gemini CLI, plus the Claude Code-only feature table;
  AGENTS.md tells agents without skill loading to read SKILL.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 20:44:05 +01:00

3.3 KiB

type tldr status decided_on decided_by affects confidence quality last_updated freshness_window_days retention
decision URL and media capture ships as an opt-in `capture` module whose stdlib script writes untrusted-marked snapshots to raw/inbox/ (Wayback only over a failed or thinner live page, agent web tool as fallback, yt-dlp only with consent, dedup at capture); skill behaviour is checked by structural evals with a prepare/check runner and recorded reference runs; multi-agent setup is documented but marked untested. accepted 2026-11-02 Michał Kopeć .agents/modules/capture/, .agents/skills/ckb-ingest/scripts/dedup_inbox.py, tests/run_evals.py, tests/evals/, AGENTS.md, docs/MANUAL.md 0.7 0.8 2026-11-02 365 high

D-0017 — A capture module, skill evals, and untested multi-agent docs

Status: Accepted · Decided: 2026-11-02 · Deciders: Michał Kopeć

Context

Release 3, batch 3: F11 (URL and media capture), F22 (skill evals) and F18 (multi-agent setup). Capture fetches content from the open web, so it brings untrusted input and a network dependency. Not every KB wants either. Skills are prose, and until now the smoke test covered only the scripts.

Decision

  1. Capture is a module (D-0001 pattern), not a base skill, so KBs that never clip the web pay no context for it.
  2. A stdlib script does the fetch (the user's choice): urllib, an HTML-to-Markdown extractor that keeps headings, lists and code, and the Wayback Machine's closest snapshot. An archived copy replaces the live page only when the live fetch failed or yielded less text. When the script can't fetch, the agent's own web tool fetches and --from-file wraps the text the same way.
  3. Every snapshot is marked untrusted (untrusted: true and a banner) and records source_url, fetched_on and fetched_via.
  4. Dedup at capture time: the script runs dedup_inbox.py, which now also compares a capture's source_url with archived captures and other inbox files, and removes its own duplicate.
  5. Media only with consent, through yt-dlp when installed, never installed by the agent.
  6. Evals are structural. tests/run_evals.py prepares a scratch KB from a fixture, and a person or a CI job runs the agent with the case's prompt. check then scores files, frontmatter, logs, unchanged pages and lint checks. The first cases cover ingest, retrieve (two), decide and retract. A case may keep a reference/ run, which the smoke test uses to check the runner both ways.
  7. Multi-agent docs are marked untested until someone confirms them. One line in AGENTS.md tells agents without skill loading to read SKILL.md. Per-folder rules stay in ## Rules, which every agent reads.

Alternatives considered

  • Capture as a base skill — context cost for every KB.
  • Agent web tool only — less reproducible; rejected by the user.
  • Evals that grade answer text with a model — costly and noisy; maybe later as an optional layer on top of the structural checks.

Consequences

  • The live Wayback path could not be exercised while archive.org was rate-limiting this machine. The smoke test covers it against a local stand-in (CKB_WAYBACK_API).
  • Evals need an agent run to mean anything; the release notes should say which cases were run.