ckb/docs/decisions/0017-capture-module-and-skill-evals.md
Michał Kopeć dad2264a65 Template 1.26.0: release 3 batch 3, capture module, skill evals, multi-agent docs
- F11 new opt-in module `capture` (skill ckb-capture, capture_url.py):
  stdlib fetch with readable-text extraction, Wayback only over a failed
  or thinner live page, --from-file for the agent's own fetch, PDFs saved
  as files, --media via yt-dlp with consent, untrusted: true marking,
  dedup at capture time; dedup_inbox.py now compares source_url (D-0017)
- F22 tests/run_evals.py (list/prepare/check/validate) and five
  structural eval cases; reference run for ckb-decide/record
- F18 manual section with untested setups for Codex CLI, Cursor,
  OpenCode and Gemini CLI, plus the Claude Code-only feature table;
  AGENTS.md tells agents without skill loading to read SKILL.md

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 20:44:05 +01:00

66 lines
3.3 KiB
Markdown

---
type: decision
tldr: URL and media capture ships as an opt-in `capture` module whose stdlib script writes untrusted-marked snapshots to raw/inbox/ (Wayback only over a failed or thinner live page, agent web tool as fallback, yt-dlp only with consent, dedup at capture); skill behaviour is checked by structural evals with a prepare/check runner and recorded reference runs; multi-agent setup is documented but marked untested.
status: accepted
decided_on: 2026-11-02
decided_by: Michał Kopeć
affects: .agents/modules/capture/, .agents/skills/ckb-ingest/scripts/dedup_inbox.py, tests/run_evals.py, tests/evals/, AGENTS.md, docs/MANUAL.md
confidence: 0.7
quality: 0.8
last_updated: 2026-11-02
freshness_window_days: 365
retention: high
---
# D-0017 — A capture module, skill evals, and untested multi-agent docs
**Status:** Accepted · **Decided:** 2026-11-02 · **Deciders:** Michał Kopeć
## Context
Release 3, batch 3: F11 (URL and media capture), F22 (skill evals) and F18
(multi-agent setup). Capture fetches content from the open web, so it brings
untrusted input and a network dependency. Not every KB wants either. Skills
are prose, and until now the smoke test covered only the scripts.
## Decision
1. **Capture is a module** (D-0001 pattern), not a base skill, so KBs that
never clip the web pay no context for it.
2. **A stdlib script does the fetch** (the user's choice): `urllib`, an
HTML-to-Markdown extractor that keeps headings, lists and code, and the
Wayback Machine's closest snapshot. An archived copy replaces the live
page only when the live fetch failed or yielded less text. When the
script can't fetch, the agent's own web tool fetches and `--from-file`
wraps the text the same way.
3. **Every snapshot is marked untrusted** (`untrusted: true` and a banner)
and records `source_url`, `fetched_on` and `fetched_via`.
4. **Dedup at capture time:** the script runs `dedup_inbox.py`, which now also
compares a capture's `source_url` with archived captures and other inbox
files, and removes its own duplicate.
5. **Media only with consent**, through `yt-dlp` when installed, never
installed by the agent.
6. **Evals are structural.** `tests/run_evals.py` prepares a scratch KB from
a fixture, and a person or a CI job runs the agent with the case's prompt.
`check` then scores files, frontmatter, logs, unchanged pages and lint
checks. The first cases cover ingest, retrieve (two), decide and retract.
A case may keep a `reference/` run, which the smoke test uses to check the
runner both ways.
7. **Multi-agent docs are marked untested** until someone confirms them. One
line in `AGENTS.md` tells agents without skill loading to read
`SKILL.md`. Per-folder rules stay in `## Rules`, which every agent reads.
## Alternatives considered
- **Capture as a base skill** — context cost for every KB.
- **Agent web tool only** — less reproducible; rejected by the user.
- **Evals that grade answer text with a model** — costly and noisy; maybe
later as an optional layer on top of the structural checks.
## Consequences
- The live Wayback path could not be exercised while archive.org was
rate-limiting this machine. The smoke test covers it against a local
stand-in (`CKB_WAYBACK_API`).
- Evals need an agent run to mean anything; the release notes should say
which cases were run.