ckb/docs/decisions/0015-section-index-and-access-log.md
Michał Kopeć 1d7a489323 Template 1.24.0: release 3 batch 1, retrieval
- F10 bm25 section search: stdlib SQLite FTS5 over page sections, with
  title/heading path/tldr in a separate context column; cached in
  tmp/ckb-fts.sqlite, refreshed by mtime then sha256; section anchors
  per candidate. New benchmark tests/bench_retrieval.py over
  tests/fixtures/bench-kb (20 questions): hit@1 14/20, hit@3 20/20 with
  and without bm25; weight kept at 1.0 (D-0015)
- F12 token estimate per candidate; layered reading in ckb-retrieve
- F16 --opened access log in tmp/access.log; lint_report.py --cold
  proposes demotion to an index `## Archive` section, never moves
- F17 ckb-retrieve offers a table/chart/deck/HTML once when it fits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 20:25:26 +01:00

3.3 KiB
Raw Blame History

type tldr status decided_on decided_by affects confidence quality last_updated freshness_window_days retention
decision Retrieval gains a stdlib SQLite FTS5 index of page sections (with a title, heading path and tldr context column), cached in tmp/ and rebuilt incrementally, fused as one more ranked list at weight 1.0; candidates carry section anchors and token estimates; pages actually read go to a per-machine tmp/access.log that lint turns into demotion proposals, never moves. accepted 2026-11-02 Michał Kopeć .agents/skills/ckb-retrieve/, .agents/skills/ckb-lint/, tests/bench_retrieval.py, tests/fixtures/bench-kb/ 0.7 0.8 2026-11-02 365 high

D-0015 — A section index for retrieval, and an access log for demotion

Status: Accepted · Decided: 2026-11-02 · Deciders: Michał Kopeć

Context

Release 3, batch 1 of the agent-knowledge plan: F10 (hybrid lexical search with section hits), F12 (layered retrieval with token cost), F16 (access log and demotion) and F17 (answer formats). Retrieval ranked whole pages from rg counts and tldr overlap, so it could say which page but not where in it, and it gave no idea what reading a candidate would cost.

Decision

  1. SQLite FTS5 over sections, standard library only. Each ##–#### section is one row. A separate context column holds the page title, heading path and tldr (Anthropic's contextual-retrieval idea): it counts for ranking pages, but a section is named only when its own text matches. Otherwise every section of a page would match the page's tldr.
  2. A cache, not a source. tmp/ckb-fts.sqlite, gitignored, refreshed per file by mtime and then sha256, rebuilt on a schema change or corruption. It is in memory when tmp/ is absent (the fixtures), and skipped with a header line when this Python's sqlite3 lacks FTS5.
  3. One more list in the fusion, weight 1.0. On the new 20-question benchmark (tests/bench_retrieval.py, tests/fixtures/bench-kb/), it leaves hit@1 (14/20) and hit@3 (20/20) unchanged. Weights of 1.5–3 move hit@1 by at most one question, so they were not tuned to the set. The misses come from the tldr signal matching generic words in vague summaries, which is recorded as a follow-up. The smoke test fails if bm25 ever makes hit@3 worse.
  4. Token estimates (bytes ÷ 4) per candidate, so reading can go index → section → page.
  5. --opened appends to tmp/access.log: per machine, uncommitted, no git churn. lint_report.py --cold [DAYS] proposes moving unread pages into a ## Archive section of their directory index. It waits until the log covers DAYS and skips infrastructure, decisions, vocabulary, lessons and retention: high. It never renames, moves or deletes.

Alternatives considered

  • Vectors or a reranker were rejected: they need dependencies the template avoids. They stay deferred until lexical search is shown to be the bottleneck.
  • A committed access log was rejected: every read would be a commit.
  • Raising the bm25 weight to win the benchmark was rejected: 20 questions are too few to tune on.

Consequences

  • Answers can cite a section anchor, and cheap questions read one section.
  • The cold check is one person's view; the skill treats it as a weak signal.
  • tldr-signal noise is now measurable and is the next ranking problem.