--- type: decision tldr: Retrieval gains a stdlib SQLite FTS5 index of page sections (with a title, heading path and tldr context column), cached in tmp/ and rebuilt incrementally, fused as one more ranked list at weight 1.0; candidates carry section anchors and token estimates; pages actually read go to a per-machine tmp/access.log that lint turns into demotion proposals, never moves. status: accepted decided_on: 2026-11-02 decided_by: Michał Kopeć affects: .agents/skills/ckb-retrieve/, .agents/skills/ckb-lint/, tests/bench_retrieval.py, tests/fixtures/bench-kb/ confidence: 0.7 quality: 0.8 last_updated: 2026-11-02 freshness_window_days: 365 retention: high --- # D-0015 — A section index for retrieval, and an access log for demotion **Status:** Accepted · **Decided:** 2026-11-02 · **Deciders:** Michał Kopeć ## Context Release 3, batch 1 of the agent-knowledge plan: F10 (hybrid lexical search with section hits), F12 (layered retrieval with token cost), F16 (access log and demotion) and F17 (answer formats). Retrieval ranked whole pages from `rg` counts and `tldr` overlap, so it could say *which page* but not *where in it*, and it gave no idea what reading a candidate would cost. ## Decision 1. **SQLite FTS5 over sections, standard library only.** Each `##`–`####` section is one row. A separate `context` column holds the page title, heading path and `tldr` (Anthropic's contextual-retrieval idea): it counts for ranking pages, but a section is *named* only when its own text matches. Otherwise every section of a page would match the page's `tldr`. 2. **A cache, not a source.** `tmp/ckb-fts.sqlite`, gitignored, refreshed per file by mtime and then sha256, rebuilt on a schema change or corruption. It is in memory when `tmp/` is absent (the fixtures), and skipped with a header line when this Python's `sqlite3` lacks FTS5. 3. **One more list in the fusion, weight 1.0.** On the new 20-question benchmark (`tests/bench_retrieval.py`, `tests/fixtures/bench-kb/`), it leaves hit@1 (14/20) and hit@3 (20/20) unchanged. Weights of 1.5–3 move hit@1 by at most one question, so they were not tuned to the set. The misses come from the `tldr` signal matching generic words in vague summaries, which is recorded as a follow-up. The smoke test fails if bm25 ever makes hit@3 worse. 4. **Token estimates** (bytes ÷ 4) per candidate, so reading can go index → section → page. 5. **`--opened` appends to `tmp/access.log`**: per machine, uncommitted, no git churn. `lint_report.py --cold [DAYS]` proposes moving unread pages into a `## Archive` section of their directory index. It waits until the log covers DAYS and skips infrastructure, decisions, vocabulary, lessons and `retention: high`. It never renames, moves or deletes. ## Alternatives considered - **Vectors or a reranker** were rejected: they need dependencies the template avoids. They stay deferred until lexical search is shown to be the bottleneck. - **A committed access log** was rejected: every read would be a commit. - **Raising the bm25 weight to win the benchmark** was rejected: 20 questions are too few to tune on. ## Consequences - Answers can cite a section anchor, and cheap questions read one section. - The cold check is one person's view; the skill treats it as a weak signal. - `tldr`-signal noise is now measurable and is the next ranking problem.