ckb/docs/decisions/0015-section-index-and-access-log.md
Michał Kopeć 1d7a489323 Template 1.24.0: release 3 batch 1, retrieval
- F10 bm25 section search: stdlib SQLite FTS5 over page sections, with
  title/heading path/tldr in a separate context column; cached in
  tmp/ckb-fts.sqlite, refreshed by mtime then sha256; section anchors
  per candidate. New benchmark tests/bench_retrieval.py over
  tests/fixtures/bench-kb (20 questions): hit@1 14/20, hit@3 20/20 with
  and without bm25; weight kept at 1.0 (D-0015)
- F12 token estimate per candidate; layered reading in ckb-retrieve
- F16 --opened access log in tmp/access.log; lint_report.py --cold
  proposes demotion to an index `## Archive` section, never moves
- F17 ckb-retrieve offers a table/chart/deck/HTML once when it fits

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-02 20:25:26 +01:00

67 lines
3.3 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

---
type: decision
tldr: Retrieval gains a stdlib SQLite FTS5 index of page sections (with a title, heading path and tldr context column), cached in tmp/ and rebuilt incrementally, fused as one more ranked list at weight 1.0; candidates carry section anchors and token estimates; pages actually read go to a per-machine tmp/access.log that lint turns into demotion proposals, never moves.
status: accepted
decided_on: 2026-11-02
decided_by: Michał Kopeć
affects: .agents/skills/ckb-retrieve/, .agents/skills/ckb-lint/, tests/bench_retrieval.py, tests/fixtures/bench-kb/
confidence: 0.7
quality: 0.8
last_updated: 2026-11-02
freshness_window_days: 365
retention: high
---
# D-0015 — A section index for retrieval, and an access log for demotion
**Status:** Accepted · **Decided:** 2026-11-02 · **Deciders:** Michał Kopeć
## Context
Release 3, batch 1 of the agent-knowledge plan: F10 (hybrid lexical search
with section hits), F12 (layered retrieval with token cost), F16 (access log
and demotion) and F17 (answer formats). Retrieval ranked whole pages from
`rg` counts and `tldr` overlap, so it could say *which page* but not *where
in it*, and it gave no idea what reading a candidate would cost.
## Decision
1. **SQLite FTS5 over sections, standard library only.** Each `##`–`####`
section is one row. A separate `context` column holds the page title,
heading path and `tldr` (Anthropic's contextual-retrieval idea): it counts
for ranking pages, but a section is *named* only when its own text
matches. Otherwise every section of a page would match the page's `tldr`.
2. **A cache, not a source.** `tmp/ckb-fts.sqlite`, gitignored, refreshed
per file by mtime and then sha256, rebuilt on a schema change or
corruption. It is in memory when `tmp/` is absent (the fixtures), and
skipped with a header line when this Python's `sqlite3` lacks FTS5.
3. **One more list in the fusion, weight 1.0.** On the new 20-question
benchmark (`tests/bench_retrieval.py`, `tests/fixtures/bench-kb/`), it
leaves hit@1 (14/20) and hit@3 (20/20) unchanged. Weights of 1.5–3 move
hit@1 by at most one question, so they were not tuned to the set. The
misses come from the `tldr` signal matching generic words in vague
summaries, which is recorded as a follow-up. The smoke test fails if bm25
ever makes hit@3 worse.
4. **Token estimates** (bytes ÷ 4) per candidate, so reading can go index →
section → page.
5. **`--opened` appends to `tmp/access.log`**: per machine, uncommitted, no
git churn. `lint_report.py --cold [DAYS]` proposes moving unread pages
into a `## Archive` section of their directory index. It waits until the
log covers DAYS and skips infrastructure, decisions, vocabulary, lessons
and `retention: high`. It never renames, moves or deletes.
## Alternatives considered
- **Vectors or a reranker** were rejected: they need dependencies the
template avoids. They stay deferred until lexical search is shown to be
the bottleneck.
- **A committed access log** was rejected: every read would be a commit.
- **Raising the bm25 weight to win the benchmark** was rejected: 20
questions are too few to tune on.
## Consequences
- Answers can cite a section anchor, and cheap questions read one section.
- The cold check is one person's view; the skill treats it as a weak
signal.
- `tldr`-signal noise is now measurable and is the next ranking problem.