- F10 bm25 section search: stdlib SQLite FTS5 over page sections, with title/heading path/tldr in a separate context column; cached in tmp/ckb-fts.sqlite, refreshed by mtime then sha256; section anchors per candidate. New benchmark tests/bench_retrieval.py over tests/fixtures/bench-kb (20 questions): hit@1 14/20, hit@3 20/20 with and without bm25; weight kept at 1.0 (D-0015) - F12 token estimate per candidate; layered reading in ckb-retrieve - F16 --opened access log in tmp/access.log; lint_report.py --cold proposes demotion to an index `## Archive` section, never moves - F17 ckb-retrieve offers a table/chart/deck/HTML once when it fits Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
67 lines
3.3 KiB
Markdown
67 lines
3.3 KiB
Markdown
---
|
||
type: decision
|
||
tldr: Retrieval gains a stdlib SQLite FTS5 index of page sections (with a title, heading path and tldr context column), cached in tmp/ and rebuilt incrementally, fused as one more ranked list at weight 1.0; candidates carry section anchors and token estimates; pages actually read go to a per-machine tmp/access.log that lint turns into demotion proposals, never moves.
|
||
status: accepted
|
||
decided_on: 2026-11-02
|
||
decided_by: Michał Kopeć
|
||
affects: .agents/skills/ckb-retrieve/, .agents/skills/ckb-lint/, tests/bench_retrieval.py, tests/fixtures/bench-kb/
|
||
confidence: 0.7
|
||
quality: 0.8
|
||
last_updated: 2026-11-02
|
||
freshness_window_days: 365
|
||
retention: high
|
||
---
|
||
|
||
# D-0015 — A section index for retrieval, and an access log for demotion
|
||
|
||
**Status:** Accepted · **Decided:** 2026-11-02 · **Deciders:** Michał Kopeć
|
||
|
||
## Context
|
||
|
||
Release 3, batch 1 of the agent-knowledge plan: F10 (hybrid lexical search
|
||
with section hits), F12 (layered retrieval with token cost), F16 (access log
|
||
and demotion) and F17 (answer formats). Retrieval ranked whole pages from
|
||
`rg` counts and `tldr` overlap, so it could say *which page* but not *where
|
||
in it*, and it gave no idea what reading a candidate would cost.
|
||
|
||
## Decision
|
||
|
||
1. **SQLite FTS5 over sections, standard library only.** Each `##`–`####`
|
||
section is one row. A separate `context` column holds the page title,
|
||
heading path and `tldr` (Anthropic's contextual-retrieval idea): it counts
|
||
for ranking pages, but a section is *named* only when its own text
|
||
matches. Otherwise every section of a page would match the page's `tldr`.
|
||
2. **A cache, not a source.** `tmp/ckb-fts.sqlite`, gitignored, refreshed
|
||
per file by mtime and then sha256, rebuilt on a schema change or
|
||
corruption. It is in memory when `tmp/` is absent (the fixtures), and
|
||
skipped with a header line when this Python's `sqlite3` lacks FTS5.
|
||
3. **One more list in the fusion, weight 1.0.** On the new 20-question
|
||
benchmark (`tests/bench_retrieval.py`, `tests/fixtures/bench-kb/`), it
|
||
leaves hit@1 (14/20) and hit@3 (20/20) unchanged. Weights of 1.5–3 move
|
||
hit@1 by at most one question, so they were not tuned to the set. The
|
||
misses come from the `tldr` signal matching generic words in vague
|
||
summaries, which is recorded as a follow-up. The smoke test fails if bm25
|
||
ever makes hit@3 worse.
|
||
4. **Token estimates** (bytes ÷ 4) per candidate, so reading can go index →
|
||
section → page.
|
||
5. **`--opened` appends to `tmp/access.log`**: per machine, uncommitted, no
|
||
git churn. `lint_report.py --cold [DAYS]` proposes moving unread pages
|
||
into a `## Archive` section of their directory index. It waits until the
|
||
log covers DAYS and skips infrastructure, decisions, vocabulary, lessons
|
||
and `retention: high`. It never renames, moves or deletes.
|
||
|
||
## Alternatives considered
|
||
|
||
- **Vectors or a reranker** were rejected: they need dependencies the
|
||
template avoids. They stay deferred until lexical search is shown to be
|
||
the bottleneck.
|
||
- **A committed access log** was rejected: every read would be a commit.
|
||
- **Raising the bm25 weight to win the benchmark** was rejected: 20
|
||
questions are too few to tune on.
|
||
|
||
## Consequences
|
||
|
||
- Answers can cite a section anchor, and cheap questions read one section.
|
||
- The cold check is one person's view; the skill treats it as a weak
|
||
signal.
|
||
- `tldr`-signal noise is now measurable and is the next ranking problem.
|