331 lines
15 KiB
Markdown
331 lines
15 KiB
Markdown
---
|
||
name: ckb-ingest
|
||
description: Process raw/inbox/ (or raw/ directly if the inbox is empty) into the structured wiki/: extract typed entities and relationships, synthesize frontmatted pages, cross-link them, then update the index and log. Use for "Ingest" or "Update the wiki". Never for a phrase containing "sync" — that is git, `ckb-sync-changes`.
|
||
---
|
||
|
||
# Ingest skill
|
||
|
||
## Purpose
|
||
|
||
Writes to `wiki/`, moves processed files within `raw/`, and leaves one
|
||
change report per run in `workload/ingest-reports/` (Step 8). Never
|
||
touches `linked/`, `libs/`, or the git remote; pushing is a separate,
|
||
explicit step (Step 9).
|
||
|
||
## How to run this skill
|
||
|
||
### Step 1 — Process the inbox
|
||
|
||
Scan `raw/inbox/` for new material. After ingesting each item, move it to
|
||
`raw/archive/<YYYY-MM-DD>/`, where the date is today's ingestion date
|
||
(create the dated folder if it doesn't exist yet).
|
||
|
||
If `raw/inbox/` is empty, scan `raw/` directly instead — excluding
|
||
`raw/archive/`, which holds material already processed by a previous
|
||
ingest, `raw/retractions/`, which holds removal requests for `ckb-retract`,
|
||
and `raw/saved-conversations/`, which holds verbatim transcripts
|
||
that are deliberately outside the wiki. If `raw/retractions/` holds any
|
||
request, say so before ingesting and offer to run `ckb-retract` first. A saved conversation is ingested
|
||
only when the user has copied its `.md` into `raw/inbox/`; treat it then
|
||
like any other transcript (the "long transcript" guidance below applies).
|
||
|
||
### Step 2 — Consult the cascade
|
||
|
||
Before writing anything, check whether the entity already exists: local
|
||
`wiki/` first, then each `linked/<name>/`, then each `libs/<name>/`
|
||
(alphabetical within each layer). The local `wiki/` always wins — upstream
|
||
content is informative context, but it can be overridden locally rather
|
||
than treated as final.
|
||
|
||
### Step 3 — Extract entities
|
||
|
||
Identify typed entities in the source material — people, projects,
|
||
libraries, concepts, systems. Create entity pages at
|
||
`wiki/entities/<entity-name>.md` if they don't already exist. Record typed
|
||
relationships between entities as edge data in `wiki/graph/edges.json`,
|
||
using the closed vocabulary in `wiki/graph/index.md` — `part_of`, `uses`,
|
||
`depends_on`, `produces`, `configures`, `validates`, `implements`, `caused`,
|
||
`contradicts`, `supersedes`. Each verb is defined there by the question it
|
||
answers; if the relationship you have in mind doesn't answer one of those
|
||
questions, it belongs in the page's prose, not in the graph.
|
||
|
||
Also record `has_expertise_in` and `owns` edges when the material actually
|
||
evidences them: `has_expertise_in` when a person demonstrably answers
|
||
questions or explains decisions on a topic, `owns` when they hold
|
||
declared responsibility for a system, area, or decision. These are what
|
||
make "who knows about X" and "who owns X" answerable as a direct graph
|
||
lookup instead of a full-text guess (see `ckb-retrieve` Step 3). Record
|
||
them only from demonstrated evidence — someone being present in a meeting
|
||
or cc'd on a thread is not expertise, and don't infer ownership from job
|
||
title alone. Both are optional, like every other edge type; an absent
|
||
edge is better than a fabricated one.
|
||
|
||
**Decisions are not entity pages.** When the material records a choice that
|
||
was actually made — a call with a rationale, a date, and someone who made it
|
||
— that belongs in `wiki/decisions/` as a numbered decision record, not in
|
||
`wiki/entities/`. Use `ckb-decide`'s format (it owns the field set, the
|
||
numbering, and the two-sided supersession links) and record them in
|
||
chronological order so decision numbers run in the same order as the
|
||
decisions. Two things worth being strict about, because ingest is where they
|
||
go wrong: a proposal that was discussed but not settled is
|
||
`status: proposed`, never `accepted`; and where the source doesn't say who
|
||
decided, `decided_by: unknown` beats attributing it to whoever was loudest in
|
||
the notes. If the material only *references* a decision without recording
|
||
it ("as agreed last quarter, we use Postgres"), that's a query-gap candidate,
|
||
not a decision record — you don't have the rationale or the date.
|
||
|
||
For recurring teams, clients, systems, or initiatives, consider whether a
|
||
plain project scope page under `wiki/projects/<name>.md` would make future
|
||
queries easier to route. A scope page should list when to use it, included
|
||
wiki/entity/source paths, exclusions, and refresh hints. Create one only
|
||
when the source material shows a real repeated scope; don't manufacture
|
||
scopes for one-off facts.
|
||
|
||
### Step 4 — Blast radius: what does this material invalidate?
|
||
|
||
Ingest is additive by habit, which is how a wiki accumulates two pages that
|
||
quietly disagree. Before writing anything, find out what the incoming
|
||
material lands on.
|
||
|
||
For every entity from Step 3 that already has a page, walk
|
||
`wiki/graph/edges.json` **backwards** — every edge whose `to` is that page —
|
||
one hop out. That set, plus the entity pages themselves, is the blast radius:
|
||
the pages whose claims could be affected by what just arrived. Read their
|
||
`tldr`s (and the bodies of any that look genuinely exposed) and sort each one
|
||
into:
|
||
|
||
- **Confirms** — the new material agrees. Note it; corroboration is a reason
|
||
to raise `confidence` in Step 5, not to rewrite anything.
|
||
- **Extends** — the new material adds detail the page doesn't have. Update
|
||
the page in Step 5.
|
||
- **Contradicts** — the new material and the page can't both be right. This
|
||
is the case worth catching: handle it as supersession (see Edge cases), and
|
||
say so explicitly in the report rather than letting the newer page silently
|
||
win.
|
||
- **Untouched** — most of the radius. Drop it and move on.
|
||
|
||
Name the owners too. Where an `owns` or `has_expertise_in` edge points at an
|
||
affected page, that person is who should review the change — surface them in
|
||
the Step 8 report and the Step 9 summary. Skip this step entirely when
|
||
Step 3 produced no entity
|
||
that already exists; a first ingest into an empty wiki has no radius.
|
||
|
||
### Step 5 — Synthesize pages
|
||
|
||
Convert the core knowledge into clean, modular Markdown files. Every page
|
||
gets frontmatter with:
|
||
- A `tldr:` — one sentence, optimised for LLM reading.
|
||
- A `confidence:` score (0.0–1.0, based on source corroboration).
|
||
- A `quality:` self-score (0.0–1.0).
|
||
- A `last_updated:` timestamp.
|
||
- A `freshness_window_days:` appropriate to the topic.
|
||
- A `retention:` level.
|
||
|
||
`type` is also required (per the page frontmatter schema in
|
||
`CLAUDE.md`/`AGENTS.md` page schema) — set it once, based on the entity/content
|
||
kind (person, project, concept, library, decision, playbook, ...).
|
||
|
||
Every synthesized page ends with the reserved body sections from the page
|
||
schema. `## Sources` and `## Crux` are written here; `## Notes` is created
|
||
empty and never touched again:
|
||
|
||
```markdown
|
||
## Sources
|
||
|
||
- `raw/archive/2026-09-21/kickoff-notes.md` — sha256:3f9a2c1e (checked 2026-09-21)
|
||
- `raw/archive/2026-09-21/access-thread.eml` — sha256:be40d7aa (checked 2026-09-21)
|
||
|
||
## Crux
|
||
|
||
> FDEs need the VDI *and* a Jira account before day one; the VDI request
|
||
> alone takes ten working days.
|
||
— `raw/archive/2026-09-21/kickoff-notes.md`, under "Access"
|
||
|
||
> Access requests go through the platform team, not through IT.
|
||
— `raw/archive/2026-09-21/access-thread.eml`
|
||
|
||
## Notes
|
||
|
||
<!-- Yours. Never rewritten by any skill. -->
|
||
```
|
||
|
||
Compute each fingerprint from the archived file, not from memory:
|
||
|
||
```bash
|
||
sha256sum raw/archive/<date>/<file> | cut -c1-8
|
||
```
|
||
|
||
Record the digest the page was actually built from. That is what lets
|
||
`ckb-lint` tell "old but still accurate" apart from "the source changed
|
||
underneath this page" without asking a model — a calendar window can't
|
||
distinguish those. When a page is built from exactly one source, also set
|
||
`source_fingerprint` and `source_checked` in frontmatter.
|
||
|
||
**`## Crux` is quoted, never paraphrased.** Lift the handful of lines that
|
||
actually carry the claim — the number, the constraint, the condition, the
|
||
commitment — verbatim, and attribute each quote to the source bullet it came
|
||
from: a block of consecutive `>` lines is followed by its own `— \`path\``
|
||
line, and a page may quote several sources this way. `ckb-lint` checks
|
||
each block only against the source named under it. Three to ten lines is the working range; a Crux approaching the length
|
||
of the summary has stopped being evidence and become a second copy of the
|
||
source. The synthesis above it says what the material means; the Crux is what
|
||
it said, so a reader can check the first against the second. This is also
|
||
what lets `ckb-retrieve` ground an answer without opening the archive every
|
||
time.
|
||
|
||
A page with no quotable source — synthesized from conversation, or from
|
||
material too diffuse to quote — simply has no `## Crux`. An empty or
|
||
paraphrased one is worse than none, because it looks like evidence.
|
||
|
||
For long conversations, meeting notes, transcripts, or chat exports, use a
|
||
structured distillation before writing the final page:
|
||
- `Question` or searchable problem statement, when there is one.
|
||
- `Summary` of the thread/note.
|
||
- `Resolution` or `Decision`, if the material contains one.
|
||
- `Systems and code references` mentioned.
|
||
- `People involved` or apparent owners/experts.
|
||
- `High-signal excerpts` for dense technical paragraphs or consecutive
|
||
messages that would be lost in a single summary.
|
||
|
||
"High-signal" needs an acceptance test, or every excerpt looks worth
|
||
keeping and the page becomes a second copy of the transcript. Promote a
|
||
run of text to its own section or linked page only when it clears all
|
||
three:
|
||
- **It carries a rare term.** Something specific enough that a future
|
||
search would use it — a config flag, an error string, a hostname, a
|
||
contract clause, a version number. Check with `rg -c` across `wiki/`:
|
||
if the term already appears on many pages it isn't a distinguishing
|
||
handle, and the excerpt adds no findability the summary doesn't have.
|
||
- **It's substantial.** Roughly 200 characters or more, or a few
|
||
consecutive paragraphs/messages from one author. A one-line "yes, do
|
||
that" is a resolution to fold into `Resolution`, not an excerpt.
|
||
- **Something corroborates it.** It was agreed with, acted on, corrected,
|
||
or referred back to later in the material. An unanswered assertion is a
|
||
claim, not a settled fact — keep it in the summary with that ambiguity
|
||
intact rather than promoting it.
|
||
|
||
Fail any one of the three and the content still belongs in the page, just
|
||
inside `Summary`/`Resolution` rather than as its own retrievable unit.
|
||
|
||
When you do promote an excerpt, carry its parent topic with it — the
|
||
thread question or section heading it sat under. An excerpt that reads
|
||
unambiguously on its own is the entire point; one that needs the
|
||
surrounding transcript to make sense hasn't been extracted, only moved.
|
||
|
||
### Step 6 — Link and cross-reference
|
||
|
||
Use **both** `[[Wikilinks]]` (Obsidian-compatible) and standard
|
||
`[markdown](path.md)` links on every cross-reference, so the wiki works in
|
||
Obsidian, GitHub, and CLI tools alike. Where useful, reference upstream
|
||
files directly at `linked/<name>/...` or `libs/<name>/...`.
|
||
|
||
### Step 7 — Update index and log
|
||
|
||
Add new pages to the routing table in `wiki/index.md` with a **Use when**
|
||
description. If the page lives in a subdirectory, also add it to that
|
||
subdirectory's own `index.md`. Append a log entry to the most specific
|
||
applicable log — the subdirectory's `log.md` if it has one, otherwise the
|
||
root `wiki/log.md` — following the format in Rule B (`CLAUDE.md`/
|
||
`AGENTS.md` Rule B).
|
||
|
||
If this step creates a brand-new `wiki/<topic>/` subdirectory, immediately
|
||
create that subdirectory's `index.md` per the Recursive Index & Log
|
||
Convention.
|
||
|
||
If this step creates or updates a project scope under `wiki/projects/`,
|
||
also update `wiki/projects/index.md`. If ingest closes a previously
|
||
recorded question in `wiki/query-gaps.md`, move that entry from Open to
|
||
Resolved and mention the page or source that now answers it.
|
||
|
||
### Step 8 — Write the ingest report
|
||
|
||
Every run that ingested something leaves a report file, so what an ingest
|
||
changed can be reviewed later without reconstructing it from `wiki/log.md`
|
||
and the git diff. Write it after Step 7, from what this run actually did —
|
||
not from memory of what it meant to do.
|
||
|
||
One file per run, never appended to or rewritten afterwards:
|
||
`workload/ingest-reports/YYYY-MM-DD_HHMM_<author>_ingest.md`, with the
|
||
start time of the run and the Rule D author slug. Create the folder if it
|
||
is missing. It is a subfolder on purpose: the session-start script reads
|
||
only top-level `workload/*.md` notes.
|
||
|
||
```markdown
|
||
---
|
||
type: ingest-report
|
||
author: Name
|
||
author_email: name@example.com
|
||
date: YYYY-MM-DD
|
||
started: HH:MM
|
||
---
|
||
|
||
# Ingest report — YYYY-MM-DD HH:MM
|
||
|
||
## Raw files
|
||
- `raw/inbox/kickoff.md` → `raw/archive/2026-09-21/kickoff.md` — sha256:3f9a2c1e
|
||
|
||
## Decisions recorded
|
||
- `/wiki/decisions/0007-vdi-first.md` — accepted — "VDI before Jira"
|
||
|
||
## Entities
|
||
- **Created:** `/wiki/entities/platform-team.md` — the team that owns access requests
|
||
- **Updated:** `/wiki/entities/onboarding.md` — extends: ten-day VDI lead time
|
||
|
||
## Other pages
|
||
- `/wiki/projects/fde-onboarding.md` — created
|
||
- `/wiki/index.md`, `/wiki/entities/index.md` — routing rows added
|
||
|
||
## Graph
|
||
- `platform-team` owns `vdi-access` (new edge)
|
||
|
||
## Blast radius
|
||
- **Confirmed:** …
|
||
- **Extended:** …
|
||
- **Contradicted:** `/wiki/entities/x.md` — superseded by … — review: Owner Name
|
||
- **Query gaps closed:** …
|
||
|
||
## Not ingested
|
||
- `raw/inbox/scan.pdf` — unreadable, left in the inbox
|
||
```
|
||
|
||
Every section stays, with `- none` when it is empty: an explicit "no
|
||
decisions recorded" is information, a missing heading is not. List each
|
||
page once, under the section of its type; a page a raw file produced
|
||
appears there, not again under Raw files. One line per item says what
|
||
changed, not what the page is about. **Not ingested** names every inbox
|
||
file skipped or deferred, with the reason, so nothing waits in the inbox
|
||
unnoticed.
|
||
|
||
The report is a record, not a wiki page: it is outside retrieval, needs no
|
||
`wiki/log.md` entry, and is never edited by a later ingest or by
|
||
`ckb-retract` — a retraction leaves its own trail in the log. Name the
|
||
report's path in today's Rule D workload note.
|
||
|
||
If nothing was ingested, write no report.
|
||
|
||
### Step 9 — Report, then remind to review and sync
|
||
|
||
Report what changed before the reminder: pages created vs. updated, and —
|
||
from Step 4 — which existing pages the new material confirmed, extended, or
|
||
contradicted, naming the owner of each contradicted page. A contradiction
|
||
resolved silently is the one outcome of an ingest a reviewer must not have to
|
||
discover for themselves. End the summary with the path of the Step 8 report.
|
||
|
||
Then close with a short reminder — do not sync or push
|
||
anything yourself here, and do not skip this even if the changes look
|
||
small or obviously correct:
|
||
|
||
> "Ingest complete — `wiki/` has been updated (see `wiki/log.md` and the
|
||
> ingest report in `workload/ingest-reports/` for the details). Please
|
||
> review the changes, and once you're happy with them,
|
||
> say 'sync changes' to commit and push them to `origin`."
|
||
|
||
If nothing was actually ingested (empty inbox and empty `raw/`, nothing to
|
||
process), skip this reminder — there's nothing to review or sync.
|
||
|
||
## Edge cases
|
||
|
||
Unusual situations — an empty target, a partial run, a conflicting state,
|
||
a missing dependency — are catalogued in
|
||
[`references/edge-cases.md`](references/edge-cases.md). Read that file when a
|
||
run hits something the steps above don't cover; don't load it otherwise.
|