diff --git a/.agents/skills/ckb-index-external/SKILL.md b/.agents/skills/ckb-index-external/SKILL.md index 30d163a..e9d3b7a 100644 --- a/.agents/skills/ckb-index-external/SKILL.md +++ b/.agents/skills/ckb-index-external/SKILL.md @@ -1,6 +1,6 @@ --- name: ckb-index-external -description: Walk every connector-backed libs// (identified by a source.yaml declaring a live external source — SharePoint, Google Drive, a plain URL, or another connector) and build/refresh a self-contained generated index for it — index.md/entities/graph/log.md, mirroring wiki/'s own shape but scoped entirely to that one connector. The index can optionally be published to (and fetched from) a shared location — a git repo or a shared resource — declared in source.yaml, so most users just read an already-built index instead of re-scanning the live source. Read vs. write access to a given source is a per-user, local-only setting (libs//source.local.yaml, gitignored) — read-only by default, so one or two designated admins can maintain a source for everyone else. Never touches wiki/, never touches source.yaml, never touches a git-copy lib. Use when the user says "index external sources", "index libs", "refresh the external index", or "scan the SharePoint/Drive folder". This is the on-demand workflow from CLAUDE.md/AGENTS.md §4, distinct from `ckb-ingest` (which processes raw/inbox/ into wiki/) and from a plain git-copy libs// (a frozen clone, never touched by any skill). +description: Walk every connector-backed libs// (identified by a source.yaml declaring a live external source — SharePoint, Google Drive, a plain URL, or another connector) and build/refresh a self-contained generated index for it — index.md/entities/graph/log.md, mirroring wiki/'s own shape but scoped entirely to that one connector. Indexing goes beyond one page per document: for every document found, it also extracts the people, organizations, projects, decisions, systems, and processes that document actually discusses into their own thin, pointer-style entity pages (evidence back to the specific documents that mention them, not a second copy of wiki/'s synthesis), so the index supports "what do we know about X / what's the process for Y" lookups, not just "what documents exist at this source". The index can optionally be published to (and fetched from) a shared location — a git repo or a shared resource — declared in source.yaml, so most users just read an already-built index instead of re-scanning the live source. Read vs. write access to a given source is a per-user, local-only setting (libs//source.local.yaml, gitignored) — read-only by default, so one or two designated admins can maintain a source for everyone else. Never touches wiki/, never touches source.yaml, never touches a git-copy lib. Use when the user says "index external sources", "index libs", "refresh the external index", or "scan the SharePoint/Drive folder". This is the on-demand workflow from CLAUDE.md/AGENTS.md §4, distinct from `ckb-ingest` (which processes raw/inbox/ into wiki/) and from a plain git-copy libs// (a frozen clone, never touched by any skill). --- # Index external sources skill @@ -12,16 +12,33 @@ git-copy one: a `libs//source.yaml` declares a *live* external source — a SharePoint folder, a Google Drive folder, a plain URL, or another connector — that the user doesn't want to fully mirror locally. This skill walks that source and builds a small, self-contained index of -what's there: one entity page per document found, plus the same -`index.md`/`graph`/`log.md` shape `wiki/` already uses, rooted at -`libs//` instead. +what's there, using the same `index.md`/`graph`/`log.md` shape `wiki/` +already uses, rooted at `libs//` instead. + +The index holds two kinds of pages, and both matter for retrieval: + +- **Document pages** — one per source item found (one per email, one per + file, one per SharePoint page), the original design. These answer "what + exists at this source". +- **Entity and process pages** — one per person, organization, project, + decision, system, or named process that those documents actually + *discuss*, extracted the same way `ckb-ingest` extracts entities into + `wiki/`, but kept deliberately thin here: a `tldr` plus a list of which + document(s) mention it and what they say, not a full synthesis. These + answer "what do we know about X" without forcing whoever's asking to + read every document's `tldr` by hand looking for a name. A document-only + index is fine for browsing a source but too narrow for actually + retrieving information out of it — that's what this second pass fixes. This is deliberately **not** blended into the main `wiki/entities/` or `wiki/graph/edges.json` — the generated index lives entirely inside its own `libs//`, at the lowest cascade layer, the same way a git-cloned KB's own files would. If something in it needs to override or correct what the connector says, that's what writing the corrected version into `wiki/` -is for (cascade priority already covers that — `wiki/` always wins). +is for (cascade priority already covers that — `wiki/` always wins). Where +an extracted entity already has a full page in `wiki/entities/`, this +skill's own page for it stays thin and points there rather than +re-synthesizing — see Step 6. Two refinements on top of that base design: @@ -43,7 +60,7 @@ Two refinements on top of that base design: This skill only ever writes within a connector-backed `libs//` (never `source.yaml` itself, never a git-copy lib, never anything under `wiki/`) — and it never touches this repo's own git remote; pushing the resulting -changes is a separate, explicit step (see Step 7). +changes is a separate, explicit step (see Step 8). ## Trigger phrases @@ -60,9 +77,13 @@ Use this skill when the user says: the same turn unless the user also asks for one. Do **not** use this skill for "Ingest" / "Sync the wiki" (that's -`ckb-ingest` — raw material into `wiki/`, unrelated to `libs/`) or for +`ckb-ingest` — raw material into `wiki/`, unrelated to `libs/`), for "Lint" (that's `ckb-lint`, which now also health-checks a connector's own -generated index, but doesn't build it in the first place). +generated index, but doesn't build it in the first place), or for +answering a question against an index that already exists (that's +`ckb-retrieve`, which reads what this skill built and always verifies +against the underlying source before answering — this skill never answers +questions, only builds/refreshes the index). ## Configuration @@ -159,7 +180,7 @@ This is the one place access level actually changes behavior: - **Write access**: always continue to Step 5, whether Step 3 fetched an existing index (refresh it) or found the store empty (build the very first version from scratch) — the two cases are handled identically - from here on; Step 7 is what actually creates the remote copy either + from here on; Step 8 is what actually creates the remote copy either way. ### Step 5 — Resolve the connector and enumerate/summarize documents @@ -181,8 +202,8 @@ For each document found at the location: - Read enough content to write a real 1-2 sentence `tldr` when the connector allows fetching content. - If only metadata is available (listing only, no read access), say so - honestly in the `tldr` (e.g. "Metadata only — read access not granted") - and set `confidence` low (e.g. 0.3) rather than fabricating a summary. + honestly in the `tldr` (e.g. "Metadata only — content not readable") and + set `confidence` low (e.g. 0.3) rather than fabricating a summary. Create/update one entity page per document at `libs//entities/.md`: @@ -199,7 +220,67 @@ retention: medium --- ``` -### Step 6 — Update the hub page, graph, index, and log +### Step 6 — Extract entities and processes mentioned within each document + +A document-only index tells a reader *what exists*, but not *what's known +about a given person, topic, or procedure* without reading every `tldr` by +hand. Close that gap the same way `ckb-ingest` extracts entities into +`wiki/`, but keep the result scoped and thin — this connector's own layer +is a retrieval index into its source documents, not a second knowledge +base competing with `wiki/`'s synthesis. + +For each document processed in Step 5, identify what it actually +*discusses*, not just what it *is*: named people, organizations, projects, +decisions, systems, and processes/procedures (e.g. "VDI access request +process", "candidate profile approval") mentioned in its content. Skip +this for a document that's purely about itself with nothing else +extractable (a bare calendar acceptance with no discussion, for instance) +— not every document yields additional pages. + +For each distinct entity/process found across the source's documents, +create or update one page at `libs//entities/.md`, +using the same slug convention `wiki/entities/` would use for that name +(so `damien-gultig.md`, not a date-prefixed slug — this is what keeps +entity-page filenames from colliding with document-page filenames, which +are always date-prefixed per Step 5): + +```yaml +--- +type: person | organization | project | decision | system | concept | ... # same open field as wiki/'s schema +resource: /wiki/entities/.md # OPTIONAL — only if a full page for this entity already exists in wiki/ +tldr: One sentence — who/what this is, and what these documents specifically show (not a full biography; + if wiki/ already has a full page, this tldr should say what these documents add, not restate it) +confidence: 0.3–0.9 +quality: ... +last_updated: YYYY-MM-DD +freshness_window_days: 30 +retention: medium +--- +``` + +Body: a short "Mentioned in" list, one bullet per document that discusses +this entity, linking to that document's own page +(`entities/.md`) with a one-line note of what that specific +document says — enough for `ckb-retrieve` (or a human) to know exactly +which document to open for the full context, without needing to re-derive +it from scratch. + +**If this entity already has a full page in `wiki/entities/`** (check the +cascade first, same as `ckb-ingest`'s Step 2), set `resource:` to that +page's path and keep this page's body to just the "Mentioned in" list — +don't re-synthesize what the wiki page already says. The wiki page remains +authoritative (cascade priority); this page's only job is pointing back to +*these specific documents* as additional evidence, which the wiki page may +not have cited yet. + +Record real relationships surfaced by a document between two entities +(`uses`, `depends_on`, `caused`, `contradicts`) as edges in +`libs//graph/edges.json`, same edge vocabulary `wiki/` uses. Also add +a `mentioned_in` edge from each entity to every document that discusses +it — this is what makes the graph a genuine index into the source +documents rather than just a bag of loose pages. + +### Step 7 — Update the hub page, entities index, graph, and log Create/update `libs//index.md` — the root routing page for this connector, mirroring `wiki/index.md`'s own role: @@ -207,26 +288,35 @@ connector, mirroring `wiki/index.md`'s own role: --- type: external-source resource: -tldr: One sentence — what this source is and how many documents were found +tldr: One sentence — what this source is, how many documents were found, and how many entities/processes were extracted from them last_updated: YYYY-MM-DD --- ``` followed by a short routing table pointing to `entities/index.md` and `graph/index.md`. -Record any real relationships between documents (e.g. folder hierarchy) as -edges in `libs//graph/edges.json`, using a `contains`/`part_of`-style -edge type — this is fine to be thin or empty for a flat source with no -useful structure beyond a document list. +Update `libs//entities/index.md` as two sections, so both kinds of +page stay easy to find without conflating them: +```markdown +## Documents +- [](.md) — +... -Update `libs//entities/index.md` and `libs//graph/index.md` -(flat bullet lists, no frontmatter, same convention as `wiki/entities/ -index.md`/`wiki/graph/index.md`). Log every created/updated page in -`libs//log.md`, same reverse-chronological format as Rule B -(`CLAUDE.md`/`AGENTS.md` §7) — this log is independent of `wiki/log.md`; -**nothing under `wiki/` is touched by this skill at all.** +## Entities & Processes +- [](.md) — +... +``` +Same flat-bullet, no-frontmatter convention as `wiki/entities/index.md` +otherwise — this is just a two-heading variant of it, not a new format. -### Step 7 — Publish, if this source has both write access and a configured index store +Update `libs//graph/index.md` (same convention as +`wiki/graph/index.md`) to summarize both the `mentioned_in` document +coverage and any real entity-to-entity edges found. Log every +created/updated page in `libs//log.md`, same reverse-chronological +format as Rule B (`CLAUDE.md`/`AGENTS.md` §7) — this log is independent of +`wiki/log.md`; **nothing under `wiki/` is touched by this skill at all.** + +### Step 8 — Publish, if this source has both write access and a configured index store If this user has `access: write` **and** `source.yaml` has an `index:` block, push the refreshed `libs//{index.md,entities/,graph/,log.md}` @@ -249,7 +339,7 @@ nothing is even rebuilt in `libs//` to begin with). A read-only user has, by construction, nothing of their own to save back — Step 4 already stopped them before anything was built. -### Step 8 — Remind to review and sync +### Step 9 — Remind to review and sync This is always the last step, every time this skill makes any change at all. Close with a short reminder — do not sync or push anything yourself @@ -285,18 +375,35 @@ was read-only with nothing new to fetch), skip this reminder. the whole run; don't let a transient network issue block a read-only user from seeing the last-known index. - **Re-running against an already-indexed source** — refresh existing - entity pages in place (update `tldr`/`last_updated`/`confidence`); never - duplicate a document's page. + document *and* entity/process pages in place (update + `tldr`/`last_updated`/`confidence`, add newly-seen `mentioned_in` edges); + never duplicate a document's or entity's page. - **A previously-indexed document is no longer found at the source** — don't delete its page. Update its `tldr` with a note ("No longer found at source as of YYYY-MM-DD") and set `retention: low`, so the existing `ckb-lint` retention sweep archives it naturally on a later pass — no new - archival mechanism needed. + archival mechanism needed. Leave any entity pages that cite it alone; + the citation is still historically accurate even if the document itself + is gone. +- **An extracted entity already has a full page in `wiki/entities/`** — + keep this connector-side page thin (evidence/`mentioned_in` pointers + only, `resource:` set to the wiki page) rather than re-deriving + everything the wiki page already says; that duplication is exactly what + the cascade design is meant to avoid. +- **A document is dense with names/topics and extracting every one would + produce dozens of near-duplicate pages** — extract what's clearly named + and substantively discussed (not every passing mention), and prefer + updating an existing entity page's "Mentioned in" list over creating a + marginal new one. Don't force artificial granularity just to maximize + page count. +- **A document yields no extractable entity beyond itself** — that's fine; + its Step 5 document page is the only page it produces. Not every + document needs to feed Step 6. - **A source is very large** — cap what gets fully fetched/summarized in one run and explicitly report what was skipped (e.g. "42 of ~300 documents summarized this pass — re-run to continue"). Never silently truncate while implying full coverage. -- **Write access but publishing (Step 7) conflicts with a newer version +- **Write access but publishing (Step 8) conflicts with a newer version someone else already pushed** — this is why Step 3 (fetch) always runs first, even for write-access users: rebuild on top of the latest fetched state rather than blindly overwriting it. If a real conflict still shows diff --git a/.agents/skills/ckb-retrieve/SKILL.md b/.agents/skills/ckb-retrieve/SKILL.md new file mode 100644 index 0000000..2299e41 --- /dev/null +++ b/.agents/skills/ckb-retrieve/SKILL.md @@ -0,0 +1,178 @@ +--- +name: ckb-retrieve +description: Governs how a question actually gets answered from the knowledge base. An index entry — a wiki page's `tldr`, a connector-index document/entity page — is deliberately compressed so lazy-loading stays cheap; that compression also means it can be incomplete, paraphrased loosely, or stale relative to the real source. This skill makes "go verify against the source before answering" a standing step, not an optional last resort: every time a page surfaced by the index looks relevant enough to actually ground part of the answer, follow it to its underlying source material — a wiki page's own `## Sources` citations into `raw/archive/`/`outputs/`, or a connector-index page's `resource:` pointer back to the live connector item — before treating its content as settled. Use whenever answering a question or researching a topic from `wiki/`, `linked//`, or a connector-backed `libs//` — this is the retrieval half of CLAUDE.md/AGENTS.md §5 QUERY WORKFLOW, always in play, not something the user needs to name explicitly. Distinct from `ckb-onboard-me` (produces a reading-order tour, not an answer to a specific question) and from the write-side skills `ckb-ingest`/`ckb-index-external`/`ckb-lint` (build or repair the index; this skill only ever reads it, plus the sources behind it). +--- + +# Retrieve (source-verified query) skill + +## Purpose + +The whole point of `tldr`/lazy-loading (CLAUDE.md/AGENTS.md §1 +"Lazy-Loading with 'Use When' Triggers", §5 "QUERY WORKFLOW") is that most +of the wiki never has to enter context — a one-sentence summary decides +whether a page is worth opening at all. That's the right trade for +*deciding relevance*. It's the wrong trade for *grounding an answer*: a +`tldr` is a compression of whatever the page's author judged important at +write time, a connector-index entity page is (per `ckb-index-external`) +deliberately a thin pointer rather than a synthesis, and either can have +drifted from the primary material since — a source updated, a nuance +dropped, a paraphrase that's subtly wrong. + +This skill closes that gap: once a page looks relevant enough to actually +use in an answer, don't stop at its `tldr` or even its full synthesized +body — follow it to the source material that page was built from, and +answer from there. It's the difference between citing what the index +*says about* the evidence and citing the evidence. + +This skill is **read-only**. It never writes to `wiki/`, `libs//`, +or anywhere else — it only reads what already exists (the index, and the +source material behind it) to answer the question in front of it. If +verification surfaces a real gap or contradiction worth fixing in the +wiki, say so and suggest `ckb-lint`/`ckb-ingest` rather than editing +anything mid-answer. + +## Trigger phrases + +This is the default retrieval path for **any** question answered from the +knowledge base — the user does not need to name this skill. Typical +prompts that should route here: +- A direct question answerable from the wiki ("what do we know about X", + "what's the status of Y", "who owns Z"). +- "Look up X" / "check the wiki for X" / "search for X". +- Mid-conversation moments where CLAUDE.md/AGENTS.md §5's QUERY WORKFLOW + applies — this skill *is* that workflow's implementation. + +Do **not** use this skill for: +- Building or refreshing an index in the first place — that's + `ckb-ingest` (raw material → `wiki/`) or `ckb-index-external` + (connector → `libs//`). This skill only ever consumes what those + produced. +- A guided reading order across many pages on a topic — that's + `ckb-onboard-me`. This skill answers one question with verified + evidence; it doesn't produce a curriculum. +- Health-checking the index's own structure (frontmatter, staleness, + broken links) — that's `ckb-lint`. Verification failures found here + (Edge cases, below) are worth mentioning to the user as a possible lint + finding, but this skill doesn't run the lint checklist itself. + +## How to run this skill + +### Step 1 — Read the index + +Same cascade order as CLAUDE.md/AGENTS.md §5, first match wins: +1. `wiki/index.md` — match the question against the **Use when** column. +2. `wiki/entities/index.md` — match against entity titles/`tldr`. +3. If nothing local matches: each `linked//` index (alphabetical), + then each connector-backed `libs//entities/index.md` — for a + connector-backed lib this means its *generated* index (both the + Documents and the Entities & Processes sections `ckb-index-external` + produces), never the live source directly. If it isn't built yet, + suggest "index external sources" rather than querying the live + connector ad hoc from inside this skill. + +### Step 2 — Shortlist every page that looks relevant + +Read the `tldr` of every page the index match surfaced — don't stop at +the first plausible hit. A question is often best answered by +triangulating two or three pages (e.g. a concept page plus the specific +person/decision page that qualifies it), and a page whose `tldr` looks +only tangential can still be hiding the exact fact needed in its body or +its Sources. Keep the shortlist to what's plausibly relevant; this isn't +"open everything," it's "don't stop at one." + +### Step 3 — Walk the graph for anything the shortlist missed + +If a shortlisted page has edges in `wiki/graph/edges.json` (or a +connector's own `graph/edges.json`), follow `depends_on`/`uses`/`caused` +edges one hop out — the same way §5 Step 4 already directs — to catch a +connected page the index text match alone wouldn't have surfaced. + +### Step 4 — Follow every page on the shortlist to its source before answering from it + +This is the step this skill exists to enforce. For each page on the +shortlist that will actually ground part of the answer: + +- **A `wiki/` page** — every page synthesized via `ckb-ingest` carries a + `## Sources` section citing the exact `raw/archive//...` or + `outputs/...` file(s) it was built from. Open the cited file(s) — or + the specific section of a large one — and confirm the wiki's claim + matches what the primary material actually says. This also often + surfaces adjacent detail the synthesis compressed away that's directly + useful for the current question. +- **A connector-index page in `libs//`** (a document page, or an + entity/process page from the `ckb-index-external` extraction) — read + its `resource:` field. If the connector is authorized this session (per + `ToolSearch`/MCP auth state), re-fetch the live item for current, + complete content rather than trusting the cached `tldr` — connector + pages use a deliberately short `freshness_window_days` (30, vs. a + typical wiki page's 60-90) precisely because external sources drift + without notice. If the connector isn't authorized, or the item is a + calendar acceptance / metadata-only page with nothing more to fetch, + say plainly that the answer relies on the cached index rather than a + live re-check, so the user can weigh that. +- **A `linked//` page** — this is already full mirrored content, + not a summary; reading the page itself already is reading the source. + No extra fetch needed unless *that* page itself cites something further + outside the mirror. + +Target the specific claim, not the whole file — a large transcript or +deck doesn't need a full read every time, just enough (search for the +name/topic, read the surrounding context) to confirm the point actually +being used. + +### Step 5 — Reconcile and answer + +If the source confirms the index, answer normally — but note what +grounded it (e.g. "per the kickoff transcript cited in +`gt-fde-access-requirements.md`") rather than presenting the answer as if +sourced from the `tldr` alone. + +If the source contradicts, extends, or is more precise than what the +index said, prefer the source for the answer and say so explicitly — this +is exactly the kind of drift `ckb-lint`'s freshness/confidence checks +exist to eventually catch, so mention it's worth a lint pass if the gap +looks like more than a one-off, but don't rewrite the wiki page yourself +mid-answer unless the user asks for that separately. + +If no page anywhere is even plausibly relevant, say so — don't verify +against a source that has nothing to do with the question just to appear +thorough, and don't fabricate an answer either. + +## Edge cases + +- **A cited source file no longer exists** (moved, renamed, or cleaned + up) — say so plainly rather than silently falling back to the `tldr` as + if it had been verified. Flag it as a likely `ckb-lint` finding (a + broken `Sources` reference) rather than treating it as a dead end. +- **Connector item unreachable or connector not authorized this + session** — state clearly that the answer is based on the cached + connector index, not a live re-check, and name which connector would + need authorizing to go further. +- **Question is low-stakes and the matched page is high-confidence, + recently updated, and well within its `freshness_window_days`** — still + open the source at least once to ground the answer; skip only a *second* + redundant read if the same source has already been opened earlier in + the same retrieval pass for another part of the same question. +- **Many pages are relevant and reading every cited source in full would + be excessive** — prioritize the sources that actually carry the + specific fact the question needs, not every source any shortlisted page + has ever cited. Say what was skipped rather than silently narrowing + without a note. +- **The "source" is itself a generated artifact** (e.g. an + `outputs/emails/*.md` draft cited as a wiki page's Source) rather than + raw external material — that's still the source relative to the wiki + page; no further hop beyond it is required. +- **A connector-index entity/process page's `resource:` points at a full + `wiki/entities/` page** (per `ckb-index-external`'s thin-page + convention) — treat the wiki page as the real source to verify against; + the connector page is just the pointer that got you there. +- **Nothing in the index is even plausibly relevant** — say the knowledge + base has nothing on this yet, and suggest `ckb-ingest` (for new raw + material) or `ckb-index-external` (for a connector-backed source) if + that seems like the actual gap. Don't stretch a weak match into an + answer just to have one. + +--- + +*Licensed under the Apache License, Version 2.0 — see [LICENSE](../../../LICENSE) +at the repository root, or .*