michal/ckb
1
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

Extract entities/processes in connector indexes, add source-verified retrieval

Browse source 
ckb-index-external now goes beyond one page per document: it also
extracts people, organizations, projects, decisions, systems, and
processes each document discusses into thin, pointer-style entity
pages within the connector's own index (evidence back to source
documents, deferring to a full wiki/entities/ page where one already
exists), so the index supports "what do we know about X" lookups, not
just "what documents exist here".

New ckb-retrieve skill formalizes the retrieval half of the query
workflow: before grounding an answer in a page's tldr, follow it back
to its underlying source (a wiki page's Sources citations, or a
connector page's resource: pointer) rather than trusting the
compressed index entry as settled.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
This commit is contained in:
Michał Kopeć 2026-07-22 08:55:12 +02:00
parent f3dbce7327
commit 4e70a63ca4
2 changed files with 314 additions and 29 deletions
Show stats Download patch file Download diff file Expand all files Collapse all files

165
.agents/skills/ckb-index-external/SKILL.md
View file

@ -1,6 +1,6 @@
---
name: ckb-index-external
description: Walk every connector-backed libs/<name>/ (identified by a source.yaml declaring a live external source — SharePoint, Google Drive, a plain URL, or another connector) and build/refresh a self-contained generated index for it — index.md/entities/graph/log.md, mirroring wiki/'s own shape but scoped entirely to that one connector. The index can optionally be published to (and fetched from) a shared location — a git repo or a shared resource — declared in source.yaml, so most users just read an already-built index instead of re-scanning the live source. Read vs. write access to a given source is a per-user, local-only setting (libs/<name>/source.local.yaml, gitignored) — read-only by default, so one or two designated admins can maintain a source for everyone else. Never touches wiki/, never touches source.yaml, never touches a git-copy lib. Use when the user says "index external sources", "index libs", "refresh the external index", or "scan the SharePoint/Drive folder". This is the on-demand workflow from CLAUDE.md/AGENTS.md §4, distinct from `ckb-ingest` (which processes raw/inbox/ into wiki/) and from a plain git-copy libs/<name>/ (a frozen clone, never touched by any skill).
description: Walk every connector-backed libs/<name>/ (identified by a source.yaml declaring a live external source — SharePoint, Google Drive, a plain URL, or another connector) and build/refresh a self-contained generated index for it — index.md/entities/graph/log.md, mirroring wiki/'s own shape but scoped entirely to that one connector. Indexing goes beyond one page per document: for every document found, it also extracts the people, organizations, projects, decisions, systems, and processes that document actually discusses into their own thin, pointer-style entity pages (evidence back to the specific documents that mention them, not a second copy of wiki/'s synthesis), so the index supports "what do we know about X / what's the process for Y" lookups, not just "what documents exist at this source". The index can optionally be published to (and fetched from) a shared location — a git repo or a shared resource — declared in source.yaml, so most users just read an already-built index instead of re-scanning the live source. Read vs. write access to a given source is a per-user, local-only setting (libs/<name>/source.local.yaml, gitignored) — read-only by default, so one or two designated admins can maintain a source for everyone else. Never touches wiki/, never touches source.yaml, never touches a git-copy lib. Use when the user says "index external sources", "index libs", "refresh the external index", or "scan the SharePoint/Drive folder". This is the on-demand workflow from CLAUDE.md/AGENTS.md §4, distinct from `ckb-ingest` (which processes raw/inbox/ into wiki/) and from a plain git-copy libs/<name>/ (a frozen clone, never touched by any skill).
---
# Index external sources skill
@ -12,16 +12,33 @@ git-copy one: a `libs/<name>/source.yaml` declares a *live* external
source — a SharePoint folder, a Google Drive folder, a plain URL, or
another connector — that the user doesn't want to fully mirror locally.
This skill walks that source and builds a small, self-contained index of
what's there: one entity page per document found, plus the same
`index.md`/`graph`/`log.md` shape `wiki/` already uses, rooted at
`libs/<name>/` instead.
what's there, using the same `index.md`/`graph`/`log.md` shape `wiki/`
already uses, rooted at `libs/<name>/` instead.
The index holds two kinds of pages, and both matter for retrieval:
- **Document pages** — one per source item found (one per email, one per
file, one per SharePoint page), the original design. These answer "what
exists at this source".
- **Entity and process pages** — one per person, organization, project,
decision, system, or named process that those documents actually
*discuss*, extracted the same way `ckb-ingest` extracts entities into
`wiki/`, but kept deliberately thin here: a `tldr` plus a list of which
document(s) mention it and what they say, not a full synthesis. These
answer "what do we know about X" without forcing whoever's asking to
read every document's `tldr` by hand looking for a name. A document-only
index is fine for browsing a source but too narrow for actually
retrieving information out of it — that's what this second pass fixes.
This is deliberately **not** blended into the main `wiki/entities/` or
`wiki/graph/edges.json` — the generated index lives entirely inside its own
`libs/<name>/`, at the lowest cascade layer, the same way a git-cloned
KB's own files would. If something in it needs to override or correct what
the connector says, that's what writing the corrected version into `wiki/`
is for (cascade priority already covers that — `wiki/` always wins).
is for (cascade priority already covers that — `wiki/` always wins). Where
an extracted entity already has a full page in `wiki/entities/`, this
skill's own page for it stays thin and points there rather than
re-synthesizing — see Step 6.
Two refinements on top of that base design:
@ -43,7 +60,7 @@ Two refinements on top of that base design:
This skill only ever writes within a connector-backed `libs/<name>/` (never
`source.yaml` itself, never a git-copy lib, never anything under `wiki/`)
— and it never touches this repo's own git remote; pushing the resulting
changes is a separate, explicit step (see Step 7).
changes is a separate, explicit step (see Step 8).
## Trigger phrases
@ -60,9 +77,13 @@ Use this skill when the user says:
the same turn unless the user also asks for one.
Do **not** use this skill for "Ingest" / "Sync the wiki" (that's
`ckb-ingest` — raw material into `wiki/`, unrelated to `libs/`) or for
`ckb-ingest` — raw material into `wiki/`, unrelated to `libs/`), for
"Lint" (that's `ckb-lint`, which now also health-checks a connector's own
generated index, but doesn't build it in the first place).
generated index, but doesn't build it in the first place), or for
answering a question against an index that already exists (that's
`ckb-retrieve`, which reads what this skill built and always verifies
against the underlying source before answering — this skill never answers
questions, only builds/refreshes the index).
## Configuration
@ -159,7 +180,7 @@ This is the one place access level actually changes behavior:
- **Write access**: always continue to Step 5, whether Step 3 fetched an
existing index (refresh it) or found the store empty (build the very
first version from scratch) — the two cases are handled identically
from here on; Step 7 is what actually creates the remote copy either
from here on; Step 8 is what actually creates the remote copy either
way.
### Step 5 — Resolve the connector and enumerate/summarize documents
@ -181,8 +202,8 @@ For each document found at the location:
- Read enough content to write a real 1-2 sentence `tldr` when the
connector allows fetching content.
- If only metadata is available (listing only, no read access), say so
honestly in the `tldr` (e.g. "Metadata only — read access not granted")
and set `confidence` low (e.g. 0.3) rather than fabricating a summary.
honestly in the `tldr` (e.g. "Metadata only — content not readable") and
set `confidence` low (e.g. 0.3) rather than fabricating a summary.
Create/update one entity page per document at
`libs/<name>/entities/<doc-slug>.md`:
@ -199,7 +220,67 @@ retention: medium
---
```
### Step 6 — Update the hub page, graph, index, and log
### Step 6 — Extract entities and processes mentioned within each document
A document-only index tells a reader *what exists*, but not *what's known
about a given person, topic, or procedure* without reading every `tldr` by
hand. Close that gap the same way `ckb-ingest` extracts entities into
`wiki/`, but keep the result scoped and thin — this connector's own layer
is a retrieval index into its source documents, not a second knowledge
base competing with `wiki/`'s synthesis.
For each document processed in Step 5, identify what it actually
*discusses*, not just what it *is*: named people, organizations, projects,
decisions, systems, and processes/procedures (e.g. "VDI access request
process", "candidate profile approval") mentioned in its content. Skip
this for a document that's purely about itself with nothing else
extractable (a bare calendar acceptance with no discussion, for instance)
— not every document yields additional pages.
For each distinct entity/process found across the source's documents,
create or update one page at `libs/<name>/entities/<entity-slug>.md`,
using the same slug convention `wiki/entities/` would use for that name
(so `damien-gultig.md`, not a date-prefixed slug — this is what keeps
entity-page filenames from colliding with document-page filenames, which
are always date-prefixed per Step 5):
```yaml
---
type: person | organization | project | decision | system | concept | ... # same open field as wiki/'s schema
resource: /wiki/entities/<slug>.md # OPTIONAL — only if a full page for this entity already exists in wiki/
tldr: One sentence — who/what this is, and what these documents specifically show (not a full biography;
if wiki/ already has a full page, this tldr should say what these documents add, not restate it)
confidence: 0.3–0.9
quality: ...
last_updated: YYYY-MM-DD
freshness_window_days: 30
retention: medium
---
```
Body: a short "Mentioned in" list, one bullet per document that discusses
this entity, linking to that document's own page
(`entities/<doc-slug>.md`) with a one-line note of what that specific
document says — enough for `ckb-retrieve` (or a human) to know exactly
which document to open for the full context, without needing to re-derive
it from scratch.
**If this entity already has a full page in `wiki/entities/`** (check the
cascade first, same as `ckb-ingest`'s Step 2), set `resource:` to that
page's path and keep this page's body to just the "Mentioned in" list —
don't re-synthesize what the wiki page already says. The wiki page remains
authoritative (cascade priority); this page's only job is pointing back to
*these specific documents* as additional evidence, which the wiki page may
not have cited yet.
Record real relationships surfaced by a document between two entities
(`uses`, `depends_on`, `caused`, `contradicts`) as edges in
`libs/<name>/graph/edges.json`, same edge vocabulary `wiki/` uses. Also add
a `mentioned_in` edge from each entity to every document that discusses
it — this is what makes the graph a genuine index into the source
documents rather than just a bag of loose pages.
### Step 7 — Update the hub page, entities index, graph, and log
Create/update `libs/<name>/index.md` — the root routing page for this
connector, mirroring `wiki/index.md`'s own role:
@ -207,26 +288,35 @@ connector, mirroring `wiki/index.md`'s own role:
---
type: external-source
resource: <location, from source.yaml>
tldr: One sentence — what this source is and how many documents were found
tldr: One sentence — what this source is, how many documents were found, and how many entities/processes were extracted from them
last_updated: YYYY-MM-DD
---
```
followed by a short routing table pointing to `entities/index.md` and
`graph/index.md`.
Record any real relationships between documents (e.g. folder hierarchy) as
edges in `libs/<name>/graph/edges.json`, using a `contains`/`part_of`-style
edge type — this is fine to be thin or empty for a flat source with no
useful structure beyond a document list.
Update `libs/<name>/entities/index.md` as two sections, so both kinds of
page stay easy to find without conflating them:
```markdown
## Documents
- [<doc-slug>](<doc-slug>.md) — <tldr>
...
Update `libs/<name>/entities/index.md` and `libs/<name>/graph/index.md`
(flat bullet lists, no frontmatter, same convention as `wiki/entities/
index.md`/`wiki/graph/index.md`). Log every created/updated page in
`libs/<name>/log.md`, same reverse-chronological format as Rule B
(`CLAUDE.md`/`AGENTS.md` §7) — this log is independent of `wiki/log.md`;
**nothing under `wiki/` is touched by this skill at all.**
## Entities & Processes
- [<entity-slug>](<entity-slug>.md) — <tldr>
...
```
Same flat-bullet, no-frontmatter convention as `wiki/entities/index.md`
otherwise — this is just a two-heading variant of it, not a new format.
### Step 7 — Publish, if this source has both write access and a configured index store
Update `libs/<name>/graph/index.md` (same convention as
`wiki/graph/index.md`) to summarize both the `mentioned_in` document
coverage and any real entity-to-entity edges found. Log every
created/updated page in `libs/<name>/log.md`, same reverse-chronological
format as Rule B (`CLAUDE.md`/`AGENTS.md` §7) — this log is independent of
`wiki/log.md`; **nothing under `wiki/` is touched by this skill at all.**
### Step 8 — Publish, if this source has both write access and a configured index store
If this user has `access: write` **and** `source.yaml` has an `index:`
block, push the refreshed `libs/<name>/{index.md,entities/,graph/,log.md}`
@ -249,7 +339,7 @@ nothing is even rebuilt in `libs/<name>/` to begin with). A read-only user
has, by construction, nothing of their own to save back — Step 4 already
stopped them before anything was built.
### Step 8 — Remind to review and sync
### Step 9 — Remind to review and sync
This is always the last step, every time this skill makes any change at
all. Close with a short reminder — do not sync or push anything yourself
@ -285,18 +375,35 @@ was read-only with nothing new to fetch), skip this reminder.
the whole run; don't let a transient network issue block a read-only
user from seeing the last-known index.
- **Re-running against an already-indexed source** — refresh existing
entity pages in place (update `tldr`/`last_updated`/`confidence`); never
duplicate a document's page.
document *and* entity/process pages in place (update
`tldr`/`last_updated`/`confidence`, add newly-seen `mentioned_in` edges);
never duplicate a document's or entity's page.
- **A previously-indexed document is no longer found at the source** —
don't delete its page. Update its `tldr` with a note ("No longer found
at source as of YYYY-MM-DD") and set `retention: low`, so the existing
`ckb-lint` retention sweep archives it naturally on a later pass — no new
archival mechanism needed.
archival mechanism needed. Leave any entity pages that cite it alone;
the citation is still historically accurate even if the document itself
is gone.
- **An extracted entity already has a full page in `wiki/entities/`** —
keep this connector-side page thin (evidence/`mentioned_in` pointers
only, `resource:` set to the wiki page) rather than re-deriving
everything the wiki page already says; that duplication is exactly what
the cascade design is meant to avoid.
- **A document is dense with names/topics and extracting every one would
produce dozens of near-duplicate pages** — extract what's clearly named
and substantively discussed (not every passing mention), and prefer
updating an existing entity page's "Mentioned in" list over creating a
marginal new one. Don't force artificial granularity just to maximize
page count.
- **A document yields no extractable entity beyond itself** — that's fine;
its Step 5 document page is the only page it produces. Not every
document needs to feed Step 6.
- **A source is very large** — cap what gets fully fetched/summarized in
one run and explicitly report what was skipped (e.g. "42 of ~300
documents summarized this pass — re-run to continue"). Never silently
truncate while implying full coverage.
- **Write access but publishing (Step 7) conflicts with a newer version
- **Write access but publishing (Step 8) conflicts with a newer version
someone else already pushed** — this is why Step 3 (fetch) always runs
first, even for write-access users: rebuild on top of the latest fetched
state rather than blindly overwriting it. If a real conflict still shows

178
.agents/skills/ckb-retrieve/SKILL.md Normal file
View file

@ -0,0 +1,178 @@
---
name: ckb-retrieve
description: Governs how a question actually gets answered from the knowledge base. An index entry — a wiki page's `tldr`, a connector-index document/entity page — is deliberately compressed so lazy-loading stays cheap; that compression also means it can be incomplete, paraphrased loosely, or stale relative to the real source. This skill makes "go verify against the source before answering" a standing step, not an optional last resort: every time a page surfaced by the index looks relevant enough to actually ground part of the answer, follow it to its underlying source material — a wiki page's own `## Sources` citations into `raw/archive/`/`outputs/`, or a connector-index page's `resource:` pointer back to the live connector item — before treating its content as settled. Use whenever answering a question or researching a topic from `wiki/`, `linked/<name>/`, or a connector-backed `libs/<name>/` — this is the retrieval half of CLAUDE.md/AGENTS.md §5 QUERY WORKFLOW, always in play, not something the user needs to name explicitly. Distinct from `ckb-onboard-me` (produces a reading-order tour, not an answer to a specific question) and from the write-side skills `ckb-ingest`/`ckb-index-external`/`ckb-lint` (build or repair the index; this skill only ever reads it, plus the sources behind it).
---
# Retrieve (source-verified query) skill
## Purpose
The whole point of `tldr`/lazy-loading (CLAUDE.md/AGENTS.md §1
"Lazy-Loading with 'Use When' Triggers", §5 "QUERY WORKFLOW") is that most
of the wiki never has to enter context — a one-sentence summary decides
whether a page is worth opening at all. That's the right trade for
*deciding relevance*. It's the wrong trade for *grounding an answer*: a
`tldr` is a compression of whatever the page's author judged important at
write time, a connector-index entity page is (per `ckb-index-external`)
deliberately a thin pointer rather than a synthesis, and either can have
drifted from the primary material since — a source updated, a nuance
dropped, a paraphrase that's subtly wrong.
This skill closes that gap: once a page looks relevant enough to actually
use in an answer, don't stop at its `tldr` or even its full synthesized
body — follow it to the source material that page was built from, and
answer from there. It's the difference between citing what the index
*says about* the evidence and citing the evidence.
This skill is **read-only**. It never writes to `wiki/`, `libs/<name>/`,
or anywhere else — it only reads what already exists (the index, and the
source material behind it) to answer the question in front of it. If
verification surfaces a real gap or contradiction worth fixing in the
wiki, say so and suggest `ckb-lint`/`ckb-ingest` rather than editing
anything mid-answer.
## Trigger phrases
This is the default retrieval path for **any** question answered from the
knowledge base — the user does not need to name this skill. Typical
prompts that should route here:
- A direct question answerable from the wiki ("what do we know about X",
"what's the status of Y", "who owns Z").
- "Look up X" / "check the wiki for X" / "search for X".
- Mid-conversation moments where CLAUDE.md/AGENTS.md §5's QUERY WORKFLOW
applies — this skill *is* that workflow's implementation.
Do **not** use this skill for:
- Building or refreshing an index in the first place — that's
`ckb-ingest` (raw material → `wiki/`) or `ckb-index-external`
(connector → `libs/<name>/`). This skill only ever consumes what those
produced.
- A guided reading order across many pages on a topic — that's
`ckb-onboard-me`. This skill answers one question with verified
evidence; it doesn't produce a curriculum.
- Health-checking the index's own structure (frontmatter, staleness,
broken links) — that's `ckb-lint`. Verification failures found here
(Edge cases, below) are worth mentioning to the user as a possible lint
finding, but this skill doesn't run the lint checklist itself.
## How to run this skill
### Step 1 — Read the index
Same cascade order as CLAUDE.md/AGENTS.md §5, first match wins:
1. `wiki/index.md` — match the question against the **Use when** column.
2. `wiki/entities/index.md` — match against entity titles/`tldr`.
3. If nothing local matches: each `linked/<name>/` index (alphabetical),
then each connector-backed `libs/<name>/entities/index.md` — for a
connector-backed lib this means its *generated* index (both the
Documents and the Entities & Processes sections `ckb-index-external`
produces), never the live source directly. If it isn't built yet,
suggest "index external sources" rather than querying the live
connector ad hoc from inside this skill.
### Step 2 — Shortlist every page that looks relevant
Read the `tldr` of every page the index match surfaced — don't stop at
the first plausible hit. A question is often best answered by
triangulating two or three pages (e.g. a concept page plus the specific
person/decision page that qualifies it), and a page whose `tldr` looks
only tangential can still be hiding the exact fact needed in its body or
its Sources. Keep the shortlist to what's plausibly relevant; this isn't
"open everything," it's "don't stop at one."
### Step 3 — Walk the graph for anything the shortlist missed
If a shortlisted page has edges in `wiki/graph/edges.json` (or a
connector's own `graph/edges.json`), follow `depends_on`/`uses`/`caused`
edges one hop out — the same way §5 Step 4 already directs — to catch a
connected page the index text match alone wouldn't have surfaced.
### Step 4 — Follow every page on the shortlist to its source before answering from it
This is the step this skill exists to enforce. For each page on the
shortlist that will actually ground part of the answer:
- **A `wiki/` page** — every page synthesized via `ckb-ingest` carries a
`## Sources` section citing the exact `raw/archive/<date>/...` or
`outputs/...` file(s) it was built from. Open the cited file(s) — or
the specific section of a large one — and confirm the wiki's claim
matches what the primary material actually says. This also often
surfaces adjacent detail the synthesis compressed away that's directly
useful for the current question.
- **A connector-index page in `libs/<name>/`** (a document page, or an
entity/process page from the `ckb-index-external` extraction) — read
its `resource:` field. If the connector is authorized this session (per
`ToolSearch`/MCP auth state), re-fetch the live item for current,
complete content rather than trusting the cached `tldr` — connector
pages use a deliberately short `freshness_window_days` (30, vs. a
typical wiki page's 60-90) precisely because external sources drift
without notice. If the connector isn't authorized, or the item is a
calendar acceptance / metadata-only page with nothing more to fetch,
say plainly that the answer relies on the cached index rather than a
live re-check, so the user can weigh that.
- **A `linked/<name>/` page** — this is already full mirrored content,
not a summary; reading the page itself already is reading the source.
No extra fetch needed unless *that* page itself cites something further
outside the mirror.
Target the specific claim, not the whole file — a large transcript or
deck doesn't need a full read every time, just enough (search for the
name/topic, read the surrounding context) to confirm the point actually
being used.
### Step 5 — Reconcile and answer
If the source confirms the index, answer normally — but note what
grounded it (e.g. "per the kickoff transcript cited in
`gt-fde-access-requirements.md`") rather than presenting the answer as if
sourced from the `tldr` alone.
If the source contradicts, extends, or is more precise than what the
index said, prefer the source for the answer and say so explicitly — this
is exactly the kind of drift `ckb-lint`'s freshness/confidence checks
exist to eventually catch, so mention it's worth a lint pass if the gap
looks like more than a one-off, but don't rewrite the wiki page yourself
mid-answer unless the user asks for that separately.
If no page anywhere is even plausibly relevant, say so — don't verify
against a source that has nothing to do with the question just to appear
thorough, and don't fabricate an answer either.
## Edge cases
- **A cited source file no longer exists** (moved, renamed, or cleaned
up) — say so plainly rather than silently falling back to the `tldr` as
if it had been verified. Flag it as a likely `ckb-lint` finding (a
broken `Sources` reference) rather than treating it as a dead end.
- **Connector item unreachable or connector not authorized this
session** — state clearly that the answer is based on the cached
connector index, not a live re-check, and name which connector would
need authorizing to go further.
- **Question is low-stakes and the matched page is high-confidence,
recently updated, and well within its `freshness_window_days`** — still
open the source at least once to ground the answer; skip only a *second*
redundant read if the same source has already been opened earlier in
the same retrieval pass for another part of the same question.
- **Many pages are relevant and reading every cited source in full would
be excessive** — prioritize the sources that actually carry the
specific fact the question needs, not every source any shortlisted page
has ever cited. Say what was skipped rather than silently narrowing
without a note.
- **The "source" is itself a generated artifact** (e.g. an
`outputs/emails/*.md` draft cited as a wiki page's Source) rather than
raw external material — that's still the source relative to the wiki
page; no further hop beyond it is required.
- **A connector-index entity/process page's `resource:` points at a full
`wiki/entities/` page** (per `ckb-index-external`'s thin-page
convention) — treat the wiki page as the real source to verify against;
the connector page is just the pointer that got you there.
- **Nothing in the index is even plausibly relevant** — say the knowledge
base has nothing on this yet, and suggest `ckb-ingest` (for new raw
material) or `ckb-index-external` (for a connector-backed source) if
that seems like the actual gap. Don't stretch a weak match into an
answer just to have one.
---
*Licensed under the Apache License, Version 2.0 — see [LICENSE](../../../LICENSE)
at the repository root, or <http://www.apache.org/licenses/LICENSE-2.0>.*