--- name: ckb-index-external description: Walk every connector-backed libs// (identified by a source.yaml declaring a live external source — SharePoint, Google Drive, a plain URL, or another connector) and build/refresh a self-contained generated index for it — index.md/entities/graph/log.md, mirroring wiki/'s own shape but scoped entirely to that one connector. Indexing goes beyond one page per document: for every document found, it also extracts the people, organizations, projects, decisions, systems, and processes that document actually discusses into their own thin, pointer-style entity pages (evidence back to the specific documents that mention them, not a second copy of wiki/'s synthesis), so the index supports "what do we know about X / what's the process for Y" lookups, not just "what documents exist at this source". The index can optionally be published to (and fetched from) a shared location — a git repo or a shared resource — declared in source.yaml, so most users just read an already-built index instead of re-scanning the live source. Read vs. write access to a given source is a per-user, local-only setting (libs//source.local.yaml, gitignored) — read-only by default, so one or two designated admins can maintain a source for everyone else. Never touches wiki/, never touches source.yaml, never touches a git-copy lib. Use when the user says "index external sources", "index libs", "refresh the external index", or "scan the SharePoint/Drive folder". This is the on-demand workflow from CLAUDE.md/AGENTS.md §4, distinct from `ckb-ingest` (which processes raw/inbox/ into wiki/) and from a plain git-copy libs// (a frozen clone, never touched by any skill). --- # Index external sources skill ## Purpose Give `libs//` a second population mode alongside the existing git-copy one: a `libs//source.yaml` declares a *live* external source — a SharePoint folder, a Google Drive folder, a plain URL, or another connector — that the user doesn't want to fully mirror locally. This skill walks that source and builds a small, self-contained index of what's there, using the same `index.md`/`graph`/`log.md` shape `wiki/` already uses, rooted at `libs//` instead. The index holds two kinds of pages, and both matter for retrieval: - **Document pages** — one per source item found (one per email, one per file, one per SharePoint page), the original design. These answer "what exists at this source". - **Entity and process pages** — one per person, organization, project, decision, system, or named process that those documents actually *discuss*, extracted the same way `ckb-ingest` extracts entities into `wiki/`, but kept deliberately thin here: a `tldr` plus a list of which document(s) mention it and what they say, not a full synthesis. These answer "what do we know about X" without forcing whoever's asking to read every document's `tldr` by hand looking for a name. A document-only index is fine for browsing a source but too narrow for actually retrieving information out of it — that's what this second pass fixes. This is deliberately **not** blended into the main `wiki/entities/` or `wiki/graph/edges.json` — the generated index lives entirely inside its own `libs//`, at the lowest cascade layer, the same way a git-cloned KB's own files would. If something in it needs to override or correct what the connector says, that's what writing the corrected version into `wiki/` is for (cascade priority already covers that — `wiki/` always wins). Where an extracted entity already has a full page in `wiki/entities/`, this skill's own page for it stays thin and points there rather than re-synthesizing — see Step 6. Two refinements on top of that base design: - **Shared, pre-built indexes.** `source.yaml` can declare *where the already-built index lives* (a git repo, or a shared resource such as a network path or another connector-reachable location). Every run checks that location: if it already has an index, fetch it; if it doesn't yet, a write-access user's run is what creates it there for the first time. Either way, a read-only user just gets whatever's already there — they never have to build the index from scratch themselves. - **Read vs. write access, per user, per source.** Building/refreshing an index from the live connector is gated by a **local, per-user** setting (`libs//source.local.yaml`, never committed) — read-only by default. This lets a team designate one or two people as the "admin" of a given external source, while everyone else just consumes whatever index the admin has already published, without needing their own connector authorization or triggering redundant/conflicting rebuilds. This skill only ever writes within a connector-backed `libs//` (never `source.yaml` itself, never a git-copy lib, never anything under `wiki/`) — and it never touches this repo's own git remote; pushing the resulting changes is a separate, explicit step (see Step 8). ## Trigger phrases Use this skill when the user says: - "Index external sources" - "Index libs" - "Refresh the external index" - "Scan the SharePoint/Drive folder" (or similarly names a specific connector) - "Make me the admin for ``" / "give me write access to ``" / "I don't want write access to `` anymore" — a lighter-weight request that only touches that one source's `source.local.yaml` (see Configuration below) and doesn't necessarily trigger a full re-index in the same turn unless the user also asks for one. Do **not** use this skill for "Ingest" / "Sync the wiki" (that's `ckb-ingest` — raw material into `wiki/`, unrelated to `libs/`), for "Lint" (that's `ckb-lint`, which now also health-checks a connector's own generated index, but doesn't build it in the first place), or for answering a question against an index that already exists (that's `ckb-retrieve`, which reads what this skill built and always verifies against the underlying source before answering — this skill never answers questions, only builds/refreshes the index). ## Configuration ### `libs//source.yaml` (shared, tracked in git, same for everyone) ```yaml connector: sharepoint # sharepoint | google_drive | web | custom — open string, not a fixed enum location: "https://contoso.sharepoint.com/sites/Finance/Shared Documents/Reports" description: "Finance team's shared reports folder" # optional index: # optional — omit entirely for the original default: the index lives only # here, in this repo, tracked by this repo's own git (nothing to fetch/publish) store: git # git | shared — open string, where the built index is published to / fetched from location: "https://github.com/org/finance-index-cache.git" # a git repo URL (store: git), or a # filesystem/network path or connector-reachable location (store: shared) ref: main # optional — branch, tag, or subpath hint within that store ``` ### `libs//source.local.yaml` (per-user, per-machine, **never committed**) ```yaml access: write # write | read — defaults to read if this file is absent or the field is omitted ``` This file is intentionally **not** carved out of `.gitignore` — it stays caught by the existing `libs/*/*` catch-all, exactly like any other unlisted file inside a connector-backed `libs//`. It's local machine/user state, the same category as a personal `.env` — never synced, never reviewed by anyone else, and it's what makes "read vs. write" a per-user decision rather than a shared one baked into `source.yaml`. Unlike `source.yaml`, this skill *may* create or edit `source.local.yaml` on this user's behalf — but only when they explicitly ask to become (or stop being) a given source's admin (e.g. "make me the admin for the finance reports source" / "I don't want write access to X anymore"), never as a side effect of just running "index external sources." ## How to run this skill ### Step 1 — Find connector-backed libs Scan `libs/*/` for a `source.yaml`. A `libs//` with one is connector-backed and in scope; a `libs//` without one is a git-copy lib and stays completely untouched by this skill. If no `libs//source.yaml` exists anywhere, tell the user there's nothing to index yet and how to add one (a `source.yaml` with `connector` and `location`), and stop — don't fabricate an index from nothing. ### Step 2 — Read the local access setting Read `libs//source.local.yaml` if it exists. `access: write` means this user is an admin for this one source; anything else (the field is `read`, or the file doesn't exist at all) means read-only. This check is **per connector-backed lib** — a user can be write-access for one source and read-only for another. ### Step 3 — Check the shared index store, and fetch it if it exists If `source.yaml` has an `index:` block, check whether `index.location` already has a published index (per `index.store`: a `git` remote with commits/a reachable ref, or a filesystem/network/`shared` location with files already in it): - **It exists** → fetch/pull it (`git pull`/clone for `git`, a plain file copy for a filesystem/network `shared` path, or the matching connector tool for a `shared` location that needs one) and use it to refresh `libs//{index.md,entities/,graph/,log.md}`. Do this **regardless of this user's access level** — reading the published index never requires write access, and it's exactly what lets a read-only user skip building anything themselves. - **It's empty / nothing published yet** → there's nothing to fetch. Don't treat this as an error; it's the normal first-time state before anyone with write access has run this. Continue to Step 4 — whether that turns into "create it" or "nothing to report" depends entirely on this user's access level. If no `index:` block is configured at all, skip this check — the index (if any exists yet) already lives directly in `libs//`, same as before. ### Step 4 — Decide whether to build or refresh from the live connector This is the one place access level actually changes behavior: - **Read-only** (the common case, and the default for anyone who hasn't set `access: write` locally): stop here, regardless of what Step 3 found. Never touch the live connector on a read-only user's behalf. - If Step 3 fetched something (or one already lived in `libs//` with no `index:` block): report what it covers. - If nothing exists anywhere yet (Step 3 found the shared store empty, or there's no `index:` block and `libs//` is empty too): say so plainly, and suggest asking whoever administers this source to run it, or setting `access: write` locally if this user is meant to be that admin. - **Write access**: always continue to Step 5, whether Step 3 fetched an existing index (refresh it) or found the store empty (build the very first version from scratch) — the two cases are handled identically from here on; Step 8 is what actually creates the remote copy either way. ### Step 5 — Resolve the connector and enumerate/summarize documents Resolve `connector` to whatever live tool is actually available this session: - `sharepoint` / a Microsoft-hosted location → search (`ToolSearch`) for a connected Microsoft 365 MCP tool. - `google_drive` → search for a connected Google Drive MCP tool. - `web` / `custom` / anything else with a plain URL → `WebFetch`. If the needed connector isn't authorized in this session (per the current MCP auth state), **don't fail the whole run** — tell the user which connector needs authorizing (via claude.ai connector settings, or `claude mcp`/`/mcp` for non-claude.ai servers), skip that source, and continue with any others. For each document found at the location: - Read enough content to write a real 1-2 sentence `tldr` when the connector allows fetching content. - If only metadata is available (listing only, no read access), say so honestly in the `tldr` (e.g. "Metadata only — content not readable") and set `confidence` low (e.g. 0.3) rather than fabricating a summary. Create/update one entity page per document at `libs//entities/.md`: ```yaml --- type: document resource: tldr: ... confidence: 0.3–0.9 # per the rule above quality: ... last_updated: YYYY-MM-DD freshness_window_days: 30 # shorter than a typical wiki page — external sources change without notice retention: medium --- ``` ### Step 6 — Extract entities and processes mentioned within each document A document-only index tells a reader *what exists*, but not *what's known about a given person, topic, or procedure* without reading every `tldr` by hand. Close that gap the same way `ckb-ingest` extracts entities into `wiki/`, but keep the result scoped and thin — this connector's own layer is a retrieval index into its source documents, not a second knowledge base competing with `wiki/`'s synthesis. For each document processed in Step 5, identify what it actually *discusses*, not just what it *is*: named people, organizations, projects, decisions, systems, and processes/procedures (e.g. "VDI access request process", "candidate profile approval") mentioned in its content. Skip this for a document that's purely about itself with nothing else extractable (a bare calendar acceptance with no discussion, for instance) — not every document yields additional pages. For each distinct entity/process found across the source's documents, create or update one page at `libs//entities/.md`, using the same slug convention `wiki/entities/` would use for that name (so `damien-gultig.md`, not a date-prefixed slug — this is what keeps entity-page filenames from colliding with document-page filenames, which are always date-prefixed per Step 5): ```yaml --- type: person | organization | project | decision | system | concept | ... # same open field as wiki/'s schema resource: /wiki/entities/.md # OPTIONAL — only if a full page for this entity already exists in wiki/ tldr: One sentence — who/what this is, and what these documents specifically show (not a full biography; if wiki/ already has a full page, this tldr should say what these documents add, not restate it) confidence: 0.3–0.9 quality: ... last_updated: YYYY-MM-DD freshness_window_days: 30 retention: medium --- ``` Body: a short "Mentioned in" list, one bullet per document that discusses this entity, linking to that document's own page (`entities/.md`) with a one-line note of what that specific document says — enough for `ckb-retrieve` (or a human) to know exactly which document to open for the full context, without needing to re-derive it from scratch. **If this entity already has a full page in `wiki/entities/`** (check the cascade first, same as `ckb-ingest`'s Step 2), set `resource:` to that page's path and keep this page's body to just the "Mentioned in" list — don't re-synthesize what the wiki page already says. The wiki page remains authoritative (cascade priority); this page's only job is pointing back to *these specific documents* as additional evidence, which the wiki page may not have cited yet. Record real relationships surfaced by a document between two entities (`uses`, `depends_on`, `caused`, `contradicts`) as edges in `libs//graph/edges.json`, same edge vocabulary `wiki/` uses. Also add a `mentioned_in` edge from each entity to every document that discusses it — this is what makes the graph a genuine index into the source documents rather than just a bag of loose pages. ### Step 7 — Update the hub page, entities index, graph, and log Create/update `libs//index.md` — the root routing page for this connector, mirroring `wiki/index.md`'s own role: ```yaml --- type: external-source resource: tldr: One sentence — what this source is, how many documents were found, and how many entities/processes were extracted from them last_updated: YYYY-MM-DD --- ``` followed by a short routing table pointing to `entities/index.md` and `graph/index.md`. Update `libs//entities/index.md` as two sections, so both kinds of page stay easy to find without conflating them: ```markdown ## Documents - [](.md) — ... ## Entities & Processes - [](.md) — ... ``` Same flat-bullet, no-frontmatter convention as `wiki/entities/index.md` otherwise — this is just a two-heading variant of it, not a new format. Update `libs//graph/index.md` (same convention as `wiki/graph/index.md`) to summarize both the `mentioned_in` document coverage and any real entity-to-entity edges found. Log every created/updated page in `libs//log.md`, same reverse-chronological format as Rule B (`CLAUDE.md`/`AGENTS.md` §7) — this log is independent of `wiki/log.md`; **nothing under `wiki/` is touched by this skill at all.** ### Step 8 — Publish, if this source has both write access and a configured index store If this user has `access: write` **and** `source.yaml` has an `index:` block, push the refreshed `libs//{index.md,entities/,graph/,log.md}` back out to `index.location` (`git push` for `store: git`, a file copy back for a filesystem/network `store: shared` path, or the matching connector tool for a `shared` location that needs one) — **every single run**, not just the first one, so every other user's next fetch (Step 3) picks up the update. This is exactly how a shared index gets created in the first place: if Step 3 found the store empty, this step's push is what populates it for the first time; there's no separate "initialize" action. If no `index:` block is configured, there's nothing external to publish; the refreshed files staying inside this repo's own `libs//` (tracked by this repo's own git) *is* the publish step, same as the original default behavior. If this user is **read-only**, this step never runs — nothing is ever pushed or copied to `index.location` (or, in the no-`index:`-block case, nothing is even rebuilt in `libs//` to begin with). A read-only user has, by construction, nothing of their own to save back — Step 4 already stopped them before anything was built. ### Step 9 — Remind to review and sync This is always the last step, every time this skill makes any change at all. Close with a short reminder — do not sync or push anything yourself here: > "External source indexing complete for `libs//` — see > `libs//log.md` for the details. Please review the changes, and > once you're happy with them, say 'sync changes' to commit and push them > to `origin`." If nothing was actually indexed or fetched (no connector-backed libs found, every source was skipped for lack of authorization, or every source was read-only with nothing new to fetch), skip this reminder. ## Edge cases - **No connector-backed `libs//` exists** — report that and explain how to add one; don't touch anything. - **`source.local.yaml` doesn't exist yet for a source** — treat as read-only; this is the correct default, not a misconfiguration worth flagging loudly. Mention once, briefly, how to become the admin (`access: write` in that file) if the user seems to expect a rebuild. - **Everyone is read-only and nothing has ever been published** — report that clearly rather than silently doing nothing; this is the one case worth flagging as a real gap (nobody can currently index this source). - **Connector not authorized this session** — report which one and how to authorize it (claude.ai connector settings / `claude mcp` / `/mcp`), skip that source, keep going with the rest. Never ask the user for tokens or callback URLs. - **Fetching a published index (Step 3) fails or the remote is unreachable** — report it and fall back to whatever's already cached locally in `libs//` from a previous fetch, rather than blocking the whole run; don't let a transient network issue block a read-only user from seeing the last-known index. - **Re-running against an already-indexed source** — refresh existing document *and* entity/process pages in place (update `tldr`/`last_updated`/`confidence`, add newly-seen `mentioned_in` edges); never duplicate a document's or entity's page. - **A previously-indexed document is no longer found at the source** — don't delete its page. Update its `tldr` with a note ("No longer found at source as of YYYY-MM-DD") and set `retention: low`, so the existing `ckb-lint` retention sweep archives it naturally on a later pass — no new archival mechanism needed. Leave any entity pages that cite it alone; the citation is still historically accurate even if the document itself is gone. - **An extracted entity already has a full page in `wiki/entities/`** — keep this connector-side page thin (evidence/`mentioned_in` pointers only, `resource:` set to the wiki page) rather than re-deriving everything the wiki page already says; that duplication is exactly what the cascade design is meant to avoid. - **A document is dense with names/topics and extracting every one would produce dozens of near-duplicate pages** — extract what's clearly named and substantively discussed (not every passing mention), and prefer updating an existing entity page's "Mentioned in" list over creating a marginal new one. Don't force artificial granularity just to maximize page count. - **A document yields no extractable entity beyond itself** — that's fine; its Step 5 document page is the only page it produces. Not every document needs to feed Step 6. - **A source is very large** — cap what gets fully fetched/summarized in one run and explicitly report what was skipped (e.g. "42 of ~300 documents summarized this pass — re-run to continue"). Never silently truncate while implying full coverage. - **Write access but publishing (Step 8) conflicts with a newer version someone else already pushed** — this is why Step 3 (fetch) always runs first, even for write-access users: rebuild on top of the latest fetched state rather than blindly overwriting it. If a real conflict still shows up at publish time, report it rather than force-pushing over it. - **`libs//` has both real content files and a `source.yaml`** — ambiguous and out of scope for this skill to resolve; report it and stop for that folder (this is the same case `ckb-lint`'s external-source config check flags). - **`source.yaml` is missing `connector` or `location`** — report it as malformed and skip that source; don't guess at intent. --- *Licensed under the Apache License, Version 2.0 — see [LICENSE](../../../LICENSE) at the repository root, or .*