skills-marketplace/plugins/cascade-knowledge-base/skills/ckb-index-external/SKILL.md

---

name: ckb-index-external description: Walk every connector-backed libs// (identified by a source.yaml declaring a live external source — SharePoint, Google Drive, a plain URL, or another connector) and build/refresh a self-contained generated index for it — index.md/entities/graph/log.md, mirroring wiki/'s own shape but scoped entirely to that one connector. Indexing goes beyond one page per document: for every document found, it also extracts the people, organizations, projects, decisions, systems, and processes that document actually discusses into their own thin, pointer-style entity pages (evidence back to the specific documents that mention them, not a second copy of wiki/'s synthesis), so the index supports "what do we know about X / what's the process for Y" lookups, not just "what documents exist at this source". The index can optionally be published to (and fetched from) a shared location — a git repo or a shared resource — declared in source.yaml, so most users just read an already-built index instead of re-scanning the live source. Read vs. write access to a given source is a per-user, local-only setting (libs//source.local.yaml, gitignored) — read-only by default, so one or two designated admins can maintain a source for everyone else. Never touches wiki/, never touches source.yaml, never touches a git-copy lib. Use when the user says "index external sources", "index libs", "refresh the external index", or "scan the SharePoint/Drive folder". This is the on-demand external-source workflow routed by CLAUDE.md/AGENTS.md, distinct from ckb-ingest (which processes raw/inbox/ into wiki/) and from a plain git-copy libs// (a frozen clone, never touched by any skill).

Index external sources skill

Purpose

Give libs/<name>/ a second population mode alongside the existing git-copy one: a libs/<name>/source.yaml declares a live external source — a SharePoint folder, a Google Drive folder, a plain URL, or another connector — that the user doesn't want to fully mirror locally. This skill walks that source and builds a small, self-contained index of what's there, using the same index.md/graph/log.md shape wiki/ already uses, rooted at libs/<name>/ instead.

The index holds two kinds of pages, and both matter for retrieval:

• Document pages — one per source item found (one per email, one per file, one per SharePoint page), the original design. These answer "what exists at this source".
• Entity and process pages — one per person, organization, project, decision, system, or named process that those documents actually discuss, extracted the same way ckb-ingest extracts entities into wiki/, but kept deliberately thin here: a tldr plus a list of which document(s) mention it and what they say, not a full synthesis. These answer "what do we know about X" without forcing whoever's asking to read every document's tldr by hand looking for a name. A document-only index is fine for browsing a source but too narrow for actually retrieving information out of it — that's what this second pass fixes.

This is deliberately not blended into the main wiki/entities/ or wiki/graph/edges.json — the generated index lives entirely inside its own libs/<name>/, at the lowest cascade layer, the same way a git-cloned KB's own files would. If something in it needs to override or correct what the connector says, that's what writing the corrected version into wiki/ is for (cascade priority already covers that — wiki/ always wins). Where an extracted entity already has a full page in wiki/entities/, this skill's own page for it stays thin and points there rather than re-synthesizing — see Step 6.

Two refinements on top of that base design:

• Shared, pre-built indexes. source.yaml can declare where the already-built index lives (a git repo, or a shared resource such as a network path or another connector-reachable location). Every run checks that location: if it already has an index, fetch it; if it doesn't yet, a write-access user's run is what creates it there for the first time. Either way, a read-only user just gets whatever's already there — they never have to build the index from scratch themselves.
• Read vs. write access, per user, per source. Building/refreshing an index from the live connector is gated by a local, per-user setting (libs/<name>/source.local.yaml, never committed) — read-only by default. This lets a team designate one or two people as the "admin" of a given external source, while everyone else just consumes whatever index the admin has already published, without needing their own connector authorization or triggering redundant/conflicting rebuilds.

This skill only ever writes within a connector-backed libs/<name>/ (never source.yaml itself, never a git-copy lib, never anything under wiki/) — and it never touches this repo's own git remote; pushing the resulting changes is a separate, explicit step (see Step 8).

Trigger phrases

Use this skill when the user says:

• "Index external sources"
• "Index libs"
• "Refresh the external index"
• "Scan the SharePoint/Drive folder" (or similarly names a specific connector)
• "Make me the admin for <source>" / "give me write access to <source>" / "I don't want write access to <source> anymore" — a lighter-weight request that only touches that one source's source.local.yaml (see Configuration below) and doesn't necessarily trigger a full re-index in the same turn unless the user also asks for one.

Do not use this skill for "Ingest" / "Sync the wiki" (that's ckb-ingest — raw material into wiki/, unrelated to libs/), for "Lint" (that's ckb-lint, which now also health-checks a connector's own generated index, but doesn't build it in the first place), or for answering a question against an index that already exists (that's ckb-retrieve, which reads what this skill built and always verifies against the underlying source before answering — this skill never answers questions, only builds/refreshes the index).

Configuration

libs/<name>/source.yaml (shared, tracked in git, same for everyone)

connector: sharepoint       # sharepoint | google_drive | web | custom — open string, not a fixed enum
location: "https://contoso.sharepoint.com/sites/Finance/Shared Documents/Reports"
description: "Finance team's shared reports folder"   # optional

refresh_interval_days: 7    # optional — how often a write-access run should rebuild this source,
                             #   and the freshness_window_days stamped on its generated pages.
                             #   Omit to use the default of 30. Tune per source: a busy folder
                             #   that changes daily deserves a shorter window than a quarterly
                             #   reports archive that barely moves.

index:                      # optional — omit entirely for the original default: the index lives only
                             #   here, in this repo, tracked by this repo's own git (nothing to fetch/publish)
  store: git                # git | shared — open string, where the built index is published to / fetched from
  location: "https://github.com/org/finance-index-cache.git"   # a git repo URL (store: git), or a
                             #   filesystem/network path or connector-reachable location (store: shared)
  ref: main                 # optional — branch, tag, or subpath hint within that store

libs/<name>/source.local.yaml (per-user, per-machine, never committed)

access: write                # write | read — defaults to read if this file is absent or the field is omitted

This file is intentionally not carved out of .gitignore — it stays caught by the existing libs/*/* catch-all, exactly like any other unlisted file inside a connector-backed libs/<name>/. It's local machine/user state, the same category as a personal .env — never synced, never reviewed by anyone else, and it's what makes "read vs. write" a per-user decision rather than a shared one baked into source.yaml.

Unlike source.yaml, this skill may create or edit source.local.yaml on this user's behalf — but only when they explicitly ask to become (or stop being) a given source's admin (e.g. "make me the admin for the finance reports source" / "I don't want write access to X anymore"), never as a side effect of just running "index external sources."

How to run this skill

Step 1 — Find connector-backed libs

Scan libs/*/ for a source.yaml. A libs/<name>/ with one is connector-backed and in scope; a libs/<name>/ without one is a git-copy lib and stays completely untouched by this skill.

If no libs/<name>/source.yaml exists anywhere, tell the user there's nothing to index yet and how to add one (a source.yaml with connector and location), and stop — don't fabricate an index from nothing.

Step 2 — Read the local access setting

Read libs/<name>/source.local.yaml if it exists. access: write means this user is an admin for this one source; anything else (the field is read, or the file doesn't exist at all) means read-only. This check is per connector-backed lib — a user can be write-access for one source and read-only for another.

Step 3 — Check the shared index store, and fetch it if it exists

If source.yaml has an index: block, check whether index.location already has a published index (per index.store: a git remote with commits/a reachable ref, or a filesystem/network/shared location with files already in it):

• It exists → fetch/pull it (git pull/clone for git, a plain file copy for a filesystem/network shared path, or the matching connector tool for a shared location that needs one) and use it to refresh libs/<name>/{index.md,entities/,graph/,log.md}. Do this regardless of this user's access level — reading the published index never requires write access, and it's exactly what lets a read-only user skip building anything themselves.
• It's empty / nothing published yet → there's nothing to fetch. Don't treat this as an error; it's the normal first-time state before anyone with write access has run this. Continue to Step 4 — whether that turns into "create it" or "nothing to report" depends entirely on this user's access level.

If no index: block is configured at all, skip this check — the index (if any exists yet) already lives directly in libs/<name>/, same as before.

Step 4 — Decide whether to build or refresh from the live connector

This is the one place access level actually changes behavior:

• Read-only (the common case, and the default for anyone who hasn't set access: write locally): stop here, regardless of what Step 3 found. Never touch the live connector on a read-only user's behalf.
  • If Step 3 fetched something (or one already lived in libs/<name>/ with no index: block): report what it covers.
  • If nothing exists anywhere yet (Step 3 found the shared store empty, or there's no index: block and libs/<name>/ is empty too): say so plainly, and suggest asking whoever administers this source to run it, or setting access: write locally if this user is meant to be that admin.
• Write access: always continue to Step 5, whether Step 3 fetched an existing index (refresh it) or found the store empty (build the very first version from scratch) — the two cases are handled identically from here on; Step 8 is what actually creates the remote copy either way.

Either way, compare the existing index's newest last_updated against this source's refresh_interval_days (default 30) and say where it stands. A read-only user needs to know they're reading a copy that's three weeks past due so they can go ask the admin rather than quietly trusting it; a write-access user is about to rebuild anyway, but "this was 40 days stale" is worth saying, because a source that's routinely overdue is either configured with too tight an interval or has nobody actually maintaining it. Both are worth surfacing rather than silently correcting.

Step 5 — Resolve the connector and enumerate/summarize documents

Resolve connector to whatever live tool is actually available this session:

• sharepoint / a Microsoft-hosted location → search (ToolSearch) for a connected Microsoft 365 MCP tool.
• google_drive → search for a connected Google Drive MCP tool.
• web / custom / anything else with a plain URL → WebFetch.

If the needed connector isn't authorized in this session (per the current MCP auth state), don't fail the whole run — tell the user which connector needs authorizing (via claude.ai connector settings, or claude mcp//mcp for non-claude.ai servers), skip that source, and continue with any others.

For each document found at the location:

• Read enough content to write a real 1-2 sentence tldr when the connector allows fetching content.
• If only metadata is available (listing only, no read access), say so honestly in the tldr (e.g. "Metadata only — content not readable") and set confidence low (e.g. 0.3) rather than fabricating a summary.

Create/update one entity page per document at libs/<name>/entities/<doc-slug>.md:

---
type: document
resource: <document's own URL/path>
tldr: ...
confidence: 0.3–0.9   # per the rule above
quality: ...
last_updated: YYYY-MM-DD
freshness_window_days: 30   # this source's refresh_interval_days, or 30 if unset — shorter than a
                             #   typical wiki page, because external sources change without notice
retention: medium
---

Step 6 — Extract entities and processes mentioned within each document

A document-only index tells a reader what exists, but not what's known about a given person, topic, or procedure without reading every tldr by hand. Close that gap the same way ckb-ingest extracts entities into wiki/, but keep the result scoped and thin — this connector's own layer is a retrieval index into its source documents, not a second knowledge base competing with wiki/'s synthesis.

For each document processed in Step 5, identify what it actually discusses, not just what it is: named people, organizations, projects, decisions, systems, and processes/procedures (e.g. "VDI access request process", "candidate profile approval") mentioned in its content. Skip this for a document that's purely about itself with nothing else extractable (a bare calendar acceptance with no discussion, for instance) — not every document yields additional pages.

For each distinct entity/process found across the source's documents, create or update one page at libs/<name>/entities/<entity-slug>.md, using the same slug convention wiki/entities/ would use for that name (so damien-gultig.md, not a date-prefixed slug — this is what keeps entity-page filenames from colliding with document-page filenames, which are always date-prefixed per Step 5):

---
type: person | organization | project | decision | system | concept | ...   # same open field as wiki/'s schema
resource: /wiki/entities/<slug>.md   # OPTIONAL — only if a full page for this entity already exists in wiki/
tldr: One sentence — who/what this is, and what these documents specifically show (not a full biography;
      if wiki/ already has a full page, this tldr should say what these documents add, not restate it)
confidence: 0.3–0.9
quality: ...
last_updated: YYYY-MM-DD
freshness_window_days: 30   # or this source's refresh_interval_days, if set
retention: medium
---

Body: a short "Mentioned in" list, one bullet per document that discusses this entity, linking to that document's own page (entities/<doc-slug>.md) with a one-line note of what that specific document says — enough for ckb-retrieve (or a human) to know exactly which document to open for the full context, without needing to re-derive it from scratch.

If this entity already has a full page in wiki/entities/ (check the cascade first, same as ckb-ingest's Step 2), set resource: to that page's path and keep this page's body to just the "Mentioned in" list — don't re-synthesize what the wiki page already says. The wiki page remains authoritative (cascade priority); this page's only job is pointing back to these specific documents as additional evidence, which the wiki page may not have cited yet.

Record real relationships surfaced by a document between two entities (uses, depends_on, caused, contradicts) as edges in libs/<name>/graph/edges.json, same edge vocabulary wiki/ uses. Also add a mentioned_in edge from each entity to every document that discusses it — this is what makes the graph a genuine index into the source documents rather than just a bag of loose pages.

Step 7 — Update the hub page, entities index, graph, and log

Create/update libs/<name>/index.md — the root routing page for this connector, mirroring wiki/index.md's own role:

---
type: external-source
resource: <location, from source.yaml>
tldr: One sentence — what this source is, how many documents were found, and how many entities/processes were extracted from them
last_updated: YYYY-MM-DD
---

followed by a short routing table pointing to entities/index.md and graph/index.md.

Update libs/<name>/entities/index.md as two sections, so both kinds of page stay easy to find without conflating them:

## Documents
- [<doc-slug>](<doc-slug>.md) — <tldr>
...

## Entities & Processes
- [<entity-slug>](<entity-slug>.md) — <tldr>
...

Same flat-bullet, no-frontmatter convention as wiki/entities/index.md otherwise — this is just a two-heading variant of it, not a new format.

Update libs/<name>/graph/index.md (same convention as wiki/graph/index.md) to summarize both the mentioned_in document coverage and any real entity-to-entity edges found. Log every created/updated page in libs/<name>/log.md, same reverse-chronological format as Rule B in CLAUDE.md/AGENTS.md — this log is independent of wiki/log.md; nothing under wiki/ is touched by this skill at all.

Step 8 — Publish, if this source has both write access and a configured index store

If this user has access: write and source.yaml has an index: block, push the refreshed libs/<name>/{index.md,entities/,graph/,log.md} back out to index.location (git push for store: git, a file copy back for a filesystem/network store: shared path, or the matching connector tool for a shared location that needs one) — every single run, not just the first one, so every other user's next fetch (Step 3) picks up the update. This is exactly how a shared index gets created in the first place: if Step 3 found the store empty, this step's push is what populates it for the first time; there's no separate "initialize" action.

If no index: block is configured, there's nothing external to publish; the refreshed files staying inside this repo's own libs/<name>/ (tracked by this repo's own git) is the publish step, same as the original default behavior.

If this user is read-only, this step never runs — nothing is ever pushed or copied to index.location (or, in the no-index:-block case, nothing is even rebuilt in libs/<name>/ to begin with). A read-only user has, by construction, nothing of their own to save back — Step 4 already stopped them before anything was built.

Step 9 — Remind to review and sync

This is always the last step, every time this skill makes any change at all. Close with a short reminder — do not sync or push anything yourself here:

"External source indexing complete for libs/<name>/ — see libs/<name>/log.md for the details. Please review the changes, and once you're happy with them, say 'sync changes' to commit and push them to origin."

If nothing was actually indexed or fetched (no connector-backed libs found, every source was skipped for lack of authorization, or every source was read-only with nothing new to fetch), skip this reminder.

Edge cases

• No connector-backed libs/<name>/ exists — report that and explain how to add one; don't touch anything.
• source.local.yaml doesn't exist yet for a source — treat as read-only; this is the correct default, not a misconfiguration worth flagging loudly. Mention once, briefly, how to become the admin (access: write in that file) if the user seems to expect a rebuild.
• Everyone is read-only and nothing has ever been published — report that clearly rather than silently doing nothing; this is the one case worth flagging as a real gap (nobody can currently index this source).
• Connector not authorized this session — report which one and how to authorize it (claude.ai connector settings / claude mcp / /mcp), skip that source, keep going with the rest. Never ask the user for tokens or callback URLs.
• Fetching a published index (Step 3) fails or the remote is unreachable — report it and fall back to whatever's already cached locally in libs/<name>/ from a previous fetch, rather than blocking the whole run; don't let a transient network issue block a read-only user from seeing the last-known index.
• Re-running against an already-indexed source — refresh existing document and entity/process pages in place (update tldr/last_updated/confidence, add newly-seen mentioned_in edges); never duplicate a document's or entity's page.
• A previously-indexed document is no longer found at the source — don't delete its page. Update its tldr with a note ("No longer found at source as of YYYY-MM-DD") and set retention: low, so the existing ckb-lint retention sweep archives it naturally on a later pass — no new archival mechanism needed. Leave any entity pages that cite it alone; the citation is still historically accurate even if the document itself is gone.
• An extracted entity already has a full page in wiki/entities/ — keep this connector-side page thin (evidence/mentioned_in pointers only, resource: set to the wiki page) rather than re-deriving everything the wiki page already says; that duplication is exactly what the cascade design is meant to avoid.
• A document is dense with names/topics and extracting every one would produce dozens of near-duplicate pages — extract what's clearly named and substantively discussed (not every passing mention), and prefer updating an existing entity page's "Mentioned in" list over creating a marginal new one. Don't force artificial granularity just to maximize page count.
• A document yields no extractable entity beyond itself — that's fine; its Step 5 document page is the only page it produces. Not every document needs to feed Step 6.
• A source is very large — cap what gets fully fetched/summarized in one run and explicitly report what was skipped (e.g. "42 of ~300 documents summarized this pass — re-run to continue"). Never silently truncate while implying full coverage.
• Write access but publishing (Step 8) conflicts with a newer version someone else already pushed — this is why Step 3 (fetch) always runs first, even for write-access users: rebuild on top of the latest fetched state rather than blindly overwriting it. If a real conflict still shows up at publish time, report it rather than force-pushing over it.
• libs/<name>/ has both real content files and a source.yaml — ambiguous and out of scope for this skill to resolve; report it and stop for that folder (this is the same case ckb-lint's external-source config check flags).
• source.yaml is missing connector or location — report it as malformed and skip that source; don't guess at intent.

---

Licensed under the Apache License, Version 2.0 — see LICENSE at the repository root, or http://www.apache.org/licenses/LICENSE-2.0.