ckb/.agents/skills/ckb-index-external/SKILL.md
Michał Kopeć bbfda5a65d Sync: local changes as of 2026-10-29T13:50:12Z
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-29 14:50:12 +01:00

18 KiB
Raw Blame History

name description
ckb-index-external Build or refresh the generated index (index.md/entities/graph/log.md) inside each connector-backed libs/<name>/ — one declaring a live external source via source.yaml, such as SharePoint, Google Drive, or a URL. Indexes both the documents found and the people, projects, decisions and processes those documents discuss. Use for "index external sources", "index libs", "refresh the external index", or "scan the SharePoint/Drive folder".

Index external sources skill

Purpose

The index lives entirely inside its own libs/<name>/, at the lowest cascade layer, in the same index.md/entities//graph//log.md shape as wiki/: one document page per source item, plus deliberately thin entity and process pages (a tldr and which documents say what) for the people, projects, decisions and systems those documents discuss. Where wiki/entities/ already has a full page, the thin page points there (Step 6); to correct the connector, write into wiki/ and let cascade priority win.

source.yaml may name a shared, pre-built index that every run fetches first, so a read-only user never builds from scratch; rebuilding from the live connector is gated per user by source.local.yaml (never committed). Writes only within that libs/<name>/, never source.yaml, a git-copy lib, wiki/, or the git remote; pushing is a separate step (Step 8).

Configuration

libs/<name>/source.yaml (shared, tracked in git, same for everyone)

connector: sharepoint       # sharepoint | google_drive | web | custom — open string, not a fixed enum
location: "https://contoso.sharepoint.com/sites/Finance/Shared Documents/Reports"
description: "Finance team's shared reports folder"   # optional

refresh_interval_days: 7    # optional — how often a write-access run should rebuild this source,
                             #   and the freshness_window_days stamped on its generated pages.
                             #   Omit to use the default of 30. Tune per source: a busy folder
                             #   that changes daily deserves a shorter window than a quarterly
                             #   reports archive that barely moves.

index:                      # optional — omit entirely for the original default: the index lives only
                             #   here, in this repo, tracked by this repo's own git (nothing to fetch/publish)
  store: git                # git | shared — open string, where the built index is published to / fetched from
  location: "https://github.com/org/finance-index-cache.git"   # a git repo URL (store: git), or a
                             #   filesystem/network path or connector-reachable location (store: shared)
  ref: main                 # optional — branch, tag, or subpath hint within that store

libs/<name>/source.local.yaml (per-user, per-machine, never committed)

access: write                # write | read — defaults to read if this file is absent or the field is omitted

This file is intentionally not carved out of .gitignore — it stays caught by the existing libs/*/* catch-all, exactly like any other unlisted file inside a connector-backed libs/<name>/. It's local machine/user state, the same category as a personal .env — never synced, never reviewed by anyone else, and it's what makes "read vs. write" a per-user decision rather than a shared one baked into source.yaml.

Unlike source.yaml, this skill may create or edit source.local.yaml on this user's behalf — but only when they explicitly ask to become (or stop being) a given source's admin (e.g. "make me the admin for the finance reports source" / "I don't want write access to X anymore"), never as a side effect of just running "index external sources."

Regeneration rule: ## Notes is never overwritten

Every page this skill writes is generated — a later run rebuilds it from the connector and the previous contents are gone. That makes this index the one place in the KB where a human annotation is most useful and most fragile: the person who knows that a document is superseded, or that a name in it refers to someone who has since left, has nowhere safe to write it down.

## Notes is that place. Before rewriting any page under libs/<name>/, read the existing file and lift its ## Notes section out verbatim; after regenerating everything above it, append it back byte-for-byte. Not reflowed, not summarized, not merged into the tldr, not "improved" — unchanged, including whitespace and any half-finished sentence.

Practically: ## Notes is the last section of every generated page, so a rebuild is "replace everything above the ## Notes heading". Create it empty on a page's first write so the affordance exists before anyone needs it:

## Notes

<!-- Yours. Never rewritten by any skill. -->

If a page somehow has content after ## Notes that this skill would otherwise generate, preserve the whole tail rather than guessing where the human part ends. Losing generated content costs one re-run; losing a person's annotation costs the thing itself.

How to run this skill

Step 1 — Find connector-backed libs

Scan libs/*/ for a source.yaml. A libs/<name>/ with one is connector-backed and in scope; a libs/<name>/ without one is a git-copy lib and stays completely untouched by this skill.

If no libs/<name>/source.yaml exists anywhere, tell the user there's nothing to index yet and how to add one (a source.yaml with connector and location), and stop — don't fabricate an index from nothing.

Step 2 — Read the local access setting

Read libs/<name>/source.local.yaml if it exists. access: write means this user is an admin for this one source; anything else (the field is read, or the file doesn't exist at all) means read-only. This check is per connector-backed lib — a user can be write-access for one source and read-only for another.

Step 3 — Check the shared index store, and fetch it if it exists

If source.yaml has an index: block, check whether index.location already has a published index (per index.store: a git remote with commits/a reachable ref, or a filesystem/network/shared location with files already in it):

  • It exists → fetch/pull it (git pull/clone for git, a plain file copy for a filesystem/network shared path, or the matching connector tool for a shared location that needs one) and use it to refresh libs/<name>/{index.md,entities/,graph/,log.md}. Do this regardless of this user's access level — reading the published index never requires write access, and it's exactly what lets a read-only user skip building anything themselves.
  • It's empty / nothing published yet → there's nothing to fetch. Don't treat this as an error; it's the normal first-time state before anyone with write access has run this. Continue to Step 4 — whether that turns into "create it" or "nothing to report" depends entirely on this user's access level.

If no index: block is configured at all, skip this check — the index (if any exists yet) already lives directly in libs/<name>/, same as before.

Step 4 — Decide whether to build or refresh from the live connector

This is the one place access level actually changes behavior:

  • Read-only (the common case, and the default for anyone who hasn't set access: write locally): stop here, regardless of what Step 3 found. Never touch the live connector on a read-only user's behalf.
    • If Step 3 fetched something (or one already lived in libs/<name>/ with no index: block): report what it covers.
    • If nothing exists anywhere yet (Step 3 found the shared store empty, or there's no index: block and libs/<name>/ is empty too): say so plainly, and suggest asking whoever administers this source to run it, or setting access: write locally if this user is meant to be that admin.
  • Write access: always continue to Step 5, whether Step 3 fetched an existing index (refresh it) or found the store empty (build the very first version from scratch) — the two cases are handled identically from here on; Step 8 is what actually creates the remote copy either way.

Either way, compare the existing index's newest last_updated against this source's refresh_interval_days (default 30) and say where it stands. A read-only user needs to know they're reading a copy that's three weeks past due so they can go ask the admin rather than quietly trusting it; a write-access user is about to rebuild anyway, but "this was 40 days stale" is worth saying, because a source that's routinely overdue is either configured with too tight an interval or has nobody actually maintaining it. Both are worth surfacing rather than silently correcting.

Step 5 — Resolve the connector and enumerate/summarize documents

Resolve connector to whatever live tool is actually available this session:

  • sharepoint / a Microsoft-hosted location → search (ToolSearch) for a connected Microsoft 365 MCP tool.
  • google_drive → search for a connected Google Drive MCP tool.
  • web / custom / anything else with a plain URL → WebFetch.

If the needed connector isn't authorized in this session (per the current MCP auth state), don't fail the whole run — tell the user which connector needs authorizing (via claude.ai connector settings, or claude mcp//mcp for non-claude.ai servers), skip that source, and continue with any others.

For each document found at the location:

  • Read enough content to write a real 1-2 sentence tldr when the connector allows fetching content.
  • If only metadata is available (listing only, no read access), say so honestly in the tldr (e.g. "Metadata only — content not readable") and set confidence low (e.g. 0.3) rather than fabricating a summary.

Create/update one entity page per document at libs/<name>/entities/<doc-slug>.md:

---
type: document
resource: <document's own URL/path>
tldr: ...
confidence: 0.3–0.9   # per the rule above
quality: ...
last_updated: YYYY-MM-DD
freshness_window_days: 30   # this source's refresh_interval_days, or 30 if unset — shorter than a
                             #   typical wiki page, because external sources change without notice
retention: medium
source_fingerprint: sha256:3f9a2c1e   # see below — what actually detects drift
source_checked: YYYY-MM-DD
---

Fingerprint every document page. A short freshness_window_days is a guess that a source might have moved; a fingerprint is a fact about whether it did. This matters far more here than in wiki/: an archived file under raw/archive/ is effectively immutable, while a SharePoint or Drive document is edited by other people with no notification to anyone.

Take the fingerprint from whatever the connector gives you, in this order of preference:

  1. A version identifier the connector already maintains — an ETag, a version, a content hash, a lastModifiedDateTime. Record it as etag:<value> / mtime:<iso8601>. This is free and needs no content fetch.
  2. Failing that, sha256 of the fetched content, recorded as sha256:<first 8 hex chars>.
  3. Metadata-only documents with neither: omit the field rather than inventing one, and leave confidence low as Step 5 already requires.

Set source_checked to the date the fingerprint was last confirmed — which is not always last_updated. A re-index that confirms a document unchanged advances source_checked and leaves last_updated alone; that distinction is exactly what tells a reader "this was verified yesterday" from "this was written six months ago and nobody has looked since".

Give every document page a trailing ## Notes section, created empty on first write (see Step 7's protection rule).

Step 6 — Extract entities and processes mentioned within each document

A document-only index tells a reader what exists, but not what's known about a given person, topic, or procedure without reading every tldr by hand. Close that gap the same way ckb-ingest extracts entities into wiki/, but keep the result scoped and thin — this connector's own layer is a retrieval index into its source documents, not a second knowledge base competing with wiki/'s synthesis.

For each document processed in Step 5, identify what it actually discusses, not just what it is: named people, organizations, projects, decisions, systems, and processes/procedures (e.g. "VDI access request process", "candidate profile approval") mentioned in its content. Skip this for a document that's purely about itself with nothing else extractable (a bare calendar acceptance with no discussion, for instance) — not every document yields additional pages.

For each distinct entity/process found across the source's documents, create or update one page at libs/<name>/entities/<entity-slug>.md, using the same slug convention wiki/entities/ would use for that name (so damien-gultig.md, not a date-prefixed slug — this is what keeps entity-page filenames from colliding with document-page filenames, which are always date-prefixed per Step 5):

---
type: person | organization | project | decision | system | concept | ...   # same open field as wiki/'s schema
resource: /wiki/entities/<slug>.md   # OPTIONAL — only if a full page for this entity already exists in wiki/
tldr: One sentence — who/what this is, and what these documents specifically show (not a full biography;
      if wiki/ already has a full page, this tldr should say what these documents add, not restate it)
confidence: 0.3–0.9
quality: ...
last_updated: YYYY-MM-DD
freshness_window_days: 30   # or this source's refresh_interval_days, if set
retention: medium
---

Body: a short "Mentioned in" list, one bullet per document that discusses this entity, linking to that document's own page (entities/<doc-slug>.md) with a one-line note of what that specific document says — enough for ckb-retrieve (or a human) to know exactly which document to open for the full context, without needing to re-derive it from scratch.

If this entity already has a full page in wiki/entities/ (check the cascade first, same as ckb-ingest's Step 2), set resource: to that page's path and keep this page's body to just the "Mentioned in" list — don't re-synthesize what the wiki page already says. The wiki page remains authoritative (cascade priority); this page's only job is pointing back to these specific documents as additional evidence, which the wiki page may not have cited yet.

Record real relationships surfaced by a document between two entities as edges in libs/<name>/graph/edges.json, using the same closed vocabulary wiki/graph/index.md defines (part_of, uses, depends_on, produces, configures, validates, implements, caused, contradicts, supersedes) — each verb answering the question listed there. Also add a mentioned_in edge from each entity to every document that discusses it — this is what makes the graph a genuine index into the source documents rather than just a bag of loose pages.

Step 7 — Update the hub page, entities index, graph, and log

Create/update libs/<name>/index.md — the root routing page for this connector, mirroring wiki/index.md's own role:

---
type: external-source
resource: <location, from source.yaml>
tldr: One sentence — what this source is, how many documents were found, and how many entities/processes were extracted from them
last_updated: YYYY-MM-DD
---

followed by a short routing table pointing to entities/index.md and graph/index.md.

Update libs/<name>/entities/index.md as two sections, so both kinds of page stay easy to find without conflating them:

## Documents
- [<doc-slug>](<doc-slug>.md) — <tldr>
...

## Entities & Processes
- [<entity-slug>](<entity-slug>.md) — <tldr>
...

Same flat-bullet, no-frontmatter convention as wiki/entities/index.md otherwise — this is just a two-heading variant of it, not a new format.

Update libs/<name>/graph/index.md (same convention as wiki/graph/index.md) to summarize both the mentioned_in document coverage and any real entity-to-entity edges found. Log every created/updated page in libs/<name>/log.md, same reverse-chronological format as Rule B in CLAUDE.md/AGENTS.md — this log is independent of wiki/log.md; nothing under wiki/ is touched by this skill at all.

Step 8 — Publish, if this source has both write access and a configured index store

If this user has access: write and source.yaml has an index: block, push the refreshed libs/<name>/{index.md,entities/,graph/,log.md} back out to index.location (git push for store: git, a file copy back for a filesystem/network store: shared path, or the matching connector tool for a shared location that needs one) — every single run, not just the first one, so every other user's next fetch (Step 3) picks up the update. This is exactly how a shared index gets created in the first place: if Step 3 found the store empty, this step's push is what populates it for the first time; there's no separate "initialize" action.

If no index: block is configured, there's nothing external to publish; the refreshed files staying inside this repo's own libs/<name>/ (tracked by this repo's own git) is the publish step, same as the original default behavior.

If this user is read-only, this step never runs — nothing is ever pushed or copied to index.location (or, in the no-index:-block case, nothing is even rebuilt in libs/<name>/ to begin with). A read-only user has, by construction, nothing of their own to save back — Step 4 already stopped them before anything was built.

Step 9 — Remind to review and sync

This is always the last step, every time this skill makes any change at all. Close with a short reminder — do not sync or push anything yourself here:

"External source indexing complete for libs/<name>/ — see libs/<name>/log.md for the details. Please review the changes, and once you're happy with them, say 'sync changes' to commit and push them to origin."

If nothing was actually indexed or fetched (no connector-backed libs found, every source was skipped for lack of authorization, or every source was read-only with nothing new to fetch), skip this reminder.

Edge cases

Unusual situations — an empty target, a partial run, a conflicting state, a missing dependency — are catalogued in references/edge-cases.md. Read that file when a run hits something the steps above don't cover; don't load it otherwise.