ckb/.agents/skills/ckb-index-external/SKILL.md

---
name: ckb-index-external
description: Walk every connector-backed libs/<name>/ (identified by a source.yaml declaring a live external source — SharePoint, Google Drive, a plain URL, or another connector) and build/refresh a self-contained generated index for it — index.md/entities/graph/log.md, mirroring wiki/'s own shape but scoped entirely to that one connector. Indexing goes beyond one page per document: for every document found, it also extracts the people, organizations, projects, decisions, systems, and processes that document actually discusses into their own thin, pointer-style entity pages (evidence back to the specific documents that mention them, not a second copy of wiki/'s synthesis), so the index supports "what do we know about X / what's the process for Y" lookups, not just "what documents exist at this source". The index can optionally be published to (and fetched from) a shared location — a git repo or a shared resource — declared in source.yaml, so most users just read an already-built index instead of re-scanning the live source. Read vs. write access to a given source is a per-user, local-only setting (libs/<name>/source.local.yaml, gitignored) — read-only by default, so one or two designated admins can maintain a source for everyone else. Never touches wiki/, never touches source.yaml, never touches a git-copy lib. Use when the user says "index external sources", "index libs", "refresh the external index", or "scan the SharePoint/Drive folder". This is the on-demand external-source workflow routed by CLAUDE.md/AGENTS.md, distinct from `ckb-ingest` (which processes raw/inbox/ into wiki/) and from a plain git-copy libs/<name>/ (a frozen clone, never touched by any skill).
---

# Index external sources skill

## Purpose

Give `libs/<name>/` a second population mode alongside the existing
git-copy one: a `libs/<name>/source.yaml` declares a *live* external
source — a SharePoint folder, a Google Drive folder, a plain URL, or
another connector — that the user doesn't want to fully mirror locally.
This skill walks that source and builds a small, self-contained index of
what's there, using the same `index.md`/`graph`/`log.md` shape `wiki/`
already uses, rooted at `libs/<name>/` instead.

The index holds two kinds of pages, and both matter for retrieval:

- **Document pages** — one per source item found (one per email, one per
  file, one per SharePoint page), the original design. These answer "what
  exists at this source".
- **Entity and process pages** — one per person, organization, project,
  decision, system, or named process that those documents actually
  *discuss*, extracted the same way `ckb-ingest` extracts entities into
  `wiki/`, but kept deliberately thin here: a `tldr` plus a list of which
  document(s) mention it and what they say, not a full synthesis. These
  answer "what do we know about X" without forcing whoever's asking to
  read every document's `tldr` by hand looking for a name. A document-only
  index is fine for browsing a source but too narrow for actually
  retrieving information out of it — that's what this second pass fixes.

This is deliberately **not** blended into the main `wiki/entities/` or
`wiki/graph/edges.json` — the generated index lives entirely inside its own
`libs/<name>/`, at the lowest cascade layer, the same way a git-cloned
KB's own files would. If something in it needs to override or correct what
the connector says, that's what writing the corrected version into `wiki/`
is for (cascade priority already covers that — `wiki/` always wins). Where
an extracted entity already has a full page in `wiki/entities/`, this
skill's own page for it stays thin and points there rather than
re-synthesizing — see Step 6.

Two refinements on top of that base design:

- **Shared, pre-built indexes.** `source.yaml` can declare *where the
  already-built index lives* (a git repo, or a shared resource such as a
  network path or another connector-reachable location). Every run checks
  that location: if it already has an index, fetch it; if it doesn't yet,
  a write-access user's run is what creates it there for the first time.
  Either way, a read-only user just gets whatever's already there — they
  never have to build the index from scratch themselves.
- **Read vs. write access, per user, per source.** Building/refreshing an
  index from the live connector is gated by a **local, per-user** setting
  (`libs/<name>/source.local.yaml`, never committed) — read-only by
  default. This lets a team designate one or two people as the "admin" of
  a given external source, while everyone else just consumes whatever
  index the admin has already published, without needing their own
  connector authorization or triggering redundant/conflicting rebuilds.

This skill only ever writes within a connector-backed `libs/<name>/` (never
`source.yaml` itself, never a git-copy lib, never anything under `wiki/`)
— and it never touches this repo's own git remote; pushing the resulting
changes is a separate, explicit step (see Step 8).

## Trigger phrases

Use this skill when the user says:
- "Index external sources"
- "Index libs"
- "Refresh the external index"
- "Scan the SharePoint/Drive folder" (or similarly names a specific
  connector)
- "Make me the admin for `<source>`" / "give me write access to `<source>`"
  / "I don't want write access to `<source>` anymore" — a lighter-weight
  request that only touches that one source's `source.local.yaml` (see
  Configuration below) and doesn't necessarily trigger a full re-index in
  the same turn unless the user also asks for one.

Do **not** use this skill for "Ingest" / "Sync the wiki" (that's
`ckb-ingest` — raw material into `wiki/`, unrelated to `libs/`), for
"Lint" (that's `ckb-lint`, which now also health-checks a connector's own
generated index, but doesn't build it in the first place), or for
answering a question against an index that already exists (that's
`ckb-retrieve`, which reads what this skill built and always verifies
against the underlying source before answering — this skill never answers
questions, only builds/refreshes the index).

## Configuration

### `libs/<name>/source.yaml` (shared, tracked in git, same for everyone)

```yaml
connector: sharepoint       # sharepoint | google_drive | web | custom — open string, not a fixed enum
location: "https://contoso.sharepoint.com/sites/Finance/Shared Documents/Reports"
description: "Finance team's shared reports folder"   # optional

refresh_interval_days: 7    # optional — how often a write-access run should rebuild this source,
                             #   and the freshness_window_days stamped on its generated pages.
                             #   Omit to use the default of 30. Tune per source: a busy folder
                             #   that changes daily deserves a shorter window than a quarterly
                             #   reports archive that barely moves.

index:                      # optional — omit entirely for the original default: the index lives only
                             #   here, in this repo, tracked by this repo's own git (nothing to fetch/publish)
  store: git                # git | shared — open string, where the built index is published to / fetched from
  location: "https://github.com/org/finance-index-cache.git"   # a git repo URL (store: git), or a
                             #   filesystem/network path or connector-reachable location (store: shared)
  ref: main                 # optional — branch, tag, or subpath hint within that store
```

### `libs/<name>/source.local.yaml` (per-user, per-machine, **never committed**)

```yaml
access: write                # write | read — defaults to read if this file is absent or the field is omitted
```

This file is intentionally **not** carved out of `.gitignore` — it stays
caught by the existing `libs/*/*` catch-all, exactly like any other
unlisted file inside a connector-backed `libs/<name>/`. It's local
machine/user state, the same category as a personal `.env` — never synced,
never reviewed by anyone else, and it's what makes "read vs. write" a
per-user decision rather than a shared one baked into `source.yaml`.

Unlike `source.yaml`, this skill *may* create or edit `source.local.yaml`
on this user's behalf — but only when they explicitly ask to become (or
stop being) a given source's admin (e.g. "make me the admin for the
finance reports source" / "I don't want write access to X anymore"), never
as a side effect of just running "index external sources."

## Regeneration rule: `## Notes` is never overwritten

Every page this skill writes is *generated* — a later run rebuilds it from
the connector and the previous contents are gone. That makes this index the
one place in the KB where a human annotation is most useful and most
fragile: the person who knows that a document is superseded, or that a name
in it refers to someone who has since left, has nowhere safe to write it
down.

`## Notes` is that place. Before rewriting any page under `libs/<name>/`,
read the existing file and lift its `## Notes` section out verbatim; after
regenerating everything above it, append it back **byte-for-byte**. Not
reflowed, not summarized, not merged into the `tldr`, not "improved" —
unchanged, including whitespace and any half-finished sentence.

Practically: `## Notes` is the last section of every generated page, so a
rebuild is "replace everything above the `## Notes` heading". Create it empty
on a page's first write so the affordance exists before anyone needs it:

```markdown
## Notes

<!-- Yours. Never rewritten by any skill. -->
```

If a page somehow has content *after* `## Notes` that this skill would
otherwise generate, preserve the whole tail rather than guessing where the
human part ends. Losing generated content costs one re-run; losing a person's
annotation costs the thing itself.

## How to run this skill

### Step 1 — Find connector-backed libs

Scan `libs/*/` for a `source.yaml`. A `libs/<name>/` with one is
connector-backed and in scope; a `libs/<name>/` without one is a git-copy
lib and stays completely untouched by this skill.

If no `libs/<name>/source.yaml` exists anywhere, tell the user there's
nothing to index yet and how to add one (a `source.yaml` with `connector`
and `location`), and stop — don't fabricate an index from nothing.

### Step 2 — Read the local access setting

Read `libs/<name>/source.local.yaml` if it exists. `access: write` means
this user is an admin for this one source; anything else (the field is
`read`, or the file doesn't exist at all) means read-only. This check is
**per connector-backed lib** — a user can be write-access for one source
and read-only for another.

### Step 3 — Check the shared index store, and fetch it if it exists

If `source.yaml` has an `index:` block, check whether `index.location`
already has a published index (per `index.store`: a `git` remote with
commits/a reachable ref, or a filesystem/network/`shared` location with
files already in it):
- **It exists** → fetch/pull it (`git pull`/clone for `git`, a plain file
  copy for a filesystem/network `shared` path, or the matching connector
  tool for a `shared` location that needs one) and use it to refresh
  `libs/<name>/{index.md,entities/,graph/,log.md}`. Do this **regardless
  of this user's access level** — reading the published index never
  requires write access, and it's exactly what lets a read-only user skip
  building anything themselves.
- **It's empty / nothing published yet** → there's nothing to fetch. Don't
  treat this as an error; it's the normal first-time state before anyone
  with write access has run this. Continue to Step 4 — whether that turns
  into "create it" or "nothing to report" depends entirely on this user's
  access level.

If no `index:` block is configured at all, skip this check — the index (if
any exists yet) already lives directly in `libs/<name>/`, same as before.

### Step 4 — Decide whether to build or refresh from the live connector

This is the one place access level actually changes behavior:

- **Read-only** (the common case, and the default for anyone who hasn't
  set `access: write` locally): stop here, regardless of what Step 3
  found. Never touch the live connector on a read-only user's behalf.
  - If Step 3 fetched something (or one already lived in `libs/<name>/`
    with no `index:` block): report what it covers.
  - If nothing exists anywhere yet (Step 3 found the shared store empty,
    or there's no `index:` block and `libs/<name>/` is empty too): say so
    plainly, and suggest asking whoever administers this source to run it,
    or setting `access: write` locally if this user is meant to be that
    admin.
- **Write access**: always continue to Step 5, whether Step 3 fetched an
  existing index (refresh it) or found the store empty (build the very
  first version from scratch) — the two cases are handled identically
  from here on; Step 8 is what actually creates the remote copy either
  way.

Either way, compare the existing index's newest `last_updated` against
this source's `refresh_interval_days` (default 30) and say where it
stands. A read-only user needs to know they're reading a copy that's three
weeks past due so they can go ask the admin rather than quietly trusting
it; a write-access user is about to rebuild anyway, but "this was 40 days
stale" is worth saying, because a source that's routinely overdue is
either configured with too tight an interval or has nobody actually
maintaining it. Both are worth surfacing rather than silently correcting.

### Step 5 — Resolve the connector and enumerate/summarize documents

Resolve `connector` to whatever live tool is actually available this
session:
- `sharepoint` / a Microsoft-hosted location → search (`ToolSearch`) for a
  connected Microsoft 365 MCP tool.
- `google_drive` → search for a connected Google Drive MCP tool.
- `web` / `custom` / anything else with a plain URL → `WebFetch`.

If the needed connector isn't authorized in this session (per the current
MCP auth state), **don't fail the whole run** — tell the user which
connector needs authorizing (via claude.ai connector settings, or `claude
mcp`/`/mcp` for non-claude.ai servers), skip that source, and continue with
any others.

For each document found at the location:
- Read enough content to write a real 1-2 sentence `tldr` when the
  connector allows fetching content.
- If only metadata is available (listing only, no read access), say so
  honestly in the `tldr` (e.g. "Metadata only — content not readable") and
  set `confidence` low (e.g. 0.3) rather than fabricating a summary.

Create/update one entity page per document at
`libs/<name>/entities/<doc-slug>.md`:
```yaml
---
type: document
resource: <document's own URL/path>
tldr: ...
confidence: 0.3–0.9   # per the rule above
quality: ...
last_updated: YYYY-MM-DD
freshness_window_days: 30   # this source's refresh_interval_days, or 30 if unset — shorter than a
                             #   typical wiki page, because external sources change without notice
retention: medium
source_fingerprint: sha256:3f9a2c1e   # see below — what actually detects drift
source_checked: YYYY-MM-DD
---
```

**Fingerprint every document page.** A short `freshness_window_days` is a
guess that a source *might* have moved; a fingerprint is a fact about whether
it *did*. This matters far more here than in `wiki/`: an archived file under
`raw/archive/` is effectively immutable, while a SharePoint or Drive document
is edited by other people with no notification to anyone.

Take the fingerprint from whatever the connector gives you, in this order of
preference:

1. A version identifier the connector already maintains — an ETag, a
   `version`, a content hash, a `lastModifiedDateTime`. Record it as
   `etag:<value>` / `mtime:<iso8601>`. This is free and needs no content
   fetch.
2. Failing that, `sha256` of the fetched content, recorded as
   `sha256:<first 8 hex chars>`.
3. Metadata-only documents with neither: omit the field rather than
   inventing one, and leave `confidence` low as Step 5 already requires.

Set `source_checked` to the date the fingerprint was last confirmed — which
is **not** always `last_updated`. A re-index that confirms a document
unchanged advances `source_checked` and leaves `last_updated` alone; that
distinction is exactly what tells a reader "this was verified yesterday" from
"this was written six months ago and nobody has looked since".

Give every document page a trailing `## Notes` section, created empty on
first write (see Step 7's protection rule).

### Step 6 — Extract entities and processes mentioned within each document

A document-only index tells a reader *what exists*, but not *what's known
about a given person, topic, or procedure* without reading every `tldr` by
hand. Close that gap the same way `ckb-ingest` extracts entities into
`wiki/`, but keep the result scoped and thin — this connector's own layer
is a retrieval index into its source documents, not a second knowledge
base competing with `wiki/`'s synthesis.

For each document processed in Step 5, identify what it actually
*discusses*, not just what it *is*: named people, organizations, projects,
decisions, systems, and processes/procedures (e.g. "VDI access request
process", "candidate profile approval") mentioned in its content. Skip
this for a document that's purely about itself with nothing else
extractable (a bare calendar acceptance with no discussion, for instance)
— not every document yields additional pages.

For each distinct entity/process found across the source's documents,
create or update one page at `libs/<name>/entities/<entity-slug>.md`,
using the same slug convention `wiki/entities/` would use for that name
(so `damien-gultig.md`, not a date-prefixed slug — this is what keeps
entity-page filenames from colliding with document-page filenames, which
are always date-prefixed per Step 5):

```yaml
---
type: person | organization | project | decision | system | concept | ...   # same open field as wiki/'s schema
resource: /wiki/entities/<slug>.md   # OPTIONAL — only if a full page for this entity already exists in wiki/
tldr: One sentence — who/what this is, and what these documents specifically show (not a full biography;
      if wiki/ already has a full page, this tldr should say what these documents add, not restate it)
confidence: 0.3–0.9
quality: ...
last_updated: YYYY-MM-DD
freshness_window_days: 30   # or this source's refresh_interval_days, if set
retention: medium
---
```

Body: a short "Mentioned in" list, one bullet per document that discusses
this entity, linking to that document's own page
(`entities/<doc-slug>.md`) with a one-line note of what that specific
document says — enough for `ckb-retrieve` (or a human) to know exactly
which document to open for the full context, without needing to re-derive
it from scratch.

**If this entity already has a full page in `wiki/entities/`** (check the
cascade first, same as `ckb-ingest`'s Step 2), set `resource:` to that
page's path and keep this page's body to just the "Mentioned in" list —
don't re-synthesize what the wiki page already says. The wiki page remains
authoritative (cascade priority); this page's only job is pointing back to
*these specific documents* as additional evidence, which the wiki page may
not have cited yet.

Record real relationships surfaced by a document between two entities as
edges in `libs/<name>/graph/edges.json`, using the same closed vocabulary
`wiki/graph/index.md` defines (`part_of`, `uses`, `depends_on`, `produces`,
`configures`, `validates`, `implements`, `caused`, `contradicts`,
`supersedes`) — each verb answering the question listed there. Also add
a `mentioned_in` edge from each entity to every document that discusses
it — this is what makes the graph a genuine index into the source
documents rather than just a bag of loose pages.

### Step 7 — Update the hub page, entities index, graph, and log

Create/update `libs/<name>/index.md` — the root routing page for this
connector, mirroring `wiki/index.md`'s own role:
```yaml
---
type: external-source
resource: <location, from source.yaml>
tldr: One sentence — what this source is, how many documents were found, and how many entities/processes were extracted from them
last_updated: YYYY-MM-DD
---
```
followed by a short routing table pointing to `entities/index.md` and
`graph/index.md`.

Update `libs/<name>/entities/index.md` as two sections, so both kinds of
page stay easy to find without conflating them:
```markdown
## Documents
- [<doc-slug>](<doc-slug>.md) — <tldr>
...

## Entities & Processes
- [<entity-slug>](<entity-slug>.md) — <tldr>
...
```
Same flat-bullet, no-frontmatter convention as `wiki/entities/index.md`
otherwise — this is just a two-heading variant of it, not a new format.

Update `libs/<name>/graph/index.md` (same convention as
`wiki/graph/index.md`) to summarize both the `mentioned_in` document
coverage and any real entity-to-entity edges found. Log every
created/updated page in `libs/<name>/log.md`, same reverse-chronological
format as Rule B in `CLAUDE.md`/`AGENTS.md` — this log is independent of
`wiki/log.md`; **nothing under `wiki/` is touched by this skill at all.**

### Step 8 — Publish, if this source has both write access and a configured index store

If this user has `access: write` **and** `source.yaml` has an `index:`
block, push the refreshed `libs/<name>/{index.md,entities/,graph/,log.md}`
back out to `index.location` (`git push` for `store: git`, a file copy
back for a filesystem/network `store: shared` path, or the matching
connector tool for a `shared` location that needs one) — **every single
run**, not just the first one, so every other user's next fetch (Step 3)
picks up the update. This is exactly how a shared index gets created in
the first place: if Step 3 found the store empty, this step's push is what
populates it for the first time; there's no separate "initialize" action.

If no `index:` block is configured, there's nothing external to publish;
the refreshed files staying inside this repo's own `libs/<name>/` (tracked
by this repo's own git) *is* the publish step, same as the original
default behavior.

If this user is **read-only**, this step never runs — nothing is ever
pushed or copied to `index.location` (or, in the no-`index:`-block case,
nothing is even rebuilt in `libs/<name>/` to begin with). A read-only user
has, by construction, nothing of their own to save back — Step 4 already
stopped them before anything was built.

### Step 9 — Remind to review and sync

This is always the last step, every time this skill makes any change at
all. Close with a short reminder — do not sync or push anything yourself
here:

> "External source indexing complete for `libs/<name>/` — see
> `libs/<name>/log.md` for the details. Please review the changes, and
> once you're happy with them, say 'sync changes' to commit and push them
> to `origin`."

If nothing was actually indexed or fetched (no connector-backed libs
found, every source was skipped for lack of authorization, or every source
was read-only with nothing new to fetch), skip this reminder.

## Edge cases

- **No connector-backed `libs/<name>/` exists** — report that and explain
  how to add one; don't touch anything.
- **`source.local.yaml` doesn't exist yet for a source** — treat as
  read-only; this is the correct default, not a misconfiguration worth
  flagging loudly. Mention once, briefly, how to become the admin
  (`access: write` in that file) if the user seems to expect a rebuild.
- **Everyone is read-only and nothing has ever been published** — report
  that clearly rather than silently doing nothing; this is the one case
  worth flagging as a real gap (nobody can currently index this source).
- **Connector not authorized this session** — report which one and how to
  authorize it (claude.ai connector settings / `claude mcp` / `/mcp`),
  skip that source, keep going with the rest. Never ask the user for
  tokens or callback URLs.
- **Fetching a published index (Step 3) fails or the remote is
  unreachable** — report it and fall back to whatever's already cached
  locally in `libs/<name>/` from a previous fetch, rather than blocking
  the whole run; don't let a transient network issue block a read-only
  user from seeing the last-known index.
- **Re-running against an already-indexed source** — refresh existing
  document *and* entity/process pages in place (update
  `tldr`/`last_updated`/`confidence`/`source_fingerprint`/`source_checked`,
  add newly-seen `mentioned_in` edges); never duplicate a document's or
  entity's page, and carry every `## Notes` section across untouched.
- **A document's fingerprint is unchanged since the last run** — don't
  re-summarize it. Advance `source_checked` to today, leave `last_updated`,
  `tldr`, and `confidence` exactly as they were, and spend the run's budget
  on the documents that did change. On a large source this is most of them,
  and it is the difference between a re-index that costs a full pass and one
  that costs a listing.
- **A document's fingerprint changed** — re-fetch and re-summarize it, and
  say so in the report. A changed fingerprint on a document some wiki page
  cites is worth calling out by name: that wiki page was built from a version
  of this document that no longer exists.
- **A previously-indexed document is no longer found at the source** —
  don't delete its page. Update its `tldr` with a note ("No longer found
  at source as of YYYY-MM-DD") and set `retention: low`, so the existing
  `ckb-lint` retention sweep archives it naturally on a later pass — no new
  archival mechanism needed. Leave any entity pages that cite it alone;
  the citation is still historically accurate even if the document itself
  is gone.
- **An extracted entity already has a full page in `wiki/entities/`** —
  keep this connector-side page thin (evidence/`mentioned_in` pointers
  only, `resource:` set to the wiki page) rather than re-deriving
  everything the wiki page already says; that duplication is exactly what
  the cascade design is meant to avoid.
- **A document is dense with names/topics and extracting every one would
  produce dozens of near-duplicate pages** — extract what's clearly named
  and substantively discussed (not every passing mention), and prefer
  updating an existing entity page's "Mentioned in" list over creating a
  marginal new one. Don't force artificial granularity just to maximize
  page count.
- **A document yields no extractable entity beyond itself** — that's fine;
  its Step 5 document page is the only page it produces. Not every
  document needs to feed Step 6.
- **A source is very large** — cap what gets fully fetched/summarized in
  one run and explicitly report what was skipped (e.g. "42 of ~300
  documents summarized this pass — re-run to continue"). Never silently
  truncate while implying full coverage.
- **Write access but publishing (Step 8) conflicts with a newer version
  someone else already pushed** — this is why Step 3 (fetch) always runs
  first, even for write-access users: rebuild on top of the latest fetched
  state rather than blindly overwriting it. If a real conflict still shows
  up at publish time, report it rather than force-pushing over it.
- **`libs/<name>/` has both real content files and a `source.yaml`** —
  ambiguous and out of scope for this skill to resolve; report it and stop
  for that folder (this is the same case `ckb-lint`'s external-source
  config check flags).
- **`source.yaml` is missing `connector` or `location`** — report it as
  malformed and skip that source; don't guess at intent.

---

*Licensed under the Apache License, Version 2.0 — see [LICENSE](../../../LICENSE)
at the repository root, or <http://www.apache.org/licenses/LICENSE-2.0>.*