375 lines
18 KiB
Markdown
375 lines
18 KiB
Markdown
---
|
||
name: ckb-index-external
|
||
description: Build or refresh the generated index (index.md/entities/graph/log.md) inside each connector-backed libs/<name>/ — one declaring a live external source via source.yaml, such as SharePoint, Google Drive, or a URL. Indexes both the documents found and the people, projects, decisions and processes those documents discuss. Use for "index external sources", "index libs", "refresh the external index", or "scan the SharePoint/Drive folder".
|
||
---
|
||
|
||
# Index external sources skill
|
||
|
||
## Purpose
|
||
|
||
The index lives entirely inside its own `libs/<name>/`, at the lowest cascade
|
||
layer, in the same `index.md`/`entities/`/`graph/`/`log.md` shape as `wiki/`:
|
||
one document page per source item, plus deliberately thin entity and process
|
||
pages (a `tldr` and which documents say what) for the people, projects,
|
||
decisions and systems those documents discuss. Where `wiki/entities/`
|
||
already has a full page, the thin page points there (Step 6); to correct the
|
||
connector, write into `wiki/` and let cascade priority win.
|
||
|
||
`source.yaml` may name a shared, pre-built index that every run fetches
|
||
first, so a read-only user never builds from scratch; rebuilding from the
|
||
live connector is gated per user by `source.local.yaml` (never committed).
|
||
Writes only within that `libs/<name>/`, never `source.yaml`, a git-copy lib,
|
||
`wiki/`, or the git remote; pushing is a separate step (Step 8).
|
||
|
||
## Configuration
|
||
|
||
### `libs/<name>/source.yaml` (shared, tracked in git, same for everyone)
|
||
|
||
```yaml
|
||
connector: sharepoint # sharepoint | google_drive | web | custom — open string, not a fixed enum
|
||
location: "https://contoso.sharepoint.com/sites/Finance/Shared Documents/Reports"
|
||
description: "Finance team's shared reports folder" # optional
|
||
|
||
refresh_interval_days: 7 # optional — how often a write-access run should rebuild this source,
|
||
# and the freshness_window_days stamped on its generated pages.
|
||
# Omit to use the default of 30. Tune per source: a busy folder
|
||
# that changes daily deserves a shorter window than a quarterly
|
||
# reports archive that barely moves.
|
||
|
||
index: # optional — omit entirely for the original default: the index lives only
|
||
# here, in this repo, tracked by this repo's own git (nothing to fetch/publish)
|
||
store: git # git | shared — open string, where the built index is published to / fetched from
|
||
location: "https://github.com/org/finance-index-cache.git" # a git repo URL (store: git), or a
|
||
# filesystem/network path or connector-reachable location (store: shared)
|
||
ref: main # optional — branch, tag, or subpath hint within that store
|
||
```
|
||
|
||
### `libs/<name>/source.local.yaml` (per-user, per-machine, **never committed**)
|
||
|
||
```yaml
|
||
access: write # write | read — defaults to read if this file is absent or the field is omitted
|
||
```
|
||
|
||
This file is intentionally **not** carved out of `.gitignore` — it stays
|
||
caught by the existing `libs/*/*` catch-all, exactly like any other
|
||
unlisted file inside a connector-backed `libs/<name>/`. It's local
|
||
machine/user state, the same category as a personal `.env` — never synced,
|
||
never reviewed by anyone else, and it's what makes "read vs. write" a
|
||
per-user decision rather than a shared one baked into `source.yaml`.
|
||
|
||
Unlike `source.yaml`, this skill *may* create or edit `source.local.yaml`
|
||
on this user's behalf — but only when they explicitly ask to become (or
|
||
stop being) a given source's admin (e.g. "make me the admin for the
|
||
finance reports source" / "I don't want write access to X anymore"), never
|
||
as a side effect of just running "index external sources."
|
||
|
||
## Regeneration rule: `## Notes` is never overwritten
|
||
|
||
Every page this skill writes is *generated* — a later run rebuilds it from
|
||
the connector and the previous contents are gone. That makes this index the
|
||
one place in the KB where a human annotation is most useful and most
|
||
fragile: the person who knows that a document is superseded, or that a name
|
||
in it refers to someone who has since left, has nowhere safe to write it
|
||
down.
|
||
|
||
`## Notes` is that place. Before rewriting any page under `libs/<name>/`,
|
||
read the existing file and lift its `## Notes` section out verbatim; after
|
||
regenerating everything above it, append it back **byte-for-byte**. Not
|
||
reflowed, not summarized, not merged into the `tldr`, not "improved" —
|
||
unchanged, including whitespace and any half-finished sentence.
|
||
|
||
Practically: `## Notes` is the last section of every generated page, so a
|
||
rebuild is "replace everything above the `## Notes` heading". Create it empty
|
||
on a page's first write so the affordance exists before anyone needs it:
|
||
|
||
```markdown
|
||
## Notes
|
||
|
||
<!-- Yours. Never rewritten by any skill. -->
|
||
```
|
||
|
||
If a page somehow has content *after* `## Notes` that this skill would
|
||
otherwise generate, preserve the whole tail rather than guessing where the
|
||
human part ends. Losing generated content costs one re-run; losing a person's
|
||
annotation costs the thing itself.
|
||
|
||
## How to run this skill
|
||
|
||
### Step 1 — Find connector-backed libs
|
||
|
||
Scan `libs/*/` for a `source.yaml`. A `libs/<name>/` with one is
|
||
connector-backed and in scope; a `libs/<name>/` without one is a git-copy
|
||
lib and stays completely untouched by this skill.
|
||
|
||
If no `libs/<name>/source.yaml` exists anywhere, tell the user there's
|
||
nothing to index yet and how to add one (a `source.yaml` with `connector`
|
||
and `location`), and stop — don't fabricate an index from nothing.
|
||
|
||
### Step 2 — Read the local access setting
|
||
|
||
Read `libs/<name>/source.local.yaml` if it exists. `access: write` means
|
||
this user is an admin for this one source; anything else (the field is
|
||
`read`, or the file doesn't exist at all) means read-only. This check is
|
||
**per connector-backed lib** — a user can be write-access for one source
|
||
and read-only for another.
|
||
|
||
### Step 3 — Check the shared index store, and fetch it if it exists
|
||
|
||
If `source.yaml` has an `index:` block, check whether `index.location`
|
||
already has a published index (per `index.store`: a `git` remote with
|
||
commits/a reachable ref, or a filesystem/network/`shared` location with
|
||
files already in it):
|
||
- **It exists** → fetch/pull it (`git pull`/clone for `git`, a plain file
|
||
copy for a filesystem/network `shared` path, or the matching connector
|
||
tool for a `shared` location that needs one) and use it to refresh
|
||
`libs/<name>/{index.md,entities/,graph/,log.md}`. Do this **regardless
|
||
of this user's access level** — reading the published index never
|
||
requires write access, and it's exactly what lets a read-only user skip
|
||
building anything themselves.
|
||
- **It's empty / nothing published yet** → there's nothing to fetch. Don't
|
||
treat this as an error; it's the normal first-time state before anyone
|
||
with write access has run this. Continue to Step 4 — whether that turns
|
||
into "create it" or "nothing to report" depends entirely on this user's
|
||
access level.
|
||
|
||
If no `index:` block is configured at all, skip this check — the index (if
|
||
any exists yet) already lives directly in `libs/<name>/`, same as before.
|
||
|
||
### Step 4 — Decide whether to build or refresh from the live connector
|
||
|
||
This is the one place access level actually changes behavior:
|
||
|
||
- **Read-only** (the common case, and the default for anyone who hasn't
|
||
set `access: write` locally): stop here, regardless of what Step 3
|
||
found. Never touch the live connector on a read-only user's behalf.
|
||
- If Step 3 fetched something (or one already lived in `libs/<name>/`
|
||
with no `index:` block): report what it covers.
|
||
- If nothing exists anywhere yet (Step 3 found the shared store empty,
|
||
or there's no `index:` block and `libs/<name>/` is empty too): say so
|
||
plainly, and suggest asking whoever administers this source to run it,
|
||
or setting `access: write` locally if this user is meant to be that
|
||
admin.
|
||
- **Write access**: always continue to Step 5, whether Step 3 fetched an
|
||
existing index (refresh it) or found the store empty (build the very
|
||
first version from scratch) — the two cases are handled identically
|
||
from here on; Step 8 is what actually creates the remote copy either
|
||
way.
|
||
|
||
Either way, compare the existing index's newest `last_updated` against
|
||
this source's `refresh_interval_days` (default 30) and say where it
|
||
stands. A read-only user needs to know they're reading a copy that's three
|
||
weeks past due so they can go ask the admin rather than quietly trusting
|
||
it; a write-access user is about to rebuild anyway, but "this was 40 days
|
||
stale" is worth saying, because a source that's routinely overdue is
|
||
either configured with too tight an interval or has nobody actually
|
||
maintaining it. Both are worth surfacing rather than silently correcting.
|
||
|
||
### Step 5 — Resolve the connector and enumerate/summarize documents
|
||
|
||
Resolve `connector` to whatever live tool is actually available this
|
||
session:
|
||
- `sharepoint` / a Microsoft-hosted location → search (`ToolSearch`) for a
|
||
connected Microsoft 365 MCP tool.
|
||
- `google_drive` → search for a connected Google Drive MCP tool.
|
||
- `web` / `custom` / anything else with a plain URL → `WebFetch`.
|
||
|
||
If the needed connector isn't authorized in this session (per the current
|
||
MCP auth state), **don't fail the whole run** — tell the user which
|
||
connector needs authorizing (via claude.ai connector settings, or `claude
|
||
mcp`/`/mcp` for non-claude.ai servers), skip that source, and continue with
|
||
any others.
|
||
|
||
For each document found at the location:
|
||
- Read enough content to write a real 1-2 sentence `tldr` when the
|
||
connector allows fetching content.
|
||
- If only metadata is available (listing only, no read access), say so
|
||
honestly in the `tldr` (e.g. "Metadata only — content not readable") and
|
||
set `confidence` low (e.g. 0.3) rather than fabricating a summary.
|
||
|
||
Create/update one entity page per document at
|
||
`libs/<name>/entities/<doc-slug>.md`:
|
||
```yaml
|
||
---
|
||
type: document
|
||
resource: <document's own URL/path>
|
||
tldr: ...
|
||
confidence: 0.3–0.9 # per the rule above
|
||
quality: ...
|
||
last_updated: YYYY-MM-DD
|
||
freshness_window_days: 30 # this source's refresh_interval_days, or 30 if unset — shorter than a
|
||
# typical wiki page, because external sources change without notice
|
||
retention: medium
|
||
source_fingerprint: sha256:3f9a2c1e # see below — what actually detects drift
|
||
source_checked: YYYY-MM-DD
|
||
---
|
||
```
|
||
|
||
**Fingerprint every document page.** A short `freshness_window_days` is a
|
||
guess that a source *might* have moved; a fingerprint is a fact about whether
|
||
it *did*. This matters far more here than in `wiki/`: an archived file under
|
||
`raw/archive/` is effectively immutable, while a SharePoint or Drive document
|
||
is edited by other people with no notification to anyone.
|
||
|
||
Take the fingerprint from whatever the connector gives you, in this order of
|
||
preference:
|
||
|
||
1. A version identifier the connector already maintains — an ETag, a
|
||
`version`, a content hash, a `lastModifiedDateTime`. Record it as
|
||
`etag:<value>` / `mtime:<iso8601>`. This is free and needs no content
|
||
fetch.
|
||
2. Failing that, `sha256` of the fetched content, recorded as
|
||
`sha256:<first 8 hex chars>`.
|
||
3. Metadata-only documents with neither: omit the field rather than
|
||
inventing one, and leave `confidence` low as Step 5 already requires.
|
||
|
||
Set `source_checked` to the date the fingerprint was last confirmed — which
|
||
is **not** always `last_updated`. A re-index that confirms a document
|
||
unchanged advances `source_checked` and leaves `last_updated` alone; that
|
||
distinction is exactly what tells a reader "this was verified yesterday" from
|
||
"this was written six months ago and nobody has looked since".
|
||
|
||
Give every document page a trailing `## Notes` section, created empty on
|
||
first write (see Step 7's protection rule).
|
||
|
||
### Step 6 — Extract entities and processes mentioned within each document
|
||
|
||
A document-only index tells a reader *what exists*, but not *what's known
|
||
about a given person, topic, or procedure* without reading every `tldr` by
|
||
hand. Close that gap the same way `ckb-ingest` extracts entities into
|
||
`wiki/`, but keep the result scoped and thin — this connector's own layer
|
||
is a retrieval index into its source documents, not a second knowledge
|
||
base competing with `wiki/`'s synthesis.
|
||
|
||
For each document processed in Step 5, identify what it actually
|
||
*discusses*, not just what it *is*: named people, organizations, projects,
|
||
decisions, systems, and processes/procedures (e.g. "VDI access request
|
||
process", "candidate profile approval") mentioned in its content. Skip
|
||
this for a document that's purely about itself with nothing else
|
||
extractable (a bare calendar acceptance with no discussion, for instance)
|
||
— not every document yields additional pages.
|
||
|
||
For each distinct entity/process found across the source's documents,
|
||
create or update one page at `libs/<name>/entities/<entity-slug>.md`,
|
||
using the same slug convention `wiki/entities/` would use for that name
|
||
(so `damien-gultig.md`, not a date-prefixed slug — this is what keeps
|
||
entity-page filenames from colliding with document-page filenames, which
|
||
are always date-prefixed per Step 5):
|
||
|
||
```yaml
|
||
---
|
||
type: person | organization | project | decision | system | concept | ... # same open field as wiki/'s schema
|
||
resource: /wiki/entities/<slug>.md # OPTIONAL — only if a full page for this entity already exists in wiki/
|
||
tldr: One sentence — who/what this is, and what these documents specifically show (not a full biography;
|
||
if wiki/ already has a full page, this tldr should say what these documents add, not restate it)
|
||
confidence: 0.3–0.9
|
||
quality: ...
|
||
last_updated: YYYY-MM-DD
|
||
freshness_window_days: 30 # or this source's refresh_interval_days, if set
|
||
retention: medium
|
||
---
|
||
```
|
||
|
||
Body: a short "Mentioned in" list, one bullet per document that discusses
|
||
this entity, linking to that document's own page
|
||
(`entities/<doc-slug>.md`) with a one-line note of what that specific
|
||
document says — enough for `ckb-retrieve` (or a human) to know exactly
|
||
which document to open for the full context, without needing to re-derive
|
||
it from scratch.
|
||
|
||
**If this entity already has a full page in `wiki/entities/`** (check the
|
||
cascade first, same as `ckb-ingest`'s Step 2), set `resource:` to that
|
||
page's path and keep this page's body to just the "Mentioned in" list —
|
||
don't re-synthesize what the wiki page already says. The wiki page remains
|
||
authoritative (cascade priority); this page's only job is pointing back to
|
||
*these specific documents* as additional evidence, which the wiki page may
|
||
not have cited yet.
|
||
|
||
Record real relationships surfaced by a document between two entities as
|
||
edges in `libs/<name>/graph/edges.json`, using the same closed vocabulary
|
||
`wiki/graph/index.md` defines (`part_of`, `uses`, `depends_on`, `produces`,
|
||
`configures`, `validates`, `implements`, `caused`, `contradicts`,
|
||
`supersedes`) — each verb answering the question listed there. Also add
|
||
a `mentioned_in` edge from each entity to every document that discusses
|
||
it — this is what makes the graph a genuine index into the source
|
||
documents rather than just a bag of loose pages.
|
||
|
||
### Step 7 — Update the hub page, entities index, graph, and log
|
||
|
||
Create/update `libs/<name>/index.md` — the root routing page for this
|
||
connector, mirroring `wiki/index.md`'s own role:
|
||
```yaml
|
||
---
|
||
type: external-source
|
||
resource: <location, from source.yaml>
|
||
tldr: One sentence — what this source is, how many documents were found, and how many entities/processes were extracted from them
|
||
last_updated: YYYY-MM-DD
|
||
---
|
||
```
|
||
followed by a short routing table pointing to `entities/index.md` and
|
||
`graph/index.md`.
|
||
|
||
Update `libs/<name>/entities/index.md` as two sections, so both kinds of
|
||
page stay easy to find without conflating them:
|
||
```markdown
|
||
## Documents
|
||
- [<doc-slug>](<doc-slug>.md) — <tldr>
|
||
...
|
||
|
||
## Entities & Processes
|
||
- [<entity-slug>](<entity-slug>.md) — <tldr>
|
||
...
|
||
```
|
||
Same flat-bullet, no-frontmatter convention as `wiki/entities/index.md`
|
||
otherwise — this is just a two-heading variant of it, not a new format.
|
||
|
||
Update `libs/<name>/graph/index.md` (same convention as
|
||
`wiki/graph/index.md`) to summarize both the `mentioned_in` document
|
||
coverage and any real entity-to-entity edges found. Log every
|
||
created/updated page in `libs/<name>/log.md`, same reverse-chronological
|
||
format as Rule B in `CLAUDE.md`/`AGENTS.md` — this log is independent of
|
||
`wiki/log.md`; **nothing under `wiki/` is touched by this skill at all.**
|
||
|
||
### Step 8 — Publish, if this source has both write access and a configured index store
|
||
|
||
If this user has `access: write` **and** `source.yaml` has an `index:`
|
||
block, push the refreshed `libs/<name>/{index.md,entities/,graph/,log.md}`
|
||
back out to `index.location` (`git push` for `store: git`, a file copy
|
||
back for a filesystem/network `store: shared` path, or the matching
|
||
connector tool for a `shared` location that needs one) — **every single
|
||
run**, not just the first one, so every other user's next fetch (Step 3)
|
||
picks up the update. This is exactly how a shared index gets created in
|
||
the first place: if Step 3 found the store empty, this step's push is what
|
||
populates it for the first time; there's no separate "initialize" action.
|
||
|
||
If no `index:` block is configured, there's nothing external to publish;
|
||
the refreshed files staying inside this repo's own `libs/<name>/` (tracked
|
||
by this repo's own git) *is* the publish step, same as the original
|
||
default behavior.
|
||
|
||
If this user is **read-only**, this step never runs — nothing is ever
|
||
pushed or copied to `index.location` (or, in the no-`index:`-block case,
|
||
nothing is even rebuilt in `libs/<name>/` to begin with). A read-only user
|
||
has, by construction, nothing of their own to save back — Step 4 already
|
||
stopped them before anything was built.
|
||
|
||
### Step 9 — Remind to review and sync
|
||
|
||
This is always the last step, every time this skill makes any change at
|
||
all. Close with a short reminder — do not sync or push anything yourself
|
||
here:
|
||
|
||
> "External source indexing complete for `libs/<name>/` — see
|
||
> `libs/<name>/log.md` for the details. Please review the changes, and
|
||
> once you're happy with them, say 'sync changes' to commit and push them
|
||
> to `origin`."
|
||
|
||
If nothing was actually indexed or fetched (no connector-backed libs
|
||
found, every source was skipped for lack of authorization, or every source
|
||
was read-only with nothing new to fetch), skip this reminder.
|
||
|
||
## Edge cases
|
||
|
||
Unusual situations — an empty target, a partial run, a conflicting state,
|
||
a missing dependency — are catalogued in
|
||
[`references/edge-cases.md`](references/edge-cases.md). Read that file when a
|
||
run hits something the steps above don't cover; don't load it otherwise.
|