ckb/.agents/skills/ckb-index-external/SKILL.md
Michał Kopeć bbfda5a65d Sync: local changes as of 2026-10-29T13:50:12Z
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
2026-10-29 14:50:12 +01:00

375 lines
18 KiB
Markdown
Raw Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

---
name: ckb-index-external
description: Build or refresh the generated index (index.md/entities/graph/log.md) inside each connector-backed libs/<name>/ — one declaring a live external source via source.yaml, such as SharePoint, Google Drive, or a URL. Indexes both the documents found and the people, projects, decisions and processes those documents discuss. Use for "index external sources", "index libs", "refresh the external index", or "scan the SharePoint/Drive folder".
---
# Index external sources skill
## Purpose
The index lives entirely inside its own `libs/<name>/`, at the lowest cascade
layer, in the same `index.md`/`entities/`/`graph/`/`log.md` shape as `wiki/`:
one document page per source item, plus deliberately thin entity and process
pages (a `tldr` and which documents say what) for the people, projects,
decisions and systems those documents discuss. Where `wiki/entities/`
already has a full page, the thin page points there (Step 6); to correct the
connector, write into `wiki/` and let cascade priority win.
`source.yaml` may name a shared, pre-built index that every run fetches
first, so a read-only user never builds from scratch; rebuilding from the
live connector is gated per user by `source.local.yaml` (never committed).
Writes only within that `libs/<name>/`, never `source.yaml`, a git-copy lib,
`wiki/`, or the git remote; pushing is a separate step (Step 8).
## Configuration
### `libs/<name>/source.yaml` (shared, tracked in git, same for everyone)
```yaml
connector: sharepoint # sharepoint | google_drive | web | custom — open string, not a fixed enum
location: "https://contoso.sharepoint.com/sites/Finance/Shared Documents/Reports"
description: "Finance team's shared reports folder" # optional
refresh_interval_days: 7 # optional — how often a write-access run should rebuild this source,
# and the freshness_window_days stamped on its generated pages.
# Omit to use the default of 30. Tune per source: a busy folder
# that changes daily deserves a shorter window than a quarterly
# reports archive that barely moves.
index: # optional — omit entirely for the original default: the index lives only
# here, in this repo, tracked by this repo's own git (nothing to fetch/publish)
store: git # git | shared — open string, where the built index is published to / fetched from
location: "https://github.com/org/finance-index-cache.git" # a git repo URL (store: git), or a
# filesystem/network path or connector-reachable location (store: shared)
ref: main # optional — branch, tag, or subpath hint within that store
```
### `libs/<name>/source.local.yaml` (per-user, per-machine, **never committed**)
```yaml
access: write # write | read — defaults to read if this file is absent or the field is omitted
```
This file is intentionally **not** carved out of `.gitignore` — it stays
caught by the existing `libs/*/*` catch-all, exactly like any other
unlisted file inside a connector-backed `libs/<name>/`. It's local
machine/user state, the same category as a personal `.env` — never synced,
never reviewed by anyone else, and it's what makes "read vs. write" a
per-user decision rather than a shared one baked into `source.yaml`.
Unlike `source.yaml`, this skill *may* create or edit `source.local.yaml`
on this user's behalf — but only when they explicitly ask to become (or
stop being) a given source's admin (e.g. "make me the admin for the
finance reports source" / "I don't want write access to X anymore"), never
as a side effect of just running "index external sources."
## Regeneration rule: `## Notes` is never overwritten
Every page this skill writes is *generated* — a later run rebuilds it from
the connector and the previous contents are gone. That makes this index the
one place in the KB where a human annotation is most useful and most
fragile: the person who knows that a document is superseded, or that a name
in it refers to someone who has since left, has nowhere safe to write it
down.
`## Notes` is that place. Before rewriting any page under `libs/<name>/`,
read the existing file and lift its `## Notes` section out verbatim; after
regenerating everything above it, append it back **byte-for-byte**. Not
reflowed, not summarized, not merged into the `tldr`, not "improved" —
unchanged, including whitespace and any half-finished sentence.
Practically: `## Notes` is the last section of every generated page, so a
rebuild is "replace everything above the `## Notes` heading". Create it empty
on a page's first write so the affordance exists before anyone needs it:
```markdown
## Notes
<!-- Yours. Never rewritten by any skill. -->
```
If a page somehow has content *after* `## Notes` that this skill would
otherwise generate, preserve the whole tail rather than guessing where the
human part ends. Losing generated content costs one re-run; losing a person's
annotation costs the thing itself.
## How to run this skill
### Step 1 — Find connector-backed libs
Scan `libs/*/` for a `source.yaml`. A `libs/<name>/` with one is
connector-backed and in scope; a `libs/<name>/` without one is a git-copy
lib and stays completely untouched by this skill.
If no `libs/<name>/source.yaml` exists anywhere, tell the user there's
nothing to index yet and how to add one (a `source.yaml` with `connector`
and `location`), and stop — don't fabricate an index from nothing.
### Step 2 — Read the local access setting
Read `libs/<name>/source.local.yaml` if it exists. `access: write` means
this user is an admin for this one source; anything else (the field is
`read`, or the file doesn't exist at all) means read-only. This check is
**per connector-backed lib** — a user can be write-access for one source
and read-only for another.
### Step 3 — Check the shared index store, and fetch it if it exists
If `source.yaml` has an `index:` block, check whether `index.location`
already has a published index (per `index.store`: a `git` remote with
commits/a reachable ref, or a filesystem/network/`shared` location with
files already in it):
- **It exists** → fetch/pull it (`git pull`/clone for `git`, a plain file
copy for a filesystem/network `shared` path, or the matching connector
tool for a `shared` location that needs one) and use it to refresh
`libs/<name>/{index.md,entities/,graph/,log.md}`. Do this **regardless
of this user's access level** — reading the published index never
requires write access, and it's exactly what lets a read-only user skip
building anything themselves.
- **It's empty / nothing published yet** → there's nothing to fetch. Don't
treat this as an error; it's the normal first-time state before anyone
with write access has run this. Continue to Step 4 — whether that turns
into "create it" or "nothing to report" depends entirely on this user's
access level.
If no `index:` block is configured at all, skip this check — the index (if
any exists yet) already lives directly in `libs/<name>/`, same as before.
### Step 4 — Decide whether to build or refresh from the live connector
This is the one place access level actually changes behavior:
- **Read-only** (the common case, and the default for anyone who hasn't
set `access: write` locally): stop here, regardless of what Step 3
found. Never touch the live connector on a read-only user's behalf.
- If Step 3 fetched something (or one already lived in `libs/<name>/`
with no `index:` block): report what it covers.
- If nothing exists anywhere yet (Step 3 found the shared store empty,
or there's no `index:` block and `libs/<name>/` is empty too): say so
plainly, and suggest asking whoever administers this source to run it,
or setting `access: write` locally if this user is meant to be that
admin.
- **Write access**: always continue to Step 5, whether Step 3 fetched an
existing index (refresh it) or found the store empty (build the very
first version from scratch) — the two cases are handled identically
from here on; Step 8 is what actually creates the remote copy either
way.
Either way, compare the existing index's newest `last_updated` against
this source's `refresh_interval_days` (default 30) and say where it
stands. A read-only user needs to know they're reading a copy that's three
weeks past due so they can go ask the admin rather than quietly trusting
it; a write-access user is about to rebuild anyway, but "this was 40 days
stale" is worth saying, because a source that's routinely overdue is
either configured with too tight an interval or has nobody actually
maintaining it. Both are worth surfacing rather than silently correcting.
### Step 5 — Resolve the connector and enumerate/summarize documents
Resolve `connector` to whatever live tool is actually available this
session:
- `sharepoint` / a Microsoft-hosted location → search (`ToolSearch`) for a
connected Microsoft 365 MCP tool.
- `google_drive` → search for a connected Google Drive MCP tool.
- `web` / `custom` / anything else with a plain URL → `WebFetch`.
If the needed connector isn't authorized in this session (per the current
MCP auth state), **don't fail the whole run** — tell the user which
connector needs authorizing (via claude.ai connector settings, or `claude
mcp`/`/mcp` for non-claude.ai servers), skip that source, and continue with
any others.
For each document found at the location:
- Read enough content to write a real 1-2 sentence `tldr` when the
connector allows fetching content.
- If only metadata is available (listing only, no read access), say so
honestly in the `tldr` (e.g. "Metadata only — content not readable") and
set `confidence` low (e.g. 0.3) rather than fabricating a summary.
Create/update one entity page per document at
`libs/<name>/entities/<doc-slug>.md`:
```yaml
---
type: document
resource: <document's own URL/path>
tldr: ...
confidence: 0.3–0.9 # per the rule above
quality: ...
last_updated: YYYY-MM-DD
freshness_window_days: 30 # this source's refresh_interval_days, or 30 if unset — shorter than a
# typical wiki page, because external sources change without notice
retention: medium
source_fingerprint: sha256:3f9a2c1e # see below — what actually detects drift
source_checked: YYYY-MM-DD
---
```
**Fingerprint every document page.** A short `freshness_window_days` is a
guess that a source *might* have moved; a fingerprint is a fact about whether
it *did*. This matters far more here than in `wiki/`: an archived file under
`raw/archive/` is effectively immutable, while a SharePoint or Drive document
is edited by other people with no notification to anyone.
Take the fingerprint from whatever the connector gives you, in this order of
preference:
1. A version identifier the connector already maintains — an ETag, a
`version`, a content hash, a `lastModifiedDateTime`. Record it as
`etag:<value>` / `mtime:<iso8601>`. This is free and needs no content
fetch.
2. Failing that, `sha256` of the fetched content, recorded as
`sha256:<first 8 hex chars>`.
3. Metadata-only documents with neither: omit the field rather than
inventing one, and leave `confidence` low as Step 5 already requires.
Set `source_checked` to the date the fingerprint was last confirmed — which
is **not** always `last_updated`. A re-index that confirms a document
unchanged advances `source_checked` and leaves `last_updated` alone; that
distinction is exactly what tells a reader "this was verified yesterday" from
"this was written six months ago and nobody has looked since".
Give every document page a trailing `## Notes` section, created empty on
first write (see Step 7's protection rule).
### Step 6 — Extract entities and processes mentioned within each document
A document-only index tells a reader *what exists*, but not *what's known
about a given person, topic, or procedure* without reading every `tldr` by
hand. Close that gap the same way `ckb-ingest` extracts entities into
`wiki/`, but keep the result scoped and thin — this connector's own layer
is a retrieval index into its source documents, not a second knowledge
base competing with `wiki/`'s synthesis.
For each document processed in Step 5, identify what it actually
*discusses*, not just what it *is*: named people, organizations, projects,
decisions, systems, and processes/procedures (e.g. "VDI access request
process", "candidate profile approval") mentioned in its content. Skip
this for a document that's purely about itself with nothing else
extractable (a bare calendar acceptance with no discussion, for instance)
— not every document yields additional pages.
For each distinct entity/process found across the source's documents,
create or update one page at `libs/<name>/entities/<entity-slug>.md`,
using the same slug convention `wiki/entities/` would use for that name
(so `damien-gultig.md`, not a date-prefixed slug — this is what keeps
entity-page filenames from colliding with document-page filenames, which
are always date-prefixed per Step 5):
```yaml
---
type: person | organization | project | decision | system | concept | ... # same open field as wiki/'s schema
resource: /wiki/entities/<slug>.md # OPTIONAL — only if a full page for this entity already exists in wiki/
tldr: One sentence — who/what this is, and what these documents specifically show (not a full biography;
if wiki/ already has a full page, this tldr should say what these documents add, not restate it)
confidence: 0.3–0.9
quality: ...
last_updated: YYYY-MM-DD
freshness_window_days: 30 # or this source's refresh_interval_days, if set
retention: medium
---
```
Body: a short "Mentioned in" list, one bullet per document that discusses
this entity, linking to that document's own page
(`entities/<doc-slug>.md`) with a one-line note of what that specific
document says — enough for `ckb-retrieve` (or a human) to know exactly
which document to open for the full context, without needing to re-derive
it from scratch.
**If this entity already has a full page in `wiki/entities/`** (check the
cascade first, same as `ckb-ingest`'s Step 2), set `resource:` to that
page's path and keep this page's body to just the "Mentioned in" list —
don't re-synthesize what the wiki page already says. The wiki page remains
authoritative (cascade priority); this page's only job is pointing back to
*these specific documents* as additional evidence, which the wiki page may
not have cited yet.
Record real relationships surfaced by a document between two entities as
edges in `libs/<name>/graph/edges.json`, using the same closed vocabulary
`wiki/graph/index.md` defines (`part_of`, `uses`, `depends_on`, `produces`,
`configures`, `validates`, `implements`, `caused`, `contradicts`,
`supersedes`) — each verb answering the question listed there. Also add
a `mentioned_in` edge from each entity to every document that discusses
it — this is what makes the graph a genuine index into the source
documents rather than just a bag of loose pages.
### Step 7 — Update the hub page, entities index, graph, and log
Create/update `libs/<name>/index.md` — the root routing page for this
connector, mirroring `wiki/index.md`'s own role:
```yaml
---
type: external-source
resource: <location, from source.yaml>
tldr: One sentence — what this source is, how many documents were found, and how many entities/processes were extracted from them
last_updated: YYYY-MM-DD
---
```
followed by a short routing table pointing to `entities/index.md` and
`graph/index.md`.
Update `libs/<name>/entities/index.md` as two sections, so both kinds of
page stay easy to find without conflating them:
```markdown
## Documents
- [<doc-slug>](<doc-slug>.md) — <tldr>
...
## Entities & Processes
- [<entity-slug>](<entity-slug>.md) — <tldr>
...
```
Same flat-bullet, no-frontmatter convention as `wiki/entities/index.md`
otherwise — this is just a two-heading variant of it, not a new format.
Update `libs/<name>/graph/index.md` (same convention as
`wiki/graph/index.md`) to summarize both the `mentioned_in` document
coverage and any real entity-to-entity edges found. Log every
created/updated page in `libs/<name>/log.md`, same reverse-chronological
format as Rule B in `CLAUDE.md`/`AGENTS.md` — this log is independent of
`wiki/log.md`; **nothing under `wiki/` is touched by this skill at all.**
### Step 8 — Publish, if this source has both write access and a configured index store
If this user has `access: write` **and** `source.yaml` has an `index:`
block, push the refreshed `libs/<name>/{index.md,entities/,graph/,log.md}`
back out to `index.location` (`git push` for `store: git`, a file copy
back for a filesystem/network `store: shared` path, or the matching
connector tool for a `shared` location that needs one) — **every single
run**, not just the first one, so every other user's next fetch (Step 3)
picks up the update. This is exactly how a shared index gets created in
the first place: if Step 3 found the store empty, this step's push is what
populates it for the first time; there's no separate "initialize" action.
If no `index:` block is configured, there's nothing external to publish;
the refreshed files staying inside this repo's own `libs/<name>/` (tracked
by this repo's own git) *is* the publish step, same as the original
default behavior.
If this user is **read-only**, this step never runs — nothing is ever
pushed or copied to `index.location` (or, in the no-`index:`-block case,
nothing is even rebuilt in `libs/<name>/` to begin with). A read-only user
has, by construction, nothing of their own to save back — Step 4 already
stopped them before anything was built.
### Step 9 — Remind to review and sync
This is always the last step, every time this skill makes any change at
all. Close with a short reminder — do not sync or push anything yourself
here:
> "External source indexing complete for `libs/<name>/` — see
> `libs/<name>/log.md` for the details. Please review the changes, and
> once you're happy with them, say 'sync changes' to commit and push them
> to `origin`."
If nothing was actually indexed or fetched (no connector-backed libs
found, every source was skipped for lack of authorization, or every source
was read-only with nothing new to fetch), skip this reminder.
## Edge cases
Unusual situations — an empty target, a partial run, a conflicting state,
a missing dependency — are catalogued in
[`references/edge-cases.md`](references/edge-cases.md). Read that file when a
run hits something the steps above don't cover; don't load it otherwise.