ckb/README.md

# Cascade Knowledge Base

*Read this in: **English** | [Polski](README.pl.md)*

**Source repo (always the most up-to-date version):**
[git.wierzbowa.cloud/michal/ckb](https://git.wierzbowa.cloud/michal/ckb)

A layered, agent-managed wiki where local content overlays read-only upstream
sources. Built on Karpathy's LLM Wiki pattern with extensions for scaling,
lifecycle management, and multi-agent support.

This document is a technical feature overview. For a task-oriented guide —
how to create a wiki, add knowledge, keep it tidy, sync with others, and
worked examples for every use case — see [MANUAL.md](MANUAL.md)
([Polski](MANUAL.pl.md)).

---

## Directory Structure

```
├── libs/         # Read-only external sources — git-copy clones (gitignored) OR
│                 #   connector configs (source.yaml) with a self-contained generated index
├── linked/       # Read-only upstream KBs mounted as symlinks
├── outputs/      # Generated artifacts, exports, compiled files
├── raw/          # User-provided source material
│   └── inbox/    # Drop zone: unprocessed material
├── tmp/          # Temporary files, caches (gitignored)
├── wiki/         # Local structured markdown wiki (agent-managed)
│   ├── index.md      # Routing table with "Use when" triggers + kb_schema_version
│   ├── overview.md   # High-level map
│   ├── log.md        # Root rollup change log
│   ├── error-book.md # Compilation errors & derived constraints
│   ├── entities/     # Typed entity pages (people, projects, concepts) + own index.md
│   └── graph/        # Edge lists and relationship data + own index.md
└── workload/     # Session summaries & decisions
    └── YYYY-MM-DD_summary.md
```

---

## Cascade Priority

When searching, layers are checked in order — first match wins:

```
wiki/        (highest)  ← agent writes here, always wins
linked/A/    (medium)   ← symlinked upstream KBs
linked/B/    (low)      ← symlinked upstream KBs
libs/A/      (lowest)   ← git-managed external KB copies, or a connector's own generated index
```

The agent never writes to `linked/` or a git-copy `libs/<name>/`. To correct
upstream content, write the right version in `wiki/` — it takes precedence
automatically. The one exception is a connector-backed `libs/<name>/` (see
"External Source Connectors & Indexing" below) — the agent owns and
maintains its generated index exactly as it would `wiki/`.

---

## Features

### Inbox-Driven Workflow
Drop any raw material (notes, articles, links) into `raw/inbox/` without
organizing. On "Ingest" (or "Sync the wiki" / "Update the wiki"), the agent
processes the inbox — extracts knowledge, files it into `wiki/`, and
archives processed items to `raw/` — then reminds you to review the result
and run "sync changes" to push it to `origin` once you're happy with it.
Implemented as a Claude Code Skill — see
`.agents/skills/ckb-ingest/SKILL.md` — rather than baked into
`CLAUDE.md`/`AGENTS.md`, so the full procedure only loads into context when
actually invoked. Distinct from the `ckb-sync-changes` skill, which is a
pure git-level operation with no wiki synthesis.

### Lazy-Loading Index with "Use When" Triggers
`wiki/index.md` is a routing table. Each entry has a **Use when** column
listing trigger keywords. The agent reads the index first (stays in context),
matches keywords against the task, and only loads matching pages. This
drops context overhead from ~12K to ~3.2K tokens per task.

### TLDR-First Query Layer
Every page carries a one-sentence `tldr` in frontmatter. When querying, the
agent reads TLDRs first. If the TLDR already answers the question, the full
body is never loaded. Fallback chain: TLDR → body → raw source.

### Page Frontmatter Schema
Every wiki page uses YAML frontmatter. `type` is required; the rest are optional:

```yaml
---
type: concept              # REQUIRED. Open string: person, project, concept, library, decision, playbook, ...
resource: https://...      # Canonical URI to the authoritative external source this page describes
tldr: One-sentence summary optimised for LLM reading
confidence: 0.0–1.0      # Source corroboration score
quality: 0.0–1.0          # Self-evaluation (below 0.7 → flagged)
supersedes: path/to/old.md
superseded_by: path/to/new.md
last_updated: YYYY-MM-DD
freshness_window_days: 90 # Days before considered stale
retention: high|medium|low
---
```

- **type** — required; unregistered string, new values always valid, readers tolerate unrecognized ones
- **resource** — optional pointer to the live/authoritative source, kept separate from the wiki's own commentary
- **confidence** — set on write, decays with time, reinforced by new sources
- **quality** — self-scored on write, pages below 0.7 flagged for review
- **supersedes / superseded_by** — when new info replaces old, link them
- **freshness_window_days** — pages older than this get flagged during lint
- **retention** — low pages are archived after 2× freshness window

`wiki/index.md` alone also carries `kb_schema_version` (e.g. `"1.1"`), declaring
which revision of this schema the wiki was authored against — bump minor for
additive optional fields, major for breaking changes.

### Entity Extraction & Knowledge Graph
During ingest, the agent extracts typed entities (people, projects, libraries,
concepts, systems) and stores them as pages in `wiki/entities/`. Typed
relationships (`uses`, `depends_on`, `caused`, `contradicts`, `supersedes`)
are recorded in `wiki/graph/edges.json`. Queries can walk the graph to
discover connected pages (e.g. "what depends on Redis?").

### Recursive Index & Log Convention
Any `wiki/` subdirectory that groups multiple pages (`entities/`, `graph/`,
future topic folders) keeps its own `index.md` — a plain link list, no
frontmatter — so subdirectory navigation stays lazy instead of requiring a
full scan. A subdirectory can also keep its own `log.md` once it has enough
independent history; `wiki/log.md` stays the root-level rollup and never
duplicates a change a subdirectory log already recorded.

### External Source Connectors & Indexing (on demand)
A `libs/<name>/` folder supports a second population mode alongside the
existing git-copy one: a small user-authored `libs/<name>/source.yaml`
declaring a *live* external source — a SharePoint folder, a Google Drive
folder, a plain URL, or another connector — that you don't want to fully
mirror locally:
```yaml
connector: sharepoint
location: "https://contoso.sharepoint.com/sites/Finance/Shared Documents/Reports"
description: "Finance team's shared reports folder"
```
Say "index external sources" and the agent walks it, resolving `connector`
to whatever live tool is available that session (a connected Microsoft
365/Google Drive MCP tool, or `WebFetch` for a plain URL), and builds a
self-contained generated index inside that same `libs/<name>/` —
`index.md`/`entities/`/`graph/`/`log.md`, mirroring `wiki/`'s own shape via
the Recursive Index & Log Convention above, but scoped entirely to that one
connector. This is a deliberate design choice: the index is **not** blended
into the main `wiki/entities/`/`wiki/graph/edges.json` — it stays separated
at the `libs/` cascade layer, the same way a git-cloned KB's own files
already are. `source.yaml` itself stays user-only, never written by the
agent.

Two refinements on top of that:

- **Shared, pre-built indexes.** `source.yaml` can add an optional `index:`
  block declaring *where the already-built index lives* — a git repo, or a
  shared resource such as a network path or another connector-reachable
  location — so most people just fetch what's already there instead of
  building it themselves:
  ```yaml
  index:
    store: git   # git | shared
    location: "https://github.com/org/finance-index-cache.git"
  ```
- **Read vs. write, per user, per source.** Whether *this* user can
  actually rebuild an index (versus only read a fetched/published one) is a
  separate, local, gitignored `libs/<name>/source.local.yaml` — read-only
  by default. Setting `access: write` there opts a given machine/user in as
  that source's admin, so a team can designate one or two people to
  maintain a source while everyone else just reads the result — no
  redundant rebuilding, no need for every user to have their own connector
  authorization.

Implemented as a Claude Code Skill — see
`.agents/skills/ckb-index-external/SKILL.md`.

### Dual-Linking (Wikilinks + Markdown)
Every cross-reference uses both `[[Wikilinks]]` (Obsidian-compatible) and
standard `[markdown](path.md)` links. Works in Obsidian graph view, GitHub
rendering, and CLI tools. Upstream references use full relative paths:
`linked/<name>/...` or `libs/<name>/...`. Intra-wiki references prefer
project-root-absolute paths (`/wiki/entities/foo.md`) over relative ones,
so links survive later file moves.

### Self-Healing Lint
Periodically (or on demand), the agent health-checks the wiki:
- **Conformance** — flags any page missing parseable frontmatter or a `type` field
- **Freshness** — flags pages past their `freshness_window_days`
- **Confidence decay** — reduces confidence on unreinforced pages
- **Retention sweep** — archives `retention: low` pages past 2× window
- **Supersession detection** — finds contradictions, links old→new
- **Orphan detection** — finds pages with no inbound links
- **Graph consistency** — verifies all edges point to existing entities
- **Index/log consistency** — verifies every subdirectory has an index.md and no change is double-logged
- **Error Book** — records systemic issues with root cause and fix

Auto-fixes what it can (broken links, missing backlinks, stale flags), and
reminds you to review the result and run "sync changes" to push it to
`origin` once you're happy with it. Implemented as a Claude Code Skill —
see `.agents/skills/ckb-lint/SKILL.md` — rather than baked into
`CLAUDE.md`/`AGENTS.md`, so the full checklist only loads into context when
actually invoked.

### Conflict Resolution (Supersession)
When new information contradicts an existing page, the agent adds
`supersedes` / `superseded_by` links. The old page is preserved but
marked stale. Version control for knowledge, not just files.

### Quality Scoring
Every page gets a quality score (0.0–1.0) on write, based on structure,
source citations, and consistency with the rest of the wiki. Pages below
0.7 are flagged for review or rewritten in the next lint pass.

### Error Book
Systematic errors (orphan links, formatting issues, cross-page contradictions)
are recorded in `wiki/error-book.md` with root cause, applied fix, and a
reusable constraint to prevent recurrence. Two-layer repair:
- **Layer 1** — deterministic auto-fix for structural issues
- **Layer 2** — agent reasoning pass for semantic/cross-page issues

### Automation Hooks
- **New source** → auto-ingest on next "Ingest" command
- **Session start** → load index + latest workload summary; check for
  unsynchronized changes (uncommitted work, or ahead/behind `origin`) and
  suggest `ckb-sync-changes` if any are found
- **Session end** → compress observations into workload/; re-check for
  unsynchronized changes (including anything the session itself just
  created) and suggest `ckb-sync-changes` if needed
- **Query** → file back valuable answers as wiki pages
- **Memory write** → check contradictions, trigger supersession
- **Schedule** → periodic lint, consolidation, retention decay

### Demand-Driven Context (DDC)
The wiki grows based on actual agent failures rather than upfront curation:
1. Agent can't answer → identifies missing knowledge
2. Proposes minimal entity/page to fill the gap
3. User approves or provides source material
4. Next ingest incorporates it

Converges to a stable KB after ~20–30 cycles.

### Session Summaries
After every conversational action, the agent appends to
`workload/YYYY-MM-DD_summary.md`. This provides continuity between sessions
and a browsable history of how the KB evolved. The agent reads the latest
summary on session start to pick up where it left off.

### Change Log
Every wiki modification is immediately logged in `wiki/log.md` in reverse
chronological order (most recent first), recording what changed, why, and
the source.

### OKF Export (on demand)
The wiki can be exported as an [Open Knowledge Format](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md)
v0.1-conformant bundle at `outputs/okf/`, consumable by any generic OKF tool
(e.g. Google's reference graph visualizer) without disturbing the richer
internal schema (`confidence`/`quality`/`retention`/`supersedes`/dual-linking)
that OKF doesn't natively understand. Implemented as a Claude Code Skill —
see `.agents/skills/ckb-export-okf/SKILL.md` — rather than baked into
`CLAUDE.md`/`AGENTS.md`, so the mapping ruleset only loads into context when
actually invoked. `outputs/okf/` is gitignored — it's a fully-regenerated
build artifact, so each machine/tool regenerates it on demand rather than
carrying it in git history.

### Starlight Export (on demand)
The wiki can also be exported into an Astro + Starlight-consumable form at
`outputs/starlight/`, producing a human-readable documentation website —
unlike the OKF export, which targets machine/tool consumption. A
deterministic Python script (`scripts/export_starlight.py`) handles
frontmatter remapping, dual-link collapsing, wikilink resolution, asset
copying, and sidebar generation; the agent's job is just to ask the two
setup questions (export scope: full runnable scaffold vs. content-only;
whether to include the log/error-book meta pages) and relay the script's
report. Also on-demand and skill-only — see
`.agents/skills/ckb-export-starlight/SKILL.md`. Like `outputs/okf/`,
`outputs/starlight/` is gitignored as a regenerated build artifact.

### Guided Onboarding Tours (on demand)
Ask "onboard me on X" (or "where do I start with X", "mini tour of X") to get
a short, read-only guided reading order: an overview paragraph plus an
ordered list of wiki pages to read, built by walking the knowledge graph
outward from the best-matching page (foundations first, then the topic
itself, then what builds on it). Never writes to `wiki/`. See
`.agents/skills/ckb-onboard-me/SKILL.md`.

### Project Summary (on demand)
Ask for "a project summary" (or "where do things stand", "catch me up on the
project") to regenerate `PROJECT-OVERVIEW.md` at the repo root — a
one-to-two-page snapshot (overview, project state, actions & status, risks,
assumptions) synthesized entirely from the current `wiki/` contents and its
knowledge graph. Always overwritten in full on re-run, never hand-appended
to. See `.agents/skills/ckb-project-summary/SKILL.md`.

### Git Sync (on demand)
This repo's own git history can be reconciled with its `origin` remote on
demand: local changes get committed, remote changes get pulled and merged,
any conflicts are presented to the user file-by-file to resolve, then the
result is pushed automatically. Say "sync changes" to trigger it. Also
implemented as a Claude Code Skill — see
`.claude/skills/ckb-sync-changes/SKILL.md` — and deliberately distinct from the
content-level "Sync the wiki" / "Ingest" workflow, which processes
`raw/inbox/` into structured `wiki/` pages and has nothing to do with git.

### Quiz Mode (on demand)
Ask to be quizzed on the wiki ("quiz me on X", "test my knowledge") for a
one-off, scored knowledge check: the agent reads the relevant pages,
generates open or multiple-choice questions grounded in specific wiki
facts, runs them one at a time with immediate feedback and a running
score, and closes with a verdict. Stateless — nothing is saved between
runs. See `.agents/skills/cbk-quiz/SKILL.md`.

### Guided Teaching Curriculum (on demand)
Ask to be taught the wiki ("teach me the wiki", "teach me about X", "run a
teaching session") for a stateful course rather than a one-off quiz. First
call plans it: scopes the material (optionally supplementing thin spots
from the web, clearly marked as non-authoritative), asks whether it should
be one session or a spaced series (duration, frequency, optional calendar
`.ics` reminders), chunks the content into session-sized portions —
preferring an extra session over cramming — and saves the accepted plan
and a progress tracker to `outputs/teaching/<topic>/`. Later calls compare
plan against progress, teach the next portion using a different technique
each time (Socratic questioning, analogies, worked examples, teach-back,
mnemonics, ...), spot-check retention, and re-teach weak spots before
advancing. Never writes to `wiki/`. See
`.agents/skills/ckb-teach-me/SKILL.md`.

---

## Quick Start

1. **Mount upstream KBs:**
   ```bash
   ln -s /path/to/other-kb ./linked/my-upstream
   git clone https://github.com/org/external-kb ./libs/external-kb
   ```
   Or, for a live source you don't want to fully mirror, drop a
   `libs/<name>/source.yaml` instead (see "External Source Connectors &
   Indexing" above) and say "index external sources."

2. **Drop raw material** into `raw/inbox/` (notes, links, articles).

3. **Tell an agent to "Ingest"** — it processes the inbox, consults the
   cascade, extracts entities, and writes structured markdown into `wiki/`.

4. **Ask questions** — the agent uses the index for routing, TLDRs for
   quick answers, and the graph for relationship discovery.

5. **Periodically ask to "Lint"** — the agent health-checks everything,
   auto-fixes what it can, and reports issues.

---

## Agent Instruction Files

| File | Purpose |
|------|---------|
| `AGENTS.md` | Full instruction for any AI coding agent |
| `CLAUDE.md` | Symlink to `AGENTS.md`, auto-detected by Claude Code |

Skills live in one shared location, `.agents/skills/`, so any agent tooling
that reads that convention picks them up. `.claude/skills` is a symlink to
`.agents/skills` — Claude Code sees the same skill set without a second
copy to keep in sync.

---

## Tips

- Upstream KBs (`linked/` and git-copy `libs/`) are **never modified** by
  agents. A connector-backed `libs/<name>/` (one with a `source.yaml`) is
  the one exception — the agent owns and maintains its generated index,
  but only for a user who's opted themselves into `access: write` locally
  (see the next point); everyone else's copy stays read-only.
- To correct upstream content, write the correct version in `wiki/` — it wins.
- Use `raw/inbox/` for anything unprocessed; the agent clears it on ingest.
- The `wiki/index.md` routing table is the most important file — keep it current.
- Confidence, quality, and freshness let you trust the right content and
  flag the rest for review.
- The `tmp/` directory is gitignored, and so is most of `libs/` — but not
  all of it: a git-copy `libs/<name>/`'s cloned content stays gitignored as
  before, while a connector-backed `libs/<name>/`'s `source.yaml` and its
  generated `index.md`/`entities/`/`graph/`/`log.md` are tracked, since
  they're synthesized knowledge worth sharing via "sync changes," not a
  disposable build artifact. `libs/<name>/source.local.yaml` (per-user
  read/write setting) is the one exception that stays gitignored right
  alongside them — it's personal machine state, never meant to sync.
  `outputs/` itself is
  tracked, but its regenerated build subdirectories, `outputs/okf/` and
  `outputs/starlight/`, are gitignored — each is fully reproducible from
  `wiki/` on demand, so there's nothing to reconcile by carrying it in git
  history. `outputs/teaching/` (per-user learning plans and session
  progress from the teaching skill) is also gitignored, since it's
  personal session state rather than shared KB content. Commit other,
  hand-maintained artifacts under `outputs/` as normal.

---

## Version & License

Current template version: [VERSION](VERSION). Licensed under the
[Apache License 2.0](LICENSE).