Michał Kopeć
c56348b15b
Sets up the layered wiki/linked/libs knowledge base described in CLAUDE.md/AGENTS.md, with a frontmatter schema, recursive index/log convention, and conformance lint checks aligned with Google's Open Knowledge Format. Adds two on-demand Claude Code Skills: export-okf (regenerate an OKF-conformant bundle at outputs/okf/) and sync-changes (reconcile this repo's git history with its origin remote, surfacing conflicts to the user). Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
242 lines
11 KiB
Markdown
242 lines
11 KiB
Markdown
# Cascade Knowledge Base
|
||
|
||
A layered, agent-managed wiki where local content overlays read-only upstream
|
||
sources. Built on Karpathy's LLM Wiki pattern with extensions for scaling,
|
||
lifecycle management, and multi-agent support.
|
||
|
||
---
|
||
|
||
## Directory Structure
|
||
|
||
```
|
||
├── libs/ # Read-only external KBs copied via git (gitignored)
|
||
├── linked/ # Read-only upstream KBs mounted as symlinks
|
||
├── outputs/ # Generated artifacts, exports, compiled files
|
||
├── raw/ # User-provided source material
|
||
│ └── inbox/ # Drop zone: unprocessed material
|
||
├── tmp/ # Temporary files, caches (gitignored)
|
||
├── wiki/ # Local structured markdown wiki (agent-managed)
|
||
│ ├── index.md # Routing table with "Use when" triggers + kb_schema_version
|
||
│ ├── overview.md # High-level map
|
||
│ ├── log.md # Root rollup change log
|
||
│ ├── error-book.md # Compilation errors & derived constraints
|
||
│ ├── entities/ # Typed entity pages (people, projects, concepts) + own index.md
|
||
│ └── graph/ # Edge lists and relationship data + own index.md
|
||
└── workload/ # Session summaries & decisions
|
||
└── YYYY-MM-DD_summary.md
|
||
```
|
||
|
||
---
|
||
|
||
## Cascade Priority
|
||
|
||
When searching, layers are checked in order — first match wins:
|
||
|
||
```
|
||
wiki/ (highest) ← agent writes here, always wins
|
||
linked/A/ (medium) ← symlinked upstream KBs
|
||
linked/B/ (low) ← symlinked upstream KBs
|
||
libs/A/ (lowest) ← git-managed external KB copies
|
||
```
|
||
|
||
The agent never writes to `linked/` or `libs/`. To correct upstream content,
|
||
write the right version in `wiki/` — it takes precedence automatically.
|
||
|
||
---
|
||
|
||
## Features
|
||
|
||
### Inbox-Driven Workflow
|
||
Drop any raw material (notes, articles, links) into `raw/inbox/` without
|
||
organizing. On "Ingest", the agent processes the inbox — extracts knowledge,
|
||
files it into `wiki/`, and archives processed items to `raw/`.
|
||
|
||
### Lazy-Loading Index with "Use When" Triggers
|
||
`wiki/index.md` is a routing table. Each entry has a **Use when** column
|
||
listing trigger keywords. The agent reads the index first (stays in context),
|
||
matches keywords against the task, and only loads matching pages. This
|
||
drops context overhead from ~12K to ~3.2K tokens per task.
|
||
|
||
### TLDR-First Query Layer
|
||
Every page carries a one-sentence `tldr` in frontmatter. When querying, the
|
||
agent reads TLDRs first. If the TLDR already answers the question, the full
|
||
body is never loaded. Fallback chain: TLDR → body → raw source.
|
||
|
||
### Page Frontmatter Schema
|
||
Every wiki page uses YAML frontmatter. `type` is required; the rest are optional:
|
||
|
||
```yaml
|
||
---
|
||
type: concept # REQUIRED. Open string: person, project, concept, library, decision, playbook, ...
|
||
resource: https://... # Canonical URI to the authoritative external source this page describes
|
||
tldr: One-sentence summary optimised for LLM reading
|
||
confidence: 0.0–1.0 # Source corroboration score
|
||
quality: 0.0–1.0 # Self-evaluation (below 0.7 → flagged)
|
||
supersedes: path/to/old.md
|
||
superseded_by: path/to/new.md
|
||
last_updated: YYYY-MM-DD
|
||
freshness_window_days: 90 # Days before considered stale
|
||
retention: high|medium|low
|
||
---
|
||
```
|
||
|
||
- **type** — required; unregistered string, new values always valid, readers tolerate unrecognized ones
|
||
- **resource** — optional pointer to the live/authoritative source, kept separate from the wiki's own commentary
|
||
- **confidence** — set on write, decays with time, reinforced by new sources
|
||
- **quality** — self-scored on write, pages below 0.7 flagged for review
|
||
- **supersedes / superseded_by** — when new info replaces old, link them
|
||
- **freshness_window_days** — pages older than this get flagged during lint
|
||
- **retention** — low pages are archived after 2× freshness window
|
||
|
||
`wiki/index.md` alone also carries `kb_schema_version` (e.g. `"1.1"`), declaring
|
||
which revision of this schema the wiki was authored against — bump minor for
|
||
additive optional fields, major for breaking changes.
|
||
|
||
### Entity Extraction & Knowledge Graph
|
||
During ingest, the agent extracts typed entities (people, projects, libraries,
|
||
concepts, systems) and stores them as pages in `wiki/entities/`. Typed
|
||
relationships (`uses`, `depends_on`, `caused`, `contradicts`, `supersedes`)
|
||
are recorded in `wiki/graph/edges.json`. Queries can walk the graph to
|
||
discover connected pages (e.g. "what depends on Redis?").
|
||
|
||
### Recursive Index & Log Convention
|
||
Any `wiki/` subdirectory that groups multiple pages (`entities/`, `graph/`,
|
||
future topic folders) keeps its own `index.md` — a plain link list, no
|
||
frontmatter — so subdirectory navigation stays lazy instead of requiring a
|
||
full scan. A subdirectory can also keep its own `log.md` once it has enough
|
||
independent history; `wiki/log.md` stays the root-level rollup and never
|
||
duplicates a change a subdirectory log already recorded.
|
||
|
||
### Dual-Linking (Wikilinks + Markdown)
|
||
Every cross-reference uses both `[[Wikilinks]]` (Obsidian-compatible) and
|
||
standard `[markdown](path.md)` links. Works in Obsidian graph view, GitHub
|
||
rendering, and CLI tools. Upstream references use full relative paths:
|
||
`linked/<name>/...` or `libs/<name>/...`. Intra-wiki references prefer
|
||
project-root-absolute paths (`/wiki/entities/foo.md`) over relative ones,
|
||
so links survive later file moves.
|
||
|
||
### Self-Healing Lint
|
||
Periodically (or on demand), the agent health-checks the wiki:
|
||
- **Conformance** — flags any page missing parseable frontmatter or a `type` field
|
||
- **Freshness** — flags pages past their `freshness_window_days`
|
||
- **Confidence decay** — reduces confidence on unreinforced pages
|
||
- **Retention sweep** — archives `retention: low` pages past 2× window
|
||
- **Supersession detection** — finds contradictions, links old→new
|
||
- **Orphan detection** — finds pages with no inbound links
|
||
- **Graph consistency** — verifies all edges point to existing entities
|
||
- **Index/log consistency** — verifies every subdirectory has an index.md and no change is double-logged
|
||
- **Error Book** — records systemic issues with root cause and fix
|
||
|
||
Auto-fixes what it can (broken links, missing backlinks, stale flags).
|
||
|
||
### Conflict Resolution (Supersession)
|
||
When new information contradicts an existing page, the agent adds
|
||
`supersedes` / `superseded_by` links. The old page is preserved but
|
||
marked stale. Version control for knowledge, not just files.
|
||
|
||
### Quality Scoring
|
||
Every page gets a quality score (0.0–1.0) on write, based on structure,
|
||
source citations, and consistency with the rest of the wiki. Pages below
|
||
0.7 are flagged for review or rewritten in the next lint pass.
|
||
|
||
### Error Book
|
||
Systematic errors (orphan links, formatting issues, cross-page contradictions)
|
||
are recorded in `wiki/error-book.md` with root cause, applied fix, and a
|
||
reusable constraint to prevent recurrence. Two-layer repair:
|
||
- **Layer 1** — deterministic auto-fix for structural issues
|
||
- **Layer 2** — agent reasoning pass for semantic/cross-page issues
|
||
|
||
### Automation Hooks
|
||
- **New source** → auto-ingest on next "Ingest" command
|
||
- **Session start** → load index + latest workload summary
|
||
- **Session end** → compress observations into workload/
|
||
- **Query** → file back valuable answers as wiki pages
|
||
- **Memory write** → check contradictions, trigger supersession
|
||
- **Schedule** → periodic lint, consolidation, retention decay
|
||
|
||
### Demand-Driven Context (DDC)
|
||
The wiki grows based on actual agent failures rather than upfront curation:
|
||
1. Agent can't answer → identifies missing knowledge
|
||
2. Proposes minimal entity/page to fill the gap
|
||
3. User approves or provides source material
|
||
4. Next ingest incorporates it
|
||
|
||
Converges to a stable KB after ~20–30 cycles.
|
||
|
||
### Session Summaries
|
||
After every conversational action, the agent appends to
|
||
`workload/YYYY-MM-DD_summary.md`. This provides continuity between sessions
|
||
and a browsable history of how the KB evolved. The agent reads the latest
|
||
summary on session start to pick up where it left off.
|
||
|
||
### Change Log
|
||
Every wiki modification is immediately logged in `wiki/log.md` in reverse
|
||
chronological order (most recent first), recording what changed, why, and
|
||
the source.
|
||
|
||
### OKF Export (on demand)
|
||
The wiki can be exported as an [Open Knowledge Format](https://github.com/GoogleCloudPlatform/knowledge-catalog/blob/main/okf/SPEC.md)
|
||
v0.1-conformant bundle at `outputs/okf/`, consumable by any generic OKF tool
|
||
(e.g. Google's reference graph visualizer) without disturbing the richer
|
||
internal schema (`confidence`/`quality`/`retention`/`supersedes`/dual-linking)
|
||
that OKF doesn't natively understand. Implemented as a Claude Code Skill —
|
||
see `.claude/skills/export-okf/SKILL.md` — rather than baked into
|
||
`CLAUDE.md`/`AGENTS.md`, so the mapping ruleset only loads into context when
|
||
actually invoked.
|
||
|
||
### Git Sync (on demand)
|
||
This repo's own git history can be reconciled with its `origin` remote on
|
||
demand: local changes get committed, remote changes get pulled and merged,
|
||
any conflicts are presented to the user file-by-file to resolve, then the
|
||
result is pushed automatically. Say "sync changes" to trigger it. Also
|
||
implemented as a Claude Code Skill — see
|
||
`.claude/skills/sync-changes/SKILL.md` — and deliberately distinct from the
|
||
content-level "Sync the wiki" / "Ingest" workflow, which processes
|
||
`raw/inbox/` into structured `wiki/` pages and has nothing to do with git.
|
||
|
||
---
|
||
|
||
## Quick Start
|
||
|
||
1. **Mount upstream KBs:**
|
||
```bash
|
||
ln -s /path/to/other-kb ./linked/my-upstream
|
||
git clone https://github.com/org/external-kb ./libs/external-kb
|
||
```
|
||
|
||
2. **Drop raw material** into `raw/inbox/` (notes, links, articles).
|
||
|
||
3. **Tell an agent to "Ingest"** — it processes the inbox, consults the
|
||
cascade, extracts entities, and writes structured markdown into `wiki/`.
|
||
|
||
4. **Ask questions** — the agent uses the index for routing, TLDRs for
|
||
quick answers, and the graph for relationship discovery.
|
||
|
||
5. **Periodically ask to "Lint"** — the agent health-checks everything,
|
||
auto-fixes what it can, and reports issues.
|
||
|
||
---
|
||
|
||
## Agent Instruction Files
|
||
|
||
| File | Purpose |
|
||
|------|---------|
|
||
| `AGENTS.md` | Full instruction for any AI coding agent |
|
||
| `CLAUDE.md` | Same instructions, auto-detected by Claude Code |
|
||
|
||
---
|
||
|
||
## Tips
|
||
|
||
- Upstream KBs (`linked/` and `libs/`) are **never modified** by agents.
|
||
- To correct upstream content, write the correct version in `wiki/` — it wins.
|
||
- Use `raw/inbox/` for anything unprocessed; the agent clears it on ingest.
|
||
- The `wiki/index.md` routing table is the most important file — keep it current.
|
||
- Confidence, quality, and freshness let you trust the right content and
|
||
flag the rest for review.
|
||
- The `tmp/` and `libs/` directories are gitignored. `outputs/` is tracked
|
||
— commit generated artifacts you want to preserve. `outputs/okf/` is a
|
||
fully-regenerated build artifact (never hand-patched); it stays tracked too
|
||
since an OKF bundle is only useful to teammates if it's actually committed,
|
||
but the export skill never commits on its own — review the diff and commit
|
||
it yourself when you want to publish an update.
|