Schema 1.8, after Google's Self-Improvement Memory paper (arXiv 2611.18856). - Error book: one entry per recurring problem with every attempt, its expected and observed result and context; aliases on merge (ckb-lint §9). - Decisions: expected_outcome, outcome, outcome_checked, ## Tried, append-only ## Outcome, an outcome-review procedure (ckb-decide); lint validates outcomes and closes a due review once outcome_checked >= review_on. - Lessons: Context line, upstream lessons from linked/ in the session brief, adopt and merge rules (ckb-profile, session_start.py); ckb-init Step 8b carries lessons and problems into a new KB; ckb-upgrade b5. - Deferred: a self-improvement loop over the template. Rejected: loading the whole record into context. No VERSION change or changelog row: made on the merge from dev (D-0022). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
125 lines
7.2 KiB
Markdown
125 lines
7.2 KiB
Markdown
---
|
||
type: decision
|
||
tldr: Schema 1.8 keeps the KB's own improvement history the way Google's Self-Improvement Memory does: error-book entries per recurring problem with every attempt, its expected and observed result and its context; decisions with an expected_outcome checked at review and a ## Tried section; lessons with a Context line, upstream lessons from linked KBs, lessons carried into a new KB, and merges that keep aliases. A self-improvement loop over the template is deferred until evals exist; loading the whole record into context unretrieved is rejected.
|
||
status: accepted
|
||
decided_on: 2026-11-18
|
||
decided_by: Michał Kopeć
|
||
affects: AGENTS.md, wiki/index.md, wiki/error-book.md, wiki/decisions/index.md, .agents/skills/ckb-lint/, .agents/skills/ckb-decide/, .agents/skills/ckb-profile/, .agents/skills/ckb-init/, .agents/skills/ckb-upgrade/
|
||
review_on: 2027-02-18
|
||
expected_outcome: By the review, at least one error-book problem in a real KB has two or more attempts with a filled-in Observed column, and at least one decision with an expected_outcome has had its outcome recorded; if neither has happened, the format is too heavy and should be cut back.
|
||
confidence: 0.7
|
||
quality: 0.8
|
||
last_updated: 2026-11-18
|
||
freshness_window_days: 365
|
||
retention: high
|
||
---
|
||
|
||
# D-0023 — Improvement memory: attempts, expected outcomes, lesson context
|
||
|
||
**Status:** Accepted · **Decided:** 2026-11-18 · **Deciders:** Michał Kopeć
|
||
|
||
## Context
|
||
|
||
Yu et al., *Self-Improvement Memory Enables Continual Recursive
|
||
Self-Improvement* (Google, arXiv 2611.18856, 2026-11-13), studies agents
|
||
that improve their own harness. When the model or the task changes, the
|
||
improved harness alone does not say why each change was made or whether it
|
||
worked. Their Self-Improvement Memory (SIM) keeps, per recurring problem,
|
||
every attempt with its rationale, the tasks it was **predicted** to fix
|
||
(written before evaluation), what it actually fixed, and the setting it ran
|
||
in. Continuing with SIM beat continuing with the harness alone in all ten
|
||
of their settings (+2.36 points on average across ten paired coding runs,
|
||
p = 0.003). The record helped most by keeping the *reason* for an inherited
|
||
change, so the next optimizer adapted it instead of reverting it, and by
|
||
keeping failed attempts so they were not retried blind.
|
||
|
||
A Cascade KB already improves itself in small ways: lint records systemic
|
||
issues in the error book, people record lessons, decisions record why. At
|
||
2.2.0 none of these held SIM's three working parts. The error book was one
|
||
row per incident. Nothing wrote down an expected result before the result
|
||
was known. Lessons did not say which model or template they were learned
|
||
under.
|
||
|
||
## Decision
|
||
|
||
Schema 1.8. Every addition is optional and additive.
|
||
|
||
1. **Error book by problem.** `wiki/error-book.md` holds one entry per
|
||
recurring problem (`## E<n> — <root-cause hypothesis>`, status
|
||
`open`/`watching`/`resolved`, a `Constraint:` once known), each with a
|
||
table of attempts: date, what changed, **Expected** (written first),
|
||
**Observed** (`pending`, then `held`/`recurred`/`partly` with a date),
|
||
**Context**. Lint check 9 appends to an existing problem before opening a
|
||
new one, fills in pending observations on each run, and never rewrites an
|
||
attempt. Old single-table error books are kept as `## Earlier entries`.
|
||
2. **Expected outcomes on decisions.** `expected_outcome` (one line, written
|
||
when the decision is recorded; one added later is marked `(added
|
||
YYYY-MM-DD)`), and at review `outcome` (`met`/`partly`/`missed`/`unclear`),
|
||
`outcome_checked` and a dated entry appended to `## Outcome`. Lint
|
||
validates the vocabulary and treats a review as done once
|
||
`outcome_checked` is on or after `review_on`.
|
||
3. **Context on lessons and attempts.** A lesson carries `- **Context:**
|
||
<agent/model>, template <VERSION>[, setting]`. A lesson from a different
|
||
context is a strong hint to recheck, not a rule; a lesson without one is
|
||
context unknown.
|
||
4. **What was tried.** Decision records gain an optional `## Tried` section,
|
||
for what was built or tested and dropped, kept apart from *Alternatives
|
||
considered* (weighed, not tried). Template ADRs use it too. Picking a
|
||
direction that is listed as tried, or reversing a rejected decision,
|
||
makes `ckb-decide` ask what is new since then.
|
||
5. **Experience across KBs.** The session brief lists new lessons from each
|
||
`linked/<name>/wiki/lessons.md` as `upstream`, ranked below local ones
|
||
and read-only. "Adopt that lesson" copies one in with a `Carried from:`
|
||
line. `ckb-init`, run inside a populated KB, offers once to carry chosen
|
||
lessons and error-book problems into the new KB; carried problems become
|
||
`watching`, both files `review: needs_update`. Profiles are never carried.
|
||
6. **Merges keep aliases.** A merged error-book entry stays as
|
||
`## E4 → E2 — merged YYYY-MM-DD`, and its attempts move across marked
|
||
`(from E4)`. A merged lesson is struck through with `merged into "<rule>"`
|
||
and the survivor gets `Also recorded:`. Struck-through lessons are no
|
||
longer listed in the brief.
|
||
|
||
## Tried
|
||
|
||
Nothing was piloted before this decision. The paper's results come from
|
||
harness optimisation on benchmarks, not from knowledge bases, so how well
|
||
this transfers is the open question. See `review_on` and `expected_outcome`
|
||
above.
|
||
|
||
## Alternatives considered
|
||
|
||
- **A self-improvement loop for the template** — the paper's own setting:
|
||
propose edits to skills, score them on an eval set, check a held-out
|
||
split and earlier passing cases, keep the winners, all with SIM recording
|
||
the attempts. **Deferred, to consider later.** It needs a behavioural eval
|
||
set first (the routing and retrieval evals already listed as gaps). Once
|
||
one exists, use the paper's two lessons: hold out part of the eval set,
|
||
and check whether previously passing cases still pass after a change.
|
||
- **Load the whole record into context, unretrieved** — SIM gives the
|
||
optimizer the full memory every round with no retrieval or truncation.
|
||
**Rejected.** It works for one optimizer's notes, which stay small. A
|
||
KB's error book, lessons and decisions grow without bound and compete
|
||
with the context budget lint enforces (check 19). The brief already lists
|
||
new lessons; the rest is read index-first when a task bears on it.
|
||
- **Machine-readable JSON for the error book**, as SIM stores it.
|
||
Rejected: every other KB record is Markdown that people edit and review,
|
||
and nothing needs to parse attempts automatically yet.
|
||
- **Backfilling expected outcomes and lesson context on upgrade.** Rejected:
|
||
a prediction written after the result is hindsight, and a guessed context
|
||
is worse than an honest "unknown".
|
||
|
||
## Consequences
|
||
|
||
- Recording a decision asks one more question (what it should change), and
|
||
lint asks for an outcome when a review comes due. That is more friction;
|
||
the review date above checks whether it is used.
|
||
- The error book becomes longer and more useful only if lint really goes
|
||
back to pending attempts. The skill says so; nothing in a script enforces
|
||
it.
|
||
- A KB that links others sees their lessons in its brief, which may be
|
||
noise if the KBs differ a lot. Upstream lessons rank last and are only
|
||
listed, never applied unasked.
|
||
|
||
## Sources
|
||
|
||
- `https://arxiv.org/abs/2611.18856` — Yu et al., 2026-11-13, read 2026-11-18 (§3.2, §3.3, App. D.2–D.3, F.6)
|