ckb/docs/decisions/0023-improvement-memory.md
Michał Kopeć 03ff809440 Improvement memory: attempts, expected outcomes, lesson context (D-0023)
Schema 1.8, after Google's Self-Improvement Memory paper (arXiv 2611.18856).
- Error book: one entry per recurring problem with every attempt, its
  expected and observed result and context; aliases on merge (ckb-lint §9).
- Decisions: expected_outcome, outcome, outcome_checked, ## Tried,
  append-only ## Outcome, an outcome-review procedure (ckb-decide); lint
  validates outcomes and closes a due review once outcome_checked >= review_on.
- Lessons: Context line, upstream lessons from linked/ in the session
  brief, adopt and merge rules (ckb-profile, session_start.py); ckb-init
  Step 8b carries lessons and problems into a new KB; ckb-upgrade b5.
- Deferred: a self-improvement loop over the template. Rejected: loading
  the whole record into context.
No VERSION change or changelog row: made on the merge from dev (D-0022).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 08:17:19 +01:00

125 lines
7.2 KiB
Markdown
Raw Permalink Blame History

This file contains ambiguous Unicode characters

This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.

---
type: decision
tldr: Schema 1.8 keeps the KB's own improvement history the way Google's Self-Improvement Memory does: error-book entries per recurring problem with every attempt, its expected and observed result and its context; decisions with an expected_outcome checked at review and a ## Tried section; lessons with a Context line, upstream lessons from linked KBs, lessons carried into a new KB, and merges that keep aliases. A self-improvement loop over the template is deferred until evals exist; loading the whole record into context unretrieved is rejected.
status: accepted
decided_on: 2026-11-18
decided_by: Michał Kopeć
affects: AGENTS.md, wiki/index.md, wiki/error-book.md, wiki/decisions/index.md, .agents/skills/ckb-lint/, .agents/skills/ckb-decide/, .agents/skills/ckb-profile/, .agents/skills/ckb-init/, .agents/skills/ckb-upgrade/
review_on: 2027-02-18
expected_outcome: By the review, at least one error-book problem in a real KB has two or more attempts with a filled-in Observed column, and at least one decision with an expected_outcome has had its outcome recorded; if neither has happened, the format is too heavy and should be cut back.
confidence: 0.7
quality: 0.8
last_updated: 2026-11-18
freshness_window_days: 365
retention: high
---
# D-0023 — Improvement memory: attempts, expected outcomes, lesson context
**Status:** Accepted · **Decided:** 2026-11-18 · **Deciders:** Michał Kopeć
## Context
Yu et al., *Self-Improvement Memory Enables Continual Recursive
Self-Improvement* (Google, arXiv 2611.18856, 2026-11-13), studies agents
that improve their own harness. When the model or the task changes, the
improved harness alone does not say why each change was made or whether it
worked. Their Self-Improvement Memory (SIM) keeps, per recurring problem,
every attempt with its rationale, the tasks it was **predicted** to fix
(written before evaluation), what it actually fixed, and the setting it ran
in. Continuing with SIM beat continuing with the harness alone in all ten
of their settings (+2.36 points on average across ten paired coding runs,
p = 0.003). The record helped most by keeping the *reason* for an inherited
change, so the next optimizer adapted it instead of reverting it, and by
keeping failed attempts so they were not retried blind.
A Cascade KB already improves itself in small ways: lint records systemic
issues in the error book, people record lessons, decisions record why. At
2.2.0 none of these held SIM's three working parts. The error book was one
row per incident. Nothing wrote down an expected result before the result
was known. Lessons did not say which model or template they were learned
under.
## Decision
Schema 1.8. Every addition is optional and additive.
1. **Error book by problem.** `wiki/error-book.md` holds one entry per
recurring problem (`## E<n> — <root-cause hypothesis>`, status
`open`/`watching`/`resolved`, a `Constraint:` once known), each with a
table of attempts: date, what changed, **Expected** (written first),
**Observed** (`pending`, then `held`/`recurred`/`partly` with a date),
**Context**. Lint check 9 appends to an existing problem before opening a
new one, fills in pending observations on each run, and never rewrites an
attempt. Old single-table error books are kept as `## Earlier entries`.
2. **Expected outcomes on decisions.** `expected_outcome` (one line, written
when the decision is recorded; one added later is marked `(added
YYYY-MM-DD)`), and at review `outcome` (`met`/`partly`/`missed`/`unclear`),
`outcome_checked` and a dated entry appended to `## Outcome`. Lint
validates the vocabulary and treats a review as done once
`outcome_checked` is on or after `review_on`.
3. **Context on lessons and attempts.** A lesson carries `- **Context:**
<agent/model>, template <VERSION>[, setting]`. A lesson from a different
context is a strong hint to recheck, not a rule; a lesson without one is
context unknown.
4. **What was tried.** Decision records gain an optional `## Tried` section,
for what was built or tested and dropped, kept apart from *Alternatives
considered* (weighed, not tried). Template ADRs use it too. Picking a
direction that is listed as tried, or reversing a rejected decision,
makes `ckb-decide` ask what is new since then.
5. **Experience across KBs.** The session brief lists new lessons from each
`linked/<name>/wiki/lessons.md` as `upstream`, ranked below local ones
and read-only. "Adopt that lesson" copies one in with a `Carried from:`
line. `ckb-init`, run inside a populated KB, offers once to carry chosen
lessons and error-book problems into the new KB; carried problems become
`watching`, both files `review: needs_update`. Profiles are never carried.
6. **Merges keep aliases.** A merged error-book entry stays as
`## E4 → E2 — merged YYYY-MM-DD`, and its attempts move across marked
`(from E4)`. A merged lesson is struck through with `merged into "<rule>"`
and the survivor gets `Also recorded:`. Struck-through lessons are no
longer listed in the brief.
## Tried
Nothing was piloted before this decision. The paper's results come from
harness optimisation on benchmarks, not from knowledge bases, so how well
this transfers is the open question. See `review_on` and `expected_outcome`
above.
## Alternatives considered
- **A self-improvement loop for the template** — the paper's own setting:
propose edits to skills, score them on an eval set, check a held-out
split and earlier passing cases, keep the winners, all with SIM recording
the attempts. **Deferred, to consider later.** It needs a behavioural eval
set first (the routing and retrieval evals already listed as gaps). Once
one exists, use the paper's two lessons: hold out part of the eval set,
and check whether previously passing cases still pass after a change.
- **Load the whole record into context, unretrieved** — SIM gives the
optimizer the full memory every round with no retrieval or truncation.
**Rejected.** It works for one optimizer's notes, which stay small. A
KB's error book, lessons and decisions grow without bound and compete
with the context budget lint enforces (check 19). The brief already lists
new lessons; the rest is read index-first when a task bears on it.
- **Machine-readable JSON for the error book**, as SIM stores it.
Rejected: every other KB record is Markdown that people edit and review,
and nothing needs to parse attempts automatically yet.
- **Backfilling expected outcomes and lesson context on upgrade.** Rejected:
a prediction written after the result is hindsight, and a guessed context
is worse than an honest "unknown".
## Consequences
- Recording a decision asks one more question (what it should change), and
lint asks for an outcome when a review comes due. That is more friction;
the review date above checks whether it is used.
- The error book becomes longer and more useful only if lint really goes
back to pending attempts. The skill says so; nothing in a script enforces
it.
- A KB that links others sees their lessons in its brief, which may be
noise if the KBs differ a lot. Upstream lessons rank last and are only
listed, never applied unasked.
## Sources
- `https://arxiv.org/abs/2611.18856` — Yu et al., 2026-11-13, read 2026-11-18 (§3.2, §3.3, App. D.2–D.3, F.6)