ckb/docs/decisions/0023-improvement-memory.md
Michał Kopeć 03ff809440 Improvement memory: attempts, expected outcomes, lesson context (D-0023)
Schema 1.8, after Google's Self-Improvement Memory paper (arXiv 2611.18856).
- Error book: one entry per recurring problem with every attempt, its
  expected and observed result and context; aliases on merge (ckb-lint §9).
- Decisions: expected_outcome, outcome, outcome_checked, ## Tried,
  append-only ## Outcome, an outcome-review procedure (ckb-decide); lint
  validates outcomes and closes a due review once outcome_checked >= review_on.
- Lessons: Context line, upstream lessons from linked/ in the session
  brief, adopt and merge rules (ckb-profile, session_start.py); ckb-init
  Step 8b carries lessons and problems into a new KB; ckb-upgrade b5.
- Deferred: a self-improvement loop over the template. Rejected: loading
  the whole record into context.
No VERSION change or changelog row: made on the merge from dev (D-0022).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
2026-11-18 08:17:19 +01:00

7.2 KiB
Raw Permalink Blame History


type: decision tldr: Schema 1.8 keeps the KB's own improvement history the way Google's Self-Improvement Memory does: error-book entries per recurring problem with every attempt, its expected and observed result and its context; decisions with an expected_outcome checked at review and a ## Tried section; lessons with a Context line, upstream lessons from linked KBs, lessons carried into a new KB, and merges that keep aliases. A self-improvement loop over the template is deferred until evals exist; loading the whole record into context unretrieved is rejected. status: accepted decided_on: 2026-11-18 decided_by: Michał Kopeć affects: AGENTS.md, wiki/index.md, wiki/error-book.md, wiki/decisions/index.md, .agents/skills/ckb-lint/, .agents/skills/ckb-decide/, .agents/skills/ckb-profile/, .agents/skills/ckb-init/, .agents/skills/ckb-upgrade/ review_on: 2027-02-18 expected_outcome: By the review, at least one error-book problem in a real KB has two or more attempts with a filled-in Observed column, and at least one decision with an expected_outcome has had its outcome recorded; if neither has happened, the format is too heavy and should be cut back. confidence: 0.7 quality: 0.8 last_updated: 2026-11-18 freshness_window_days: 365 retention: high

D-0023 — Improvement memory: attempts, expected outcomes, lesson context

Status: Accepted · Decided: 2026-11-18 · Deciders: Michał Kopeć

Context

Yu et al., Self-Improvement Memory Enables Continual Recursive Self-Improvement (Google, arXiv 2611.18856, 2026-11-13), studies agents that improve their own harness. When the model or the task changes, the improved harness alone does not say why each change was made or whether it worked. Their Self-Improvement Memory (SIM) keeps, per recurring problem, every attempt with its rationale, the tasks it was predicted to fix (written before evaluation), what it actually fixed, and the setting it ran in. Continuing with SIM beat continuing with the harness alone in all ten of their settings (+2.36 points on average across ten paired coding runs, p = 0.003). The record helped most by keeping the reason for an inherited change, so the next optimizer adapted it instead of reverting it, and by keeping failed attempts so they were not retried blind.

A Cascade KB already improves itself in small ways: lint records systemic issues in the error book, people record lessons, decisions record why. At 2.2.0 none of these held SIM's three working parts. The error book was one row per incident. Nothing wrote down an expected result before the result was known. Lessons did not say which model or template they were learned under.

Decision

Schema 1.8. Every addition is optional and additive.

  1. Error book by problem. wiki/error-book.md holds one entry per recurring problem (## E<n> — <root-cause hypothesis>, status open/watching/resolved, a Constraint: once known), each with a table of attempts: date, what changed, Expected (written first), Observed (pending, then held/recurred/partly with a date), Context. Lint check 9 appends to an existing problem before opening a new one, fills in pending observations on each run, and never rewrites an attempt. Old single-table error books are kept as ## Earlier entries.
  2. Expected outcomes on decisions. expected_outcome (one line, written when the decision is recorded; one added later is marked (added YYYY-MM-DD)), and at review outcome (met/partly/missed/unclear), outcome_checked and a dated entry appended to ## Outcome. Lint validates the vocabulary and treats a review as done once outcome_checked is on or after review_on.
  3. Context on lessons and attempts. A lesson carries - **Context:** <agent/model>, template <VERSION>[, setting]. A lesson from a different context is a strong hint to recheck, not a rule; a lesson without one is context unknown.
  4. What was tried. Decision records gain an optional ## Tried section, for what was built or tested and dropped, kept apart from Alternatives considered (weighed, not tried). Template ADRs use it too. Picking a direction that is listed as tried, or reversing a rejected decision, makes ckb-decide ask what is new since then.
  5. Experience across KBs. The session brief lists new lessons from each linked/<name>/wiki/lessons.md as upstream, ranked below local ones and read-only. "Adopt that lesson" copies one in with a Carried from: line. ckb-init, run inside a populated KB, offers once to carry chosen lessons and error-book problems into the new KB; carried problems become watching, both files review: needs_update. Profiles are never carried.
  6. Merges keep aliases. A merged error-book entry stays as ## E4 → E2 — merged YYYY-MM-DD, and its attempts move across marked (from E4). A merged lesson is struck through with merged into "<rule>" and the survivor gets Also recorded:. Struck-through lessons are no longer listed in the brief.

Tried

Nothing was piloted before this decision. The paper's results come from harness optimisation on benchmarks, not from knowledge bases, so how well this transfers is the open question. See review_on and expected_outcome above.

Alternatives considered

  • A self-improvement loop for the template — the paper's own setting: propose edits to skills, score them on an eval set, check a held-out split and earlier passing cases, keep the winners, all with SIM recording the attempts. Deferred, to consider later. It needs a behavioural eval set first (the routing and retrieval evals already listed as gaps). Once one exists, use the paper's two lessons: hold out part of the eval set, and check whether previously passing cases still pass after a change.
  • Load the whole record into context, unretrieved — SIM gives the optimizer the full memory every round with no retrieval or truncation. Rejected. It works for one optimizer's notes, which stay small. A KB's error book, lessons and decisions grow without bound and compete with the context budget lint enforces (check 19). The brief already lists new lessons; the rest is read index-first when a task bears on it.
  • Machine-readable JSON for the error book, as SIM stores it. Rejected: every other KB record is Markdown that people edit and review, and nothing needs to parse attempts automatically yet.
  • Backfilling expected outcomes and lesson context on upgrade. Rejected: a prediction written after the result is hindsight, and a guessed context is worse than an honest "unknown".

Consequences

  • Recording a decision asks one more question (what it should change), and lint asks for an outcome when a review comes due. That is more friction; the review date above checks whether it is used.
  • The error book becomes longer and more useful only if lint really goes back to pending attempts. The skill says so; nothing in a script enforces it.
  • A KB that links others sees their lessons in its brief, which may be noise if the KBs differ a lot. Upstream lessons rank last and are only listed, never applied unasked.

Sources

  • https://arxiv.org/abs/2611.18856 — Yu et al., 2026-11-13, read 2026-11-18 (§3.2, §3.3, App. D.2–D.3, F.6)