The shape of forgetting

2026-06-26 · honest negative · experiment AI

The power law is the right shape of forgetting (it is the only curve that represents spacing) but as an eviction policy for dense char-grams it loses to raw counting at every budget, because counting is its own limiting case.

The question

The strongest empirical result in the cognition literature is Anderson and Schooler's: human memory accessibility tracks the recency, frequency, and spacing statistics of the real environment, and the single curve that fits all three is a power law, not an exponential. ACT-R writes it as the base-level activation of a memory chunk, B = ln Σ (t − tₖ)^(−d): frequency adds terms, recency weights them, decay d is the leak, and need-odds rise as exp(B).

Our substrate weights memory by raw count, or by an exponential leaky use-score. Both are cruder than the power law. So the question was sharp: if we give the count model a per-context activation that is the ACT-R curve and use exp(B) for both prediction and eviction, does the better-shaped memory pay off under the one condition that makes memory pressure visible, a fixed budget?

What we tried

Char-level, orders 1 to 5, decay d = 0.5, on darwin.txt, single streaming pass. The activation is the incremental Petrov-Anderson approximation of B: a per-entry O(1) recurrence, one extra float, no stored timestamps. Three probes. The spacing effect: one motif shown 20 times, massed versus spaced, same total count, measured by final retrieval weight. Eviction quality: cap each order's table and let four policies (power-law, LRU, LFU (raw count), EMA) differ only in what they drop on overflow, scored by held-out bits-per-char. And a domain-shift variant, darwin then a shakespeare flood, evaluated back on darwin, the non-stationary regime where recency should bite.

What happened

The spacing effect is real, and the power law is the only curve that captures it:

Spaced is 8.96× more accessible than massed at equal frequency: the Anderson-Schooler curve, in the substrate. The EMA's larger ratio is not a win but a collapse: after the long gap it decays the massed motif to near zero, so its ratio only divides by near-nothing. The power law keeps the massed motif retrievable while still preferring spaced. An exponential cannot represent spacing; it only remembers the last touch.

But on eviction under a budget, raw counting wins at every cap:

LFU wins at every cap, and the same holds under the domain shift. The power law is a decisive second (it beats EMA and LRU everywhere by 0.3 to 1.0 bpc) but it never beats counting. And the reason is clean, not noise: LFU is the power law's d→0 limit. A decay sweep shows quality degrading monotonically as d grows (1.899 at d=0.05, 1.918 at d=0.5, 2.048 at d=0.9). Every bit of recency the power law adds over pure frequency is, here, a tax. A char-gram's predictive value is almost entirely its total count: at order ≤5 over one English corpus there is no "useful last week, stale now" structure, so recency adds variance without signal. Using exp(B) to weight prediction was worse still, +0.68 bpc over clean highest-order backoff (2.479 vs 1.802), a need-odds blend repeating the mistake the voting post already named.

The lesson

The power law is the right shape of forgetting (graceful on spacing where the exponential collapses) but wrong for this place. As a budgeted-eviction policy for dense char-grams it loses to plain LFU, because LFU is its d→0 special case and these contexts' value is pure frequency. Keep raw-count LFU for char-grams. Reach for ACT-R's power law only where recency and spacing carry signal that frequency does not: sparse, non-stationary, genuinely-spaced memory, word- and concept-level retrieval, not dense char-grams.

The negative is a scope statement, not a dead end. The curve is right; we were holding it to the wrong altitude. Spacing is a property of memories that recur sparsely over long gaps, and a char-gram is the opposite: it recurs constantly, so its count already ranks its usefulness, and any decay on top is just thrown-away evidence.

Lineage

Grew from learning the new without losing the old, which made the memory budget load-bearing and the leaky use-score the thing to beat; what an agent learns while it dreams, the other place a bounded memory decides what to keep; and what survives scale, which set the rule that a budget re-elevates what unbounded data buries.

Thread: online learning, and the memory budget. The keeper is a scope line: LFU for char-grams, the power law reserved for the word and concept levels where spacing is real. The idea is Anderson and Schooler's rational analysis of memory.