Starting small, on purpose

Theory update. 2026-06-26 · honest negative · experiment AK

Growing the memory budget on a schedule does not beat starting at full memory. "Starting small" was a property of the gradient optimizer, not of learning: a count learner cannot get stuck, so it needs no curriculum, only enough final memory.

The question

Elman's 1993 result is one of the prettiest in the field: a recurrent net learned complex embedded structure only if it started small, staged simple-to-hard, or its own memory started short and grew. Thrown the full problem at full capacity from step one, it failed to find the long-range structure at all. Vygotsky's zone of proximal development says the same from the other side: spend effort at the edge of the masterable.

Our substrate already runs on a bounded memory budget. So the sharp question is whether growing that budget on a schedule is itself a curriculum that beats a fixed one, and it stages memory, not data, which makes it the count-native, teacher-free version of starting small. If it worked, it would be a free win: schedule the budget and acquire long-range structure a fixed budget misses.

What we tried

Each context's next-token row is a leaky accumulator with a leak-horizon H, the effective number of recent visits that still count. We put H on a schedule. GROW raises it linearly 3→600 over the single pass; FULL holds 600 from the first character; FIXED holds 3 throughout. The test corpus is Elman's embedded-agreement paradigm made count-native: 120 distinct keys, each permanently bound to a number class, with the agreement target emitted after a clause of varied filler. The dependency is reachable by an order-8 char context but each specific key run context recurs only about every 120 sentences, so a short horizon decays a key's class to nothing before its next visit (chance), while a long horizon accumulates it across sparse visits (the agreement). Whether the dependency is acquired turns entirely on the leak-horizon, the exact dial the schedule moves. Single pass, orders 1 to 8, scored as perplexity on the cue-distant agreement token.

What happened

FULL edges GROW by a hair and never loses, and it holds across seeds 1, 2, and 3, and across a slower quadratic schedule, which made GROW strictly worse, the opposite of what a curriculum should do. Elman's starting-small effect does not reproduce for a count learner. But a permanently small budget loses decisively: FIXED is +30% perplexity on the long-range token. So the bounded-memory regime is real and the leak-horizon is load-bearing: the dependency genuinely is accumulation-bound. The schedule simply is not the lever. What matters is the final horizon; reaching it sooner is, very slightly, better.

The reason for the asymmetry with Elman is the whole point. His net failed full-from-start because gradient descent on a high-capacity net locks in a bad early solution it cannot climb out of; starting small kept it plastic until the easy structure was in place. A count model has no gradient to lock: counts are additive and self-correcting, and an early noisy high-order count is simply outvoted by later evidence, never frozen. There is nothing for a growing schedule to rescue. The ZPD overlay, down-weighting "mastered" contexts to spend effort at the edge, only hurt (−5.8%): throttling the count you add to a confident context is just discarding the sparse evidence those keys must hoard to survive the leak.

The lesson

Growing the budget ties full-from-start; only a permanently small budget loses. "Starting small" was a property of the optimizer, not of learning itself: gradient descent freezes early guesses, so it needs a gentle on-ramp; a count learner cannot get stuck, because counts are additive and self-correcting, so it needs no curriculum, only enough final memory. The bounded-memory rule stands. The scheduling of it is a no-op.

This is a clean transfer failure, and a useful one. It says the curriculum literature is solving a problem the count substrate does not have. We keep the budget; we drop the schedule; we make sure the final horizon is large enough for the dependencies we care about, and stop there.

Lineage

Grew from how big a brain the data wants, which set the question of how capacity should grow with the data, and from learning the new without losing the old, the leaky-accumulator budget this experiment schedules.

Thread: online learning, and the memory budget. The keeper is a clarifying negative: final memory matters, scheduling does not. The ideas are Elman's "starting small" and Vygotsky's zone of proximal development.