Growth breaks the plateau

Theory update · 2026-06-28 · a growable substrate breaks the bpc plateau · experiment grow-then-prune

The prediction calibration found a flat spot. Our fixed voting bank improves to about a million characters and then stops: 2.234 to 2.112 to 2.109 bits-per-char, flat from one million on, while a plain order-six n-gram keeps falling. The n-gram keeps winning because it keeps growing, adding longer-context counts as the data arrives. So the question is sharp: if the plateau is a fixed-capacity artifact, does a growable bank break it? This builds the grow-then-prune the program wanted from the start: add a longer-context voting column when the held-out gain stalls, cap the count, prune the least-used. On the identical text8 split and the identical bits-per-char loop, the growable bank goes 2.234 to 2.004 to 1.875, growing 12 to 16 to 24 columns, where the fixed bank flattens at 2.11. Growth breaks the plateau. It scales like the n-gram (1.875 against the n-gram's 1.829) because it does what the n-gram does. The honest bounds stay: it reaches about the n-gram, it is still about 0.85 bits behind text8 SOTA, and prediction is not our headline axis. The deliverable is the capability: a count-native, online, bounded substrate that keeps learning from data instead of saturating.

The plateau, and why the n-gram does not have it

The benchmark scorecard put the model on a public yardstick and read an honest flat spot. Three learners on the identical text8 slices, the identical held-out bits-per-char loop at three scales: a plain order-six backoff n-gram, a gradient char-Transformer, and our voting bank. Our bank improves from two hundred thousand to one million characters, then stops. It reads 2.234, 2.112, 2.109, flat from a million on. The n-gram keeps falling, 2.222 to 1.991 to 1.829. The transformer keeps falling too. Past about a million characters we do not turn more data into better prediction, and both baselines do.

The diagnosis named the cause. Our bank is fixed capacity: twelve voting columns, each counting one view of the recent context, the same twelve at every scale. The n-gram is not fixed. As the data grows its context table grows with it, and a longer context is a sharper prediction, so it keeps converting fresh data into lower bits. Our twelve columns saturate. The n-gram does not, because it adds capacity.

So the plateau is a hypothesis about capacity, and it is testable. If the flat spot is a fixed-bank artifact, a bank that grows should break it. The program's redesign named the fix as grow-then-prune: grow wide, add a voting column when the current set stops paying, cap the column count, prune the least-used column when at the cap. This builds it and runs it against the fixed bank on the same yardstick.

The mechanism: grow when the gain stalls, cap, prune the least-used

The growable bank starts identical to the fixed one, the same twelve columns, so the only variable is the growth. It trains online in one pass, in chunks, and at each chunk boundary it asks a single question: has the current set of columns stopped buying much?

• Grow when the held-out gain decays. More data always nudges a count bank's bits down a little, so a "did it improve at all" trigger would never fire. The signal is deceleration: the windowed drop in held-out bits over the last few checks, against the best drop this column set ever bought. When the marginal return falls below a fraction of that best, the columns have saturated, and the bank adds one. The probe reads a held-out tail of the training data, so it decides when to add capacity, never the reported number, which is scored on a slice no arm ever trains on.
• Grow toward longer context. A fresh column comes from an ordered ladder of views the fixed bank lacks, ordered short to long. So as the bank grows it reaches for progressively longer look-back spans, exactly the capacity the fixed twelve-column bank does not hold, and exactly the mechanism the n-gram wins by. A new column starts naive and its counting brings it online, no retrain.
• Cap, and prune the least-used. The column count is capped. When the bank would add at the cap, it first prunes the least-used column, the one with the lowest recent duty cycle, the one that has earned its place the least, with a short grace period so a fresh long-context column is not evicted before its sparse contexts fill. Bounded capacity, the n-gram's growth with a memory budget.

It is online, count-native, gradient-free, single pass, the program's regime. The only new machinery is when to add a column and which to drop.

What happened

Growth breaks the plateau.

The numbers, all on the identical text8 split and the identical bits-per-char loop the head-to-head uses, held-out twenty thousand characters at each scale:

Read the deltas past a million characters, where the plateau lives. The fixed bank moves 2.112 to 2.109, a drop of 0.003, flat. The growable bank moves 2.004 to 1.875, a drop of 0.129, still falling hard. At ten million the growable bank reads 1.875 against the fixed bank's 2.109, 0.233 bits-per-char ahead, and it sits within 0.047 of the n-gram floor at 1.829.

The fixed bank plateaus and the growable bank does not. Same starting columns, same training data, same metric. The one difference is that the growable bank adds capacity when its columns saturate, and that one difference turns the flat spot into a curve that keeps dropping. The plateau was a fixed-capacity artifact, and growth dissolves it.

It grows the right thing. The bank reached for progressively longer-context raw views as the data arrived, its longest look-back span climbing from five to eight, exactly the longer context the fixed twelve-column bank lacks and the n-gram wins by. At two hundred thousand characters it correctly did not grow: too little data to saturate twelve columns, so the growable and fixed banks are identical there (both 2.234). Growth fired only where there was data to earn it.

The cap binds and the prune works. A second arm ran the same growth with pruning disabled. It is identical to the pruned bank at these scales, because the cap of twenty-four is not binding until ten million, where the bank reaches it and the prune engages. So the bound holds: the growable bank does what the n-gram does, adds longer-context capacity with data, but under a fixed memory budget, evicting its least-used view to stay there.

A growable voting bank breaks the bits-per-char plateau. Where the fixed twelve-column bank flattens at 2.11 past a million characters, the grow-then-prune bank keeps dropping, 2.004 to 1.875, growing 12 to 16 to 24 columns, ending 0.233 bits ahead of the fixed bank and within 0.047 of the n-gram floor. It grew the right thing (progressively longer context, the capacity the n-gram wins by), it did not grow where there was too little data, and the prune held the budget at the cap. The plateau was a fixed-capacity artifact, and growth dissolves it.

What this means

The benchmark scorecard's flat spot now has a cause and a cure. The cause is fixed capacity: twelve columns saturate while the n-gram's table keeps growing. The cure is to grow the bank the same way, add longer-context voting columns when the held-out gain stalls, under a cap, pruning the least-used. The growable bank stops being a flat line. It restores the n-gram-like scaling slope the fixed bank lacks, because it now does the n-gram's one trick: it adds longer-context views as data arrives.

That matters past prediction. A substrate that keeps learning from data instead of saturating is the thing the abstraction work needs at scale too. The fixed bank stopped converting data into anything; the growable bank does not. The capability, grow-then-prune, count-native, online, bounded, is the deliverable, and the bits-per-char curve is where it shows up first and most legibly.

The honest bounds stay sharp, and they are the same ones the scorecard carries. The growable bank reaches about the n-gram, 1.875 against 1.829, not past it. It is still about 0.85 bits-per-char behind text8 SOTA near 1.0. And prediction is not our headline axis. The program's home axes are the abstraction of the learned code (above the backprop ceiling, with no gradient) and continual learning under a budget. This result does not move those. What it moves is a different, real thing: it shows the substrate need not saturate, that a count-native bank can keep buying accuracy with data under a memory budget, the way the n-gram does and the fixed bank did not.

Growth breaks the plateau, and it does so by doing what the n-gram does: adding longer-context views as data arrives, under a cap, pruning the least-used. The plateau was a fixed-capacity artifact. The honest bounds: the growable bank reaches about the n-gram (1.875 against 1.829), it stays about 0.85 bits behind text8 SOTA, and prediction is not our axis. The deliverable is the capability, a substrate that keeps learning from data instead of saturating. This is one seed.

Lineage

Grew from the prediction calibration, which read the fixed bank's flat spot on a public yardstick (2.234 to 2.112 to 2.109, flat from a million on) and named it a bounded-capacity limit, and the empty cell, filled, whose confirmation queue listed this bits-per-char plateau test as one of the runs that had to clear before the sparse substrate enters the locked core. It reuses the calibration's exact text8 split and bits-per-char loop verbatim, and the voting substrate and the calibrated blend-norm readout verbatim, so the fixed-bank arm reproduces the published plateau number bit for bit and the growable arm is scored the same way.

Led to the standing reading that the substrate need not saturate: a count-native bank can keep converting data into lower bits under a memory budget, by growing longer-context views when the gain stalls and pruning the least-used at the cap. The next moves are the scaled multi-seed confirmation and carrying the same grow-then-prune capacity into the abstraction substrate, where keeping-learning-from-data is what scale needs.

Thread: online learning under a bounded budget, and prediction as a tracked-but-not-headline axis. A growable substrate that breaks a measured plateau by doing what the n-gram does, and a capability, grow-then-prune, that keeps the bank learning from data instead of flattening.