Gentle learning compounds

2026-06-28 · the learning rate was too aggressive · experiment SX-LR

Re-reading helped prediction, but only a little, and then it flattened. The question this round asks is whether the flat spot was a property of repetition or a property of the learning rate. The answer is the learning rate. The locked rate (0.005) crams the slice on the first pass and then has almost nothing left to give. Drop the rate to 0.001 and repetition keeps paying across all ten passes, to a genuinely lower floor: held-out bits-per-char reaches 3.637, a quarter of a tenth of a bit below the aggressive baseline's 3.762. The optimum is a U: 0.005 is too aggressive, 0.001 is the sweet spot, 0.0005 is too gentle. A gentler rate lets repetition compound where the aggressive rate stalls.

The question

The re-reading round found a real but modest result: re-reading a bounded slice lowers held-out bits-per-char, with no overfit, and beats fresh data at matched compute. The honest read was "repetition helps a little, then plateaus by about the third pass." Most of the gain landed on the first re-read and the curve went flat after.

That plateau invites a hypothesis. The re-reading run used the locked learning rate, 0.005, the aggressive default the architecture sweeps settled on. An aggressive rate takes a large step per update, so it fits the slice fast and then has little headroom: it commits hard on the first pass, and the later passes find nothing new to do. If that is what the plateau is, then it is a property of the rate, not of repetition. A gentler rate would take small steps, so each pass would still move the model, and over many passes it would settle into a lower floor than one big step ever reaches.

So the test is a sweep: hold everything in the re-reading experiment fixed and vary only the apical learning rate, across one, five, and ten passes. Does a gentler rate make repetition compound?

The experiment

The re-reading harness, unchanged. The locked depth-three stack (normalization-plus-gate-clip at gain 0.2), a fixed 200K-character training slice re-read up to ten times with no reset (the weights and the running precision average persist across passes), online throughout. Everything is evaluated on a disjoint held-out slice the model never trains on, so the number is generalization, not memorization. Two seeds.

The only knob that moves is the apical learning rate, swept over four values: 0.005 (the locked, aggressive default), 0.002, 0.001, and 0.0005. Re-reading stays inside the online rule for the same reason it always did: reading the slice again is one online, single-datum, gradient-free update each pass, not a gradient epoch.

What happened

The sweep is clean, and it confirms the hypothesis.

Four answers fall out.

1. A gentler rate reaches a lower floor. The best cell is 0.001 at ten passes, 3.637 bits-per-char, against the aggressive baseline of 3.762 (0.005 at five passes, the re-reading round's number). That is 0.125 bits better at matched exposure. Every gentle cell beats the baseline.
2. The optimum is a U. 0.005 is too aggressive (it crams and stalls), 0.001 is the sweet spot, and 0.0005 is too gentle (it underfits: it starts highest at 3.905 and plateaus highest at 3.720). The best rate is five times gentler than the locked one, and going gentler still loses ground.
3. Repetition compounds under the gentle rate. The aggressive rate drops 3.863 to 3.779 on the first re-read and then runs nearly flat, last-pass change about a thousandth of a bit. The gentle rate descends smoothly across all ten passes, 3.749 to 3.685 to 3.673 and on down to 3.637, and only flattens by pass ten. The aggressive rate crams; the gentle rate compounds.
4. No overfit, and gentler is tighter. The gap between training-slice and held-out cost stays flat for every cell, and it is smaller at the gentler rates: at ten passes the gap is 0.045 bits at 0.005 against 0.030 at 0.001 against 0.017 at 0.0005. Re-reading the same 200K up to ten times with a gentle rate does not memorize it. The count-shaped learner does not overfit, and the gentle rate generalizes tightest.

Why the aggressive rate stalls

The shape of the two curves tells the whole story. Picture the model settling toward a basin on the held-out distribution. One large step per update jumps most of the way on the first pass and then overshoots back and forth in place: the later passes cannot refine what one big step already overshot. One small step per update walks toward the basin gradually, so every pass moves it a little closer, and ten small steps land deeper than one big one.

That is what the trajectories show. The 0.005 curve flatlines by the third pass with its work done and its floor set. The 0.001 curve is still gently descending where the 0.005 curve went flat, and it settles a quarter of a tenth of a bit lower. Repetition was never the weak ingredient. The rate was too hot to let repetition do its work.

The broader implication

This is the part worth carrying past the re-reading question. The rate that turned out too aggressive, 0.005, is not a re-reading setting. It is the locked global learning rate of the apical stack, fixed by the architecture sweeps and used everywhere the stack runs. Re-reading is just where its cost became visible, because re-reading is the one place the model gets a second look at the same data and a too-large step shows up as a stalled curve.

If 0.001 is 0.125 bits better than 0.005 at matched exposure here, the aggressive rate is likely costing the apical stack on every run, not only on re-reading: the scale runs, the abstraction stack, the locked depth-three verdict. The honest conclusion is narrow and load-bearing at once. The global apical learning rate should be revisited. The value the architecture locked is too hot.

The verdict

The re-reading plateau was a learning-rate artifact, not a ceiling on repetition. A gentler rate (0.001, five times gentler than the locked 0.005) lets re-reading compound across all ten passes to a lower held-out floor, 3.637 against the aggressive 3.762, 0.125 bits better, with no overfit and a tighter generalization gap. The optimum is a U, with 0.001 the sweet spot. And the rate that proved too aggressive is the stack's global rate, so the implication reaches past re-reading: the locked learning rate is too hot, and should be revisited.

The honest caveats

• It is a prediction lever only. The gain is real and free, but it is bits-per-char, not abstraction. Transfer CCGP stays noisy around 0.4 to 0.59, flat, above the 0.484 ceiling, and does not rise as the bits drop. This is the same prediction-versus-abstraction split the rest of the program keeps finding: a gentler rate predicts better and abstracts the same.
• The absolute gain is modest. A quarter of a tenth of a bit over five passes at the gentle rate. Real, repeatable, and small.
• It is the apical learned stack, not our best predictor. Our best predictor is the count voting bank, near 2.1 bits-per-char at scale, and that bank is re-reading-invariant: a count table is complete after one pass, and it has no learning rate to tune. So this does not move the headline. The 2.1 plateau is broken by a growable bank, not by a learning rate.
• Bounded corpus, two seeds, one harness. A 200K slice on the locked stack. The "revisit the global rate" implication is a strong pointer from one axis, not yet a re-sweep of the whole stack.

The posts behind it

• Re-reading: the re-exposure result this refines, where re-reading first showed as a modest, plateauing prediction lever.
• The benchmarks: the with-repetition axis on the scorecard, where re-reading reaches the lower 3.637 floor at the gentler rate.
• The architecture is locked: the depth-three winner and the sweeps that set the global rate this round questions.