Re-reading
2026-06-27 · a free, rule-legal lever on prediction · experiment SX
The locked architecture is stability-limited, not data-limited: more fresh text buys prediction, not abstraction. This round asks the other half of that question: what does re-reading the same text buy? Re-reading a bounded corpus (five online passes, no reset, fully within our online rule) lowers held-out bits-per-char (3.863 → 3.762) with no overfit, and does not touch abstraction. And the head-to-head is the surprise: 200K read five times beats a million fresh characters on held-out prediction (3.762 vs 3.987), because the fresh single pass drifts at depth three while re-reading stays in a stable basin. Exposure beats novelty for prediction, and ties for abstraction.
What this round was
The scale run settled one axis: feed the locked depth-three stack more novel text and it predicts better but does not abstract more. It is stability-limited, not data-limited. But "more compute on the same data" has a second form people use all the time and the program had not tested: not more new text, but the same text again. A child re-reads the same picture book a hundred times. Spaced re-reading is how anyone consolidates a bounded text. So the question SX asks is the complement of the scale run: does re-reading a fixed corpus (multiple online passes, no reset) buy what fresh data could not?
A note on the rules, because it is the whole reason this experiment is allowed. Our first law is online, gradient-free learning: each datum changes the model on arrival, single pass, no backward pass over the data. Re-reading is not a violation of that. Reading a sentence again is still one online, single-datum, gradient-free update; it is not a gradient epoch, the batch revisit that SGD does and that the rule forbids. Re-exposure is exactly the kind of "more compute on the same data" the rule permits, which is what makes it a free lever worth testing.
The experiment
The locked config (depth-three normalization-plus-gate-clip at gain 0.2), two arms at a matched total of a million characters:
- Re-exposure: fix a 200K-character training slice and stream it five times with no reset (the weights and the running precision average persist across passes), online throughout. Evaluate after each pass.
- Fresh: stream a million novel characters once (matched total = 200K × 5), same config, evaluated at matched checkpoints.
The discipline that makes the number trustworthy: everything is evaluated only on disjoint held-out and transfer slices, text neither arm ever trained on. So the number is generalization, not memorization. The training-slice bits-per-char is kept too, but only to read the overfit gap, the distance between how well the model does on text it has seen five times and text it has never seen. Two seeds.
What happened
The head-to-head is clean, and it goes the way that is interesting rather than the way that is obvious.
| arm | held-out bpc | overfit gap | transfer CCGP | the read |
|---|---|---|---|---|
| re-exposure (200K × 5) | 3.863 → 3.762 (Δ−0.101) | flat (+0.029 → +0.034) | ~0.49 to 0.59, flat (0.563 → 0.528) | generalizes, no overfit |
| fresh (1M novel) | 3.916 → 3.987 (rises) | balloons (up to +0.14) | ~0.41 to 0.54 | drifts, mild forgetting |
| head-to-head @ 1M | −0.225 for re-exposure (3.762 vs 3.987) | n/a | tie (0.528 vs 0.541) | exposure wins prediction, ties abstraction |
Four answers fall out:
- Each re-pass lowers held-out bits-per-char. 3.863 after the first pass to 3.762 after the fifth: a real generalization gain (it is measured on text the model never trained on), not just a tighter fit to the training slice. Most of the drop comes on the first re-read, and it plateaus by about the third pass.
- No overfit. The gap between training-slice and held-out cost stays flat across all five passes (+0.029 to +0.034). Re-reading the same 200K five times does not make the model memorize it at the expense of generalizing: the count-shaped learner does not overfit the way a gradient net would.
- Abstraction does not move. Transfer CCGP holds above the ceiling but does not rise with re-exposure (0.563 → 0.528, within noise). Re-reading buys what fresh data could not, but only for prediction. This confirms the scale verdict on the untested axis: the stack is stability-limited, not exposure-limited either.
- 200K read five times beats a million fresh characters on held-out bits-per-char by a wide margin (3.762 vs 3.987), at every checkpoint, and ties on abstraction.
The drift insight
That last result is the one worth pausing on, because the obvious expectation is the opposite: a million different characters is five times the information of one 200K slice, so surely the fresh arm should predict held-out text better. It does not, and the reason is the locked architecture's own character.
The fresh single pass drifts. Its held-out bits-per-char rises over the run (3.916 → 3.987): the model is mildly forgetting the fixed early held-out slice as it adapts to later text, and the overfit gap balloons as it goes. Re-reading a bounded corpus, by contrast, keeps the model in a stable basin: it is not chasing a moving distribution, so it settles rather than drifts.
So re-exposure's prediction win is partly a stability effect, not pure exposure: re-reading a small corpus holds the model still, while skimming a larger one once lets it wander. This is cognitively faithful (spaced re-reading of a smaller text generalizes better than a single skim of a larger one) and it lines up with the scale run from the opposite side: fresh data and re-reading land on the same abstraction ceiling from two directions, one by drifting up to it and one by settling at it.
The verdict
Re-reading a bounded corpus is a free, rule-legal lever on prediction with no overfit cost: held-out bits-per-char falls 3.863 → 3.762 over five passes, and 200K read five times beats a million fresh characters (3.762 vs 3.987). But it does not touch the abstraction wall, and it does not argue that the single-pass rule leaves abstraction on the table. Exposure beats novelty for prediction, and ties for abstraction.
It belongs beside the scale run, not on the abstraction line. Both say the same thing about the locked architecture's ceiling: more compute, whether fresh or re-read, improves prediction and leaves abstraction where it is. The new, useful piece is that re-reading is a cheap prediction win a single-pass learner can take for free, and the drift insight: a fixed corpus re-walked stays put, where a novel stream wanders.
The honest caveats
- It is a prediction lever only. The gain is real and free, but it is bits-per-char, not abstraction. We are not claiming the single-pass rule costs us an abstract code; it does not, abstraction is pinned at the ceiling either way.
- The win is partly stability, not pure exposure. Some of re-exposure's edge over fresh data is that re-reading holds the model in a stable basin while a fresh single pass drifts. The two effects are entangled here, and disentangling them is a follow-up.
- Bounded corpus, two seeds, the locked config. This is the locked depth-three stack on a 200K slice; the drift-vs-settle story is a property of this architecture at this scale, not a universal law.
The posts behind it
- Fast, and at scale: the scale run that found the locked stack stability-limited on fresh data, which this round tests on the re-reading axis.
- The architecture is locked: the depth-three winner above the ceiling that both runs stress-test.
- What survives a budget: the bounded-memory generalization thread, what a fixed corpus re-walked settles into.