What changed: replay at an interior altitude
Theory update. 2026-06-27 · the second anti-collapse mechanism · experiment M1 (Minsky)
On a long fresh stream the locked architecture drifts: bits-per-char rises and the deep code collapses. Replay (re-firing stored configurations) is the strongest patch for it. Minsky's K-lines say a re-evoked memory should be reinstated at an interior altitude, not the raw surface and not the top. This round makes that a sweep. Replaying at an interior level protects the deep code far better than replaying the raw input (deep dimensionality 5.04 against 1.99) and gives the lowest bits-per-char. Replaying at the top alone is too vague and craters the top's own abstraction; replaying everywhere at once over-consolidates and collapses. There is a sharp interior optimum.
The question
A 10M-character run revealed the locked architecture drifts. On a long fresh, non-stationary stream it does not stay itself: held-out bits-per-char bottoms out near a million characters and then rises, and the deep code collapses toward rank one. The drift battery found replay the strongest single patch. Replay re-fires a stored configuration (re-runs the network on a buffered input) and consolidates it again, so the model rehearses what it has already learned while it reads new text. It lowers bits-per-char and keeps the stack stable.
Minsky's Society of Mind says more about how to replay than "replay." A K-line is a wire that reactivates the set of agents that were on together when something was learned, so the mind rebuilds the state. And Minsky is specific about the altitude. Reinstate a memory too close to the raw input and it is over-specific, bound to surface detail. Reinstate it too close to the top and it is too vague, a label with no body. The useful reinstatement is at an interior altitude. We had a replay mechanism and a stack with levels. The question was whether the altitude of the replay matters the way the K-line theory predicts, and where the optimum sits.
What we tried
We took the replay mechanism from the drift battery and varied one thing: the altitude at which the replayed configuration is consolidated. The fresh online step is the identical locked hot path across every arm. The buffer is the same reservoir. A replay sample re-fires the stored configuration by re-running the locked forward, exactly as before. The arms differ only in where the consolidating update lands.
- Input. Consolidate the full locked step, the same as the drift battery's replay. The whole stack relearns the replayed input.
- Interior (the L2 band). Consolidate only the interior level. Reinstate the configuration at the middle altitude and nowhere else.
- Top only (the L3 band). Consolidate only the apex.
- Both (the dual band). Consolidate the interior and the top together.
The band-restricted update reproduces the locked math verbatim and writes only the level in the band. Everything ran on a long fresh text8 drift stream, three million characters per arm, the scale at which the baseline already drifts: by the two-million mark the no-replay baseline reads a rank-one deep code and the runner's own check says it is not stable. The headline numbers are held-out bits-per-char, the apex dimensionality (the anti-collapse read), and the transfer CCGP at the best level.
What happened
| three million chars, final checkpoint | bits-per-char | apex dimensionality | best-level CCGP | stable |
|---|---|---|---|---|
| baseline (no replay) | 3.679 | 2.39 | 0.540 | no (rank one at 2M) |
| input replay | 3.607 | 1.99 | 0.548 | yes |
| interior (L2 band) | 3.582 | 5.04 | 0.530 | yes |
| top only (L3 band) | 3.676 | 2.34 | 0.173 at the top | yes |
| both (dual band) | 3.769 | rank one | collapsed | no |
There is a sharp interior optimum: interior far ahead of input, which ties the baseline, ahead of top-only, ahead of dual.
The interior altitude protects the deep code. Replaying at the interior level reads an apex dimensionality of 5.04, the healthiest by far, against input replay's 1.99 and the baseline's 2.39, and it gives the lowest bits-per-char in the table, 3.582, beating both input replay and the baseline. Reinstating the configuration at the middle altitude keeps the deep code from collapsing and predicts better. The interior optimum holds across every checkpoint, not just the end: the apex dimensionality reads 6.68, then 5.19, then 5.04 across the run, while input replay sits at 2.72, 2.30, 1.99.
The top alone is too vague. Consolidate only the apex and the apex's own transfer CCGP craters to 0.173, and bits-per-char is no better than the baseline. Reinstating only the top degrades the top. This is Minsky's "too vague" with a number.
Both at once over-consolidates. Consolidate the interior and the top together and the stack collapses: the apex goes rank one, stability fails, and bits-per-char is the worst of any arm at 3.769. Replaying at every level at once is too much consolidation and tips the stack over.
The lesson
Replay protects the deep code best when it reinstates the configuration at an interior altitude, not at the raw input and not at the top. Interior replay reads a deep dimensionality of 5.04 against input replay's 1.99 and gives the lowest bits-per-char (3.582). Replaying at the top alone is too vague and craters the top's own abstraction (0.173); replaying everywhere at once over-consolidates and collapses. Minsky's K-line altitude is real, with a sharp interior optimum.
This is the second anti-collapse mechanism this round turned up, and it works on a different axis from the first. The lateral-inhibition result fights the collapse across columns within a level, on a bounded corpus. This one fights the drift collapse across time at an interior level, on a long fresh stream. Two wires, one enemy: the deep code falling to rank one.
The honest caveats
- The win is dimensionality and prediction, not the abstraction ceiling. The interior band's win is on the apex dimensionality (the anti-collapse read) and on bits-per-char. Its deep transfer CCGP (0.530) ties input replay's (0.548), and no arm clears the 0.484 backprop ceiling by much at this checkpoint. The CCGP is noisy at single checkpoints. So interior replay sharpens the representation-health and prediction axes of replay, not the abstraction-ceiling axis.
- Three million characters, not the full ten. The drift the baseline shows by two million is real and the interior optimum is consistent across all three checkpoints, but this is a trimmed run, not the full ten-million stream where the drift was first found.
- One mechanism, one stream. It varies the altitude of one replay mechanism on one drift stream. It shows the interior altitude is the right place to reinstate; it does not yet pit it against the other anti-collapse wires on the same test.
The frontier
Two anti-collapse mechanisms now exist, and neither has met the other. Lateral inhibition across columns and interior-altitude replay across time both fight the rank-one collapse that caps the architecture, from different directions. The next question is whether they compose. The drift result also sharpens the continual-learning frontier: the locked config drifts on a long fresh stream, and the strongest known patch is a K-line replayed at an interior level. Whether that holds the deep code stable all the way to ten million, and whether it leaves the abstraction ceiling alone or finally lifts it, is the next run.
Lineage
Grew from the scale correction, whose 10M run revealed the locked config drifts on a long fresh stream (bits-per-char rising, the deep code collapsing), naming replay among the mitigations; and from the drift battery, where input-level replay was the strongest single patch this round refines by altitude.
Thread: the collapse that caps abstraction, fought across time. The mechanism is Minsky's K-lines (reactivate a learned configuration, at an interior altitude); the frontier is continual learning under a long non-stationary stream.