Fast, and at scale
2026-06-27 · an engineering-and-scale confirmation, corrected by a 10M run that reveals drift · the locked config
With the architecture locked, the next question was not a new mechanism but a stress test: is the winner fast enough to run, and does it stay itself at scale? A Metal-fast build of the locked depth-three stack (the whole strict-online step fused into one compiled function, all state on the GPU) runs at about 6,400 characters a second (roughly three-and-a-half to ten times the old rate) and is bit-identical to the reference. Run on a million characters it stays stable the whole way, with bits-per-char still dropping (3.90 → 3.63) while abstraction holds above the ceiling but does not climb: the cleanest confirmation yet that the locked architecture is stability-limited, not data-limited. Correction (added after a 10M run): that million-character stability turned out to be a window, not a long-run property. Run ten times further (seven million fresh characters against a fixed held-out slice) and the locked config drifts: held-out bits-per-char rises (to 3.765, forgetting the fixed distribution) and the deep-level code collapses toward rank-1 (
STABLE? False). So the honest scale story is two-sided: stable and improving in the ~1M window, but a long fresh, non-stationary stream drives forgetting and collapse. Skip to it.
What this round was
This is not a new idea. It is the locked architecture (the depth-three normalization-plus-gate-clip stack at a precision-gain of about 0.2, the one the sweep round settled above the backprop ceiling) put through the two tests every "we're done here" claim owes: a speed test (can it actually run at the scale the questions need?) and a long-run stability test (does feedback alignment, which loves to diverge, hold over a real stream?).
The architecture line had a standing diagnosis from its scaling step: the stack was stability-limited, not data-limited, depth saturated at three, and more data raised the risk of divergence rather than lifting abstraction (three swings after the keystone, and the sweep round). But those runs were short. A diagnosis like that has to be checked where it matters, on a long online stream, and that needs the model to be fast. So the two tests are really one job: build a fast version, then run it far enough to see whether the stability story holds and what more data actually buys.
The fast build
The locked stack was reimplemented as a compiled GPU kernel. The whole strict-online step (the depth-three normalized forward pass, the readout, the precision gate, the per-unit apical feedback-alignment credit, and the gate-clipped credit update) is fused into one compiled function, with all of the state (including the running precision average) kept on-device, so there are zero host syncs per step. It processes the stream in chunks of 64 strict-online steps, each step still seeing one character and updating once: the online contract is intact.
The first thing to check was that "fast" did not mean "different." It does not: the compiled build is bit-identical to the single-step reference (the largest weight difference is about one part in ten million, floating-point noise), and the qualitative result matches. At 18k characters transfer CCGP 0.521, above the 0.484 ceiling; at 100k the apex is L3 0.537, strictly rising L1 0.373 < L2 0.481 < L3 0.537. Same model, much faster.
How much faster: about 3.5× step-for-step (1,834 → 6,464 characters per second at the same depth-three configuration), and roughly 7 to 10× against the end-to-end sweep rate of a few hundred characters a second. That is the difference between "a scaling run is an overnight gamble" and "a scaling run is a coffee break": fast enough to actually ask the scale question.
What happened at a million characters
The headline is the 1M-character run, and it is the long-run stability test the architecture owed.
Across the entire million-character online stream the stack stays stable: no divergence, the dimensionality never collapses (it stays healthy, roughly 9 to 12, 4 to 5, and 5 across the three levels), and no NaN. The real long-run stability test passes.
This matters because stacked feedback alignment is exactly the thing the architecture line warned would blow up at scale: the forward activations run away, the credit is an outer product of those activations so it runs away too, and the precision gate amplifies the runaway. The locked stack's stabilizers (forward-activation normalization plus gate-clip) were shown to hold on short runs; this is the proof they hold on a long one.
And the two metrics move exactly as the diagnosis predicted they would, in opposite directions:
| metric | at 100k | at 1M | reading |
|---|---|---|---|
| bits-per-char (prediction) | 3.90 | ~3.63 | keeps dropping, still learning to predict |
| transfer CCGP (abstraction) | above 0.484 | above 0.484 (peak 0.593 @200k) | holds above the ceiling, does not climb |
More data makes the model a better predictor (bits-per-char falls from 3.90 at 100k to about 3.63 at a million) and leaves its abstraction flat: the transfer CCGP holds above the 0.484 backprop ceiling (best-level around 0.49 to 0.59, peaking at 0.593 around 200k) but does not rise with more data. That is the same wall the sweep round hit, now confirmed from the data side.
What 10M revealed: drift
The million-character run looked like the stability proof, and we filed it as one. Then we ran the same locked config ten times further (train on seven million fresh characters, hold out a fixed three million) and it does not hold. This is the honest correction that run owed: the "stable at scale" reading was a window, not a true long-run property. On a long fresh, non-stationary stream the locked config drifts.
Two things go wrong at once, and both are visible above:
| fresh chars (M) | held-out bpc | apex (L3) PR | the read |
|---|---|---|---|
| 1 | 3.549 | 2 | the bottom, the 1M "stable" window |
| 2 | 3.591 | 1 | already rising |
| 3 | 3.621 | 1 | |
| 4 | 3.705 | 2 | |
| 5 | 3.802 | 1 | worst |
| 6 | 3.690 | 1 | |
| 7 | 3.765 | 1 | STABLE? False |
Held-out bits-per-char bottoms near a million (3.549) and then rises with more fresh data (3.591, 3.621, 3.705, 3.802, 3.690, 3.765) not monotone, and clearly worse past one to two million. The model keeps adapting to ever-new regions of text8 and forgets the fixed held-out distribution. And the deep-level code collapses to rank-1 over the run: the participation ratio falls L1 11 → 5, L2 1 → nan, L3 2 → 1, the representation degrades to a near-degenerate code. The runner's own stability check, which the million-character run passed, now reads STABLE? False at seven million: the apex participation ratio has collapsed to 1.0 and bits-per-char is not bounded-improving. (The apex transfer CCGP wobbles down, 0.534 → 0.336; a best-level reading of 0.528 sits above the ceiling but its participation ratio is nan, so that number is a degenerate-code artifact, not a real abstraction.) Throughput stays healthy throughout, about 6,600 characters a second, so this is not a numerical blow-up; it is drift.
The lesson. A million characters was a window in which the stabilizers held and bits-per-char fell. It was not a guarantee about the long run. On a long fresh, non-stationary stream the locked config drifts: held-out prediction degrades (forgetting) and the deep code collapses. This is a continual-learning / non-stationarity limitation that only appears at scale, and it is distinct from the abstraction wall: the abstraction wall is about the ceiling on a short window; this is about staying put over a long one.
It reconciles with, and amplifies, the re-reading result. That experiment already saw this drift in miniature: its million-character fresh arm rose 3.916 → 3.987, while re-reading a bounded 200K stayed in a stable basin. The 10M run shows that drift is not a sidebar; it is the dominant long-run behavior of a fresh stream. Re-reading stays stable precisely because it sidesteps the fresh stream's non-stationarity. So the two findings line up: a bounded distribution, re-walked, settles; a long novel stream wanders, and at ten million the wandering wins.
The mitigations, the forward pointer. Every one of these is already in the program's vocabulary: re-exposure (re-reading a bounded corpus holds the model still); a consolidation / sleep cycle (the dream-replay idea, replaying a digest rather than only chasing fresh input); the count organ (which on the benchmark suite forgets about 21× less than a recency cache); or a forgetting-aware learning rule. Making the locked config stay stable on a long non-stationary stream is the next scale-and-architecture problem.
The verdict
The locked architecture is stability-limited, not data-limited, and it is fast. In the ~1M-character window the stabilizers hold and more data buys better prediction (bits-per-char down 3.90 → 3.63), not more abstraction (CCGP flat, already above the ceiling). But that window is not the long run: at 10M on a fresh, non-stationary stream the locked config drifts, held-out bits-per-char rises and the deep code collapses toward rank-1 (
STABLE? False). So the honest verdict is stability-limited at the ceiling and drift-limited on a long fresh stream: two different walls, both real.
This closes one loop and opens another. The sweep round argued the stack was stability-limited; the million-character run shows it, with the metrics splitting exactly as predicted: prediction improving, abstraction pinned at the ceiling. What it does not show, and what we wrongly implied, is that the winner "stays itself" indefinitely: the 10M run says it does not. The fast build is a genuinely deployable-speed learner; staying stable on a long non-stationary stream is now its own open problem.
The honest caveats
- No new mechanism: the architecture is the locked one, unchanged. What is new is that it runs fast, that it is stable in the ~1M window, and (the part we got wrong on first publish) that it drifts at 10M on a fresh stream. We file the speed and the 1M result as engineering plus a scale check, and the 10M drift as the honest correction that opens the next problem.
- Abstraction flat is not abstraction lost (within the window). In the ~1M window the code holds above the ceiling; it simply does not climb with more data. That is consistent with the lock: depth-three is already at the ceiling. Separately, on the long 10M stream the deep code does collapse toward rank-1. That is the drift, not the abstraction ceiling, and it is the new limitation.
- A million was a window, not the long run; the 10M run is the one that tells the truth about stability. We first staged the bigger runs as a victory lap (ten million in about eighteen minutes, the full hundred-million-character text8, a three-billion-character corpus, memory-mapped) and called the million-character run a passed stability test. The ten-million run says otherwise: held-out bits-per-char rises and the deep code collapses (
STABLE? False). The honest reading is that the million-character "pass" was a window, and long-run stability on a fresh non-stationary stream is unsolved.
The posts behind it
- The architecture is locked: the depth-three winner above the backprop ceiling that this run stress-tests.
- Three swings after the keystone: where "stability-limited, not data-limited" was first named, on the short runs this confirms.
- Re-reading: the bounded-corpus run that saw this same drift in miniature, and stays stable by sidestepping the fresh stream's non-stationarity.
- The benchmark suite: the four-angle scorecard the locked v1 model sits at the center of; the count organ's ~21×-less forgetting is one of the named drift mitigations.