What changed: the architecture is locked

Theory update · 2026-06-27 · the architecture is locked · experiments DI / DJ / DK / A1

DH left the abstraction stack with a named wall: stability. Stacked feedback alignment grew its forward activations unbounded, the precision gate flipped from hero to destabilizer at depth, and the deep code collapsed even where it stayed alive. This round runs four parameter sweeps to map that frontier, and the verdict is decisive. Stability is solved: forward-activation normalization holds the stack stable everywhere the baseline, a gate-clip alone, and a dream-replay cycle alone all diverge. The four levers factor cleanly: stability, deep dimensionality, abstraction, and prediction each come from a different knob. The winner is a depth-three stack tuned to a precision-gain of about 0.2, reaching a transfer abstraction score of 0.50 to 0.55, above the backprop ceiling of 0.484, stable and seed-robust. And the capstone that was supposed to push it deeper (combining the stability lever with the breadth lever) ran and failed: the two levers act on different axes of variance and do not add. The architecture is locked at the depth-three winner, above the ceiling. With abstraction settled, the program's open value moves to generation.

What this round was

The apical fusion gave the program its constructive answer to abstraction, and scaling it found its ceiling and its wall. Three levels of depth, an apex nearly at the backprop mark, then saturation, divergence with more data, and a precision gate that destabilized the stack at depth. DH named the fix in one word: a real stability mechanism, then re-run the sweeps.

So this round is not a new idea. It is the careful sweep that turns a hand-tuned, fragile result into a settled one. Four parameter sweeps, all on the MLX substrate, strictly online, measuring within-slice and held-out transfer abstraction at every cell. The question for each: which knob fixes which limiter, and does combining them break the ceiling?

The four sweeps

Stability, and it is solved. The stability sweep asked, across depth, learning rate, and data, which mechanisms keep the stack from diverging. The map is unambiguous:

Forward-activation normalization is the stabilizer: necessary and sufficient, stable at every depth, rate, and data size where everything else diverges into the teens and thirties of bits-per-char. Bounding the gate alone does not stabilize; the dream-and-replay consolidation cycle alone does not stabilize either, a clean standalone negative for the replay idea, run on its own and kept honest.

But stability is not abstraction. Plain normalization stays stable yet collapses the deep code to one dimension: its high apex score is a degenerate-code artifact, the same trap flagged before. Adding the gate-clip keeps the code healthy: at depth three, normalization-plus-gate-clip reaches transfer CCGP 0.508 to 0.549, above the 0.484 backprop ceiling, with dimensionality 4 to 7 and stable bits-per-char, on both seeds. Both pieces are needed: normalization for stability, the gate-clip for an abstraction that is real rather than collapsed.

The precision-gain dial has a sharp peak. A companion sweep over the gain that scales the apical credit found a unimodal curve with a knife-edge optimum. A gain of 0.1 lands just under the ceiling at 0.449; a gain of 0.2 hits the peak, about 0.50 to 0.55, above the ceiling, seed-robust (0.508 and 0.498 at a hundred thousand characters); and a gain of 0.3 or more over-drives the stack and the deep code dies, collapsing to one dimension at about 0.30. Plain normalization at every gain stays collapsed (confirming the gate-clip is essential) and depth four collapses at every gain. The dial fixes depth-three abstraction precisely; it cannot push it deeper.

Dropout decorrelates but does not stabilize. A "tired connections" sweep (randomly dropping units, synapses, or apical wires) found that moderate dropout, around 0.1 to 0.2, lifts abstraction, decorrelates the code, and raises dimensionality (depth-three apical-wire dropout at 0.1 reaches 0.454 against a 0.385 baseline; depth-two unit dropout at 0.2 reaches 0.419 against 0.356). Dropping the apical credit wires is best at depth three, where the collapsing apex needs decorrelating. But dropout does not fix scale-divergence: at a hundred thousand characters the baseline still diverges and dropout only softens it. So dropout is a decorrelation and abstraction lever, not a stability mechanism. The user's hunch (tired connections buy reliability) holds for abstraction, not for stability. (Zero dropout is bit-identical to the base stack: a clean control.)

Breadth keeps the code high-dimensional. A breadth sweep ran many parallel column-lanes, each its own apical stack, probed on the dimensionality-preserving concatenation rather than a consensus bottleneck. Breadth keeps the code high-dimensional: concatenated dimensionality scales with the number of lanes (depth-two: 16 to 50; depth-three: 4 to 32), fixing the single-column deep-collapse. At depth two breadth also improves prediction (bits-per-char 4.01 to 3.45 as lanes grow) while holding abstraction above the ceiling, a clean win. At depth three it fixes the dimensionality collapse and holds abstraction but prediction degrades. At depth four it raises dimensionality but mostly diverges: this sweep had no normalization, so breadth alone does not stabilize. Same-view lanes behave like a single column: different views are what matter, confirming the diversity finding.

The synthesis: the levers factor

This is the load-bearing result, and it is clean. The four knobs are complementary, each fixes what the others lack:

Stability comes from forward-activation normalization, with the gate-clip. A deep, high-dimensional code comes from breadth, and from moderate dropout. Abstraction comes from per-unit precision-gated apical credit at a gain of about 0.2. Prediction comes from breadth at depth two and diverse views.

Four jobs, four knobs, no two doing the same work. The best single configuration that fell out: normalization-plus-gate-clip, precision-gain about 0.2, depth three, transfer CCGP 0.51 to 0.55, above the 0.484 backprop ceiling, stable, seed-robust. The hand-tuned, fragile result of the keystone is now a settled one.

That left exactly one prediction to test. Two of the four levers looked like each other's fix: normalization-plus-gate-clip is stable but collapses the deep code; breadth keeps the code high-dimensional but diverges deep. So the untried combine (normalization-plus-gate-clip times breadth, the "A1" capstone) should give a stable, high-dimensional, above-ceiling code at depth, and lock the architecture. It looked cheap and high-confidence. We said so, and we flagged that it had not been run.

The capstone, and the honest negative that locks it

A1 built that combine, bit-faithfully (a single lane reproduces the stability winner exactly) and tested it. The prediction is false.

It passes the first gate: a single lane at depth three reaches transfer CCGP 0.502, in the winning band, stable. It fails the second: four lanes at depth four stay stable (so the combine does fix breadth's deep divergence) but the concatenated dimensionality collapses to 1.17, not the high-dimensional code the whole capstone was for. The combine does not preserve deep dimensionality.

Three controls diagnose exactly why, and the answer is precise: the two levers act on different axes of variance.

• Normalization pins every level's activation magnitude, which drives the deep layers to a stimulus-invariant fixed point: rank-one across stimuli. The stability winner alone, single-lane, depth-four, collapses to rank-one in its deep levels.
• Breadth alone, without normalization, keeps each lane high-dimensional: the concatenation stays rich.
• The A1 combine has each lane independently collapse to rank-one across stimuli. The lanes decorrelate from each other (cross-lane decorrelation 0.977) but every lane is internally rank-one, so concatenating four rank-one lanes is still, essentially, rank-one.

The key distinction: breadth preserves dimensionality across columns; normalization destroys it across stimuli within each column. Concatenation cannot undo a per-lane, cross-stimulus collapse. The round's prediction (breadth rescues normalization's deep collapse) was wrong on the variance axis. It is the honest negative that locks the verdict.

The verdict

The architecture is locked at the depth-three winner (normalization-plus-gate-clip, precision-gain about 0.2) at a transfer abstraction score of 0.50 to 0.55, above the 0.484 backprop ceiling, stable and seed-robust. Stability is solved by forward-activation normalization. The four levers factor and are complementary. But they are not additive: abstraction does not climb past depth three by combining the existing knobs, because breadth and normalization operate on different axes of variance. Pushing deeper needs a genuine design change, of uncertain payoff. We are already at the ceiling.

This is the end of the abstraction line as a possibility question. It is no longer "can a biologically-shaped, online, local, bounded architecture build an abstract code?" (DC through DE answered yes) nor "how far does it scale?" (this round answers: to depth three, above the backprop ceiling, stably). The residual is a design problem with diminishing returns: pushing past depth three would need a soft forward bound that prevents divergence without pinning the activation magnitude, or normalization on the top level only, or an ungated dimensionality-preserving regime with only a soft bound. And since we are already at the ceiling at depth three, deeper may add dimensionality without adding abstraction.

So the program makes a deliberate call. The abstraction architecture is settled, above the ceiling, and the marginal return on pushing it deeper is low. The open value has moved to generation.

The honest caveats

We keep them, because the claim is strong.

• The capstone is a negative, and we led with the wrong prediction. The previous round called A1 cheap and high-confidence; it ran and failed. We correct it in full rather than quietly drop it: the levers are not additive, and the diagnosis (different variance axes) is the real keeper.
• Some cells are one or two seeds. Several depth-three and depth-four cells in the sweeps are thin; the headline winner (depth-three, gain 0.2, above ceiling) is the seed-robust one, but the deeper cells are not all replicated.
• Above the ceiling, but it is the one-layer backprop ceiling. The 0.484 mark is single-layer online backprop on this harness. Beating it is a real result for a gradient-free local stack; it is not a claim against deep backprop in general.
• Depth-four-and-beyond dimensionality is the open residual. Solved divergence did not solve the deep collapse past three. That is the design problem of uncertain payoff, not a closed door, recorded for if and when it is revisited.
• Still one architecture family, one probe. The MLP stack and the CCGP abstraction score, as throughout the line.

The frontier, moved

With abstraction locked above the ceiling, the program's open frontier is no longer the abstraction wall. It is generation: producing text, not just modelling it. The production library named the external half of that (an audience model, a referential game), and a sibling line names the internal half. The next post lands the first positive there.

The posts behind it

• Three swings after the keystone: the label-attractor negative, the voting-factors-prediction finding, and the scaling step that named stability as the wall.
• The apical fusion that makes abstraction climb: the keystone this round stabilizes and locks.
• A local signal clears the wall: per-unit credit, the abstraction lever this round tunes.