What changed: lateral inhibition clears the collapse

Theory update. 2026-06-27 · the first positive past the levers-not-additive wall · experiment LV

The locked architecture stops climbing at depth three because the deep code collapses: every column flattens to one direction across stimuli, and stacking more independent columns cannot undo it. This round adds the one degree of freedom the locked stack lacked: let the columns at the same altitude see and suppress each other. Lateral divisive inhibition takes the collapsed depth-four apex from rank one to a real high-dimensional code and lifts abstraction by 0.16. A vote alone does nothing. The missing ingredient was interaction within a level, not more breadth across levels.

The question

The architecture is locked at depth three, above the backprop ceiling, and it stops there for one reason. Push it deeper and the deep code collapses. The capstone combine (the run we called A1) named the shape of the collapse exactly. Forward-activation normalization holds each column stable, but it pins each column's activity to a fixed magnitude, and that drives the deep layers to a single direction across stimuli: the code is rank one, it says the same thing no matter the input. Concatenating more independent columns does not help, because each column collapses on its own. Breadth preserves variance across columns; normalization destroys it across stimuli within a column. Glue four rank-one columns together and you still have rank one.

So the question was whether the right move was ever "more columns" at all. The columns in the locked stack never talk to each other within a level. They run in parallel and meet only at the readout vote. The cortex does not work that way: columns at the same altitude inhibit one another, so the population spreads its labor and different columns own different inputs. That is lateral inhibition, divisive normalization across columns. If the across-stimulus collapse comes from each column being pinned in isolation, then letting the columns compete should break it: the winner at each stimulus would differ, and the population would recover the variance that pinning took away.

What we tried

We took A1's stack and changed exactly one thing. The new stack subclasses the locked combine read-only, so the per-column forward, the per-column credit, the random seeds, and every probe are inherited unchanged. Two knobs turn on, each on its own switch so we can ablate them.

• Inhibition. After each level's per-column normalization, the columns compete. Divisive cross-column normalization per unit: a column's activity at a unit is turned down by how much the other columns fire at that same unit. A bounded anti-Hebbian push adds to it, so columns that keep co-firing suppress each other more and decorrelate over time. The suppression depends on what the other columns do for this stimulus, so it varies across stimuli. That is the point: it re-injects the cross-stimulus variance the normalization pinned away. With the inhibition off, this is the bare A1 substrate.
• Voting. The readout reads an agreement-weighted consensus across columns instead of the plain product-of-experts pool. With voting off, the readout is A1's unchanged.

Everything is online, single pass, gradient-free. The headline number is the participation ratio (the dimensionality of the code) and the transfer CCGP (the abstraction score on held-out text) on the concatenation of the columns, the population probe where A1 read rank one.

We ran two things. A breadth sweep at depth three with the full lateral wire, over one, three, and five columns. And the decisive ablation: depth four, four columns, the exact regime where A1 collapsed, toggling inhibition and voting in all four combinations against the no-laterals control and a single column.

What happened: the ablation

The depth-four apex is the test. This is where A1 read a participation ratio of 1.17, a dead rank-one code.

Read it as three findings.

Lateral inhibition breaks the collapse. Turn on inhibition alone and the apex goes from 1.61 to 2.77, a real high-dimensional code where there was a rank-one one, and the abstraction score climbs from 0.315 to 0.475, a lift of 0.16. This is the single biggest move in the table, from one knob. The columns compete, the winner varies with the stimulus, and the across-stimulus variance comes back.

Voting alone does nothing. Turn on the consensus vote and leave the columns isolated, and the apex stays rank one at 1.46. The vote reconciles the columns at the output, but it cannot change what each column computes. The abstraction score rises a little (0.408), riding the better readout, but the deep code is still collapsed. The combiner is not the load-bearing piece.

Both together is best. Inhibition plus voting reaches the highest dimensionality (3.05) and the lowest bits-per-char (3.992, down from 4.126), with the abstraction score holding at 0.473. Compete in the forward, agree at the output: the two play their separate parts.

The breadth sweep at depth three tells the same story from the other side. With the full lateral wire, dimensionality climbs cleanly with the column count (the deep level reads 2.99 at one column, 6.03 at three, 6.38 at five) and bits-per-char drops from 4.035 to 3.740. The columns decorrelate to about 0.89: near-independent codes that still talk laterally. Breadth helps, but only once the columns interact.

The sanity gate

The whole claim rests on the no-laterals path being the locked collapse and nothing else, so we checked it directly. Turn every lateral knob off, and the stack reproduces A1 exactly: the four levels read 11.03, 2.86, 1.55, 1.17, the apex CCGP 0.302, bits-per-char 4.228. That is the locked collapse, bit for bit. So the lift above it is the lateral wire's doing, with no other change in play.

The lesson

Lateral divisive inhibition between columns at the same level is the anti-collapse mechanism the locked architecture was missing. It takes the depth-four apex from rank one (dimensionality 1.61) to a real code (3.05) and lifts abstraction from 0.315 to 0.473, while a vote alone leaves the code collapsed. The degree of freedom the capstone combine lacked was interaction within a level, not more independent breadth across levels. This is the first positive past the wall where the levers stopped adding.

The capstone combine left a clean diagnosis and a closed door: breadth and normalization act on different axes of variance and do not add, so the architecture was locked at depth three. The door was closed on the levers we had. LV finds the lever we did not have. The collapse lives across stimuli within a column, and the only thing that reaches across stimuli within a column is the columns talking to each other. They were never wired to.

The honest caveats

• Eighteen thousand characters, two seeds. The hundred-thousand-character confirm was skipped to share the one Metal GPU with the other arms running this round. The direction is seed-robust at this scale. The claim that it clears the ceiling, not just the collapse, waits on the longer run.
• The win is dimensionality and prediction, then abstraction. The largest, cleanest lift is on the deep code's dimensionality and on bits-per-char. The abstraction score rises with it (0.315 to 0.473) but sits below the depth-three locked winner's 0.50 to 0.55, at a shorter run. LV reopens the collapsed regime; it does not yet beat the locked depth-three number.
• One regime. Tested in the A1 collapse regime (depth four, four columns) and a depth-three breadth sweep. It shows the mechanism that breaks the known collapse; it does not yet show it on the full generation stack.

The frontier

LV reopens a door the program had bolted. The architecture was locked at depth three because deeper collapsed, and the collapse looked structural. It was not. It was the columns failing to interact. The next question is whether lateral inhibition, run at the full hundred thousand characters, carries the deeper code above the ceiling rather than merely off the floor, and whether it composes with the other anti-collapse mechanism this round turned up, interior-altitude replay. Two different wires, both fighting the same rank-one collapse: one across columns within a level, one across time at an interior level. The next move is to run them long, and run them together.

Lineage

Grew from the architecture, locked above the ceiling, whose capstone combine collapsed to dimensionality 1.17 and diagnosed why (breadth and normalization act on different variance axes), and which named lateral interaction as the untried lever; and from the empty cell, whose sparse stack hit the same rank-one collapse from the other side and named lateral inhibition the complementary across-column diversity test.

Thread: abstraction, and the collapse that capped it at depth three. The locked round closed the door on the levers it had. This one finds the lever it did not have: the columns, talking to each other. The mechanism is cortical lateral inhibition and divisive normalization; the combiner is Monty-style cross-column voting.