The empty cell

2026-06-27 · the empty 2×2 cell, tested · experiment P0

The architecture review found an empty cell. The whole abstraction line crosses two axes (sparse code or dense, per-unit credit or none) but only ever filled three of the four boxes. The pure-biology stack was sparse with no credit, and did not abstract. The apical fusion carried per-unit credit on a dense code, and abstracted but collapsed. Nobody had ever put per-unit credit on a sparse code, and the outside literature said that exact cell should both abstract and resist the collapse. So we built it: k-WTA on every level, credit masked to the winners, everything else held. The result is real and it is partial. The sparse code does abstract (it climbs to the backprop ceiling, beating the no-credit control by a wide margin) but it does not stay there. It peaks and collapses, because k-WTA bounds how many units fire while the dense credit pathway still drives the same few to win every time. The collapse has two causes, and we only fixed one.

The cell nobody built

The architecture review made a claim worth testing head-on: the collapses that dog the abstraction line (the rank-one degeneracy, the drift at scale) may be artifacts of the substrate, not limits of the problem. The whole line, it pointed out, lives on a dense, normalized, fully-overwritten code. And it found that the program had quietly crossed two independent axes without ever filling the box where they meet.

Read it carefully, because both filled rows carry a lesson. Pure biology ran a genuinely sparse, capped, pruned stack (the textbook brain-shaped substrate) with no per-unit credit, and its abstraction score dropped: one of six convergent negatives. So a sparse substrate on its own does not abstract; "just go back to biology" is refuted by our own data. And the apical fusion carried per-unit credit (a precision-gated top-down signal, feedback-alignment style) on a dense code, and it abstracted beautifully, climbing with depth to the backprop ceiling. But it collapsed: at depth the deep code falls to rank one across stimuli, and on a long fresh stream it drifts.

The empty cell is the synthesis: a sparse substrate carrying per-unit credit. The outside literature is unusually specific that this box has both properties we want. Sparse distributed memory (Bricken 2023) makes a top-k code a state-of-the-art continual learner. Active dendrites with k-WTA and per-unit credit (Iyer 2022) hold eighty-one percent across a hundred tasks without forgetting. Sparse feedback alignment (Crafton 2019) shows credit survives a sparse feedback matrix: it is the rank of the signal that matters, not its density. If the review's diagnosis is right (the credit math is correct, we just delivered it on the wrong substrate) then this cell should abstract like the apical fusion and resist the collapse. P0 fills it.

The minimal edit

The test is decisive only if nothing else moves. So P0 is the smallest possible change to the locked stack. It subclasses the locked apical stack (inheriting the forward shape, the fixed-random feedback wiring, the precision gate, the probe seams, all of it untouched) and changes exactly two things.

The forward becomes k-WTA. Instead of pinning every level's activations to a fixed root-mean-square (the dense "actnorm" the review names as the proximate cause of the rank-one collapse), each level keeps its top k units (about five percent of the layer, the sparse-distributed-representation density) zeros the rest, then normalizes the survivors to the hypersphere. That last step is the sparse-distributed-memory support, and it is not optional: without it the forward activations run away and bits-per-char blows up to thirty-three. Bricken found the same: remove the normalization and the k-WTA continual learner crashes from seventy-one percent to twenty. The crucial difference from dense normalization is what carries the variance: under actnorm every code has the same magnitude, so the only thing left to distinguish stimuli is direction, which the deep layers flatten. Under k-WTA the identity of the winners (which five percent fired) carries the information. Bound the activity, keep the rank. That is the whole theory of the fix.

The credit is masked to the winners. The per-unit apical signal only updates the units that fired: for a k-WTA layer, the winners are the active set. This is Iyer's "only the winning neurons update," made literal.

Everything else is held byte-for-byte: the fixed-random feedback matrices, the precision burst, the credit re-normalization, the initialization, the depth of three. Four arms fill the 2×2: sparse-with-credit the hero, sparse-no-credit the pure-biology signature in miniature, dense-with-credit the locked reference, dense-no-credit the floor. All on the same eighteen thousand characters, the same seeds, the same probes that scored every experiment back to the temporal-pooling negatives. Directly comparable, one variable at a time.

What happened

The sparse code abstracts. And then it collapses.

The numbers, at eighteen thousand characters, depth three, averaged over two seeds: deep transfer abstraction (CCGP), deep dimensionality (participation ratio), and the count of distinct winner-sets across four hundred stimuli, the direct read on whether the sparse code stayed diverse:

Read the hero arm against its controls and two things are unambiguously true.

The sparse code abstracts, for real, at peak. Sparse-with-credit reaches a deep transfer score of 0.469 averaged, 0.53 on the better seed, at the backprop ceiling of 0.484, and it beats the sparse-no-credit control by +0.154. That margin is the whole point of the cell. The no-credit arm is the pure-biology signature reproduced exactly: flat at 0.31, a single winner-set, dimensionality degenerate. Adding per-unit credit to the same sparse substrate lifts it a sixth of a point, to the ceiling. So the empty cell is not blank. Per-unit credit does carry on a sparse substrate, and it does build an abstract space: the review's synthesis is real, and the strong form of its claim survives contact.

But the sparse code does not stay abstract. By the final checkpoint the hero arm has collapsed, back to about 0.31, the baseline, the no-credit floor. It is bistable: it abstracts, then it falls. And the mechanism read says exactly why. Across four hundred stimuli the deep code ends with only about four distinct winner-sets. The same handful of units win, over and over, whatever the input. The dense reference, by contrast, holds thirty-eight to eighty-one distinct sets and stays at 0.50 the whole way.

The empty cell is not blank. Per-unit credit carries on a sparse substrate and builds an abstract space: sparse-with-credit hits the backprop ceiling, beating the no-credit control by +0.154. But it does not stay there. It peaks and collapses, and the deep code ends with about four distinct winner-sets across four hundred stimuli. The synthesis is alive, and unstable.

Why it collapses: the second cause

Here is the diagnosis, and it is the reason this result is partial rather than a clean win. We expected k-WTA to fix the collapse, because k-WTA fixes the substrate. It does. But the collapse has two causes, and k-WTA only addresses one of them.

k-WTA bounds how many units fire: exactly k of them, by construction, no matter what. That removes the activity runaway that destabilized the dense stack at depth. What k-WTA does not touch is which units fire. And the dense credit pathway (the same fixed-random feedback-alignment signal the apical fusion uses) still drives the pre-activations toward a low-rank structure. A low-rank drive means the same directions keep getting amplified, which means the same units keep crossing the top-k threshold, which means the winner-sets stop varying across stimuli. The rank-one collapse of the dense code does not disappear on a sparse substrate. It reappears, wearing a different mask, as a winner-set collapse.

So the review's headline (seventy percent a substrate artifact) is sharpened by P0 into something more precise. The collapse is dual-cause. One cause is the substrate: dense normalization pins magnitude and throws away the variance, and k-WTA genuinely fixes that, the peak at the ceiling is the proof. The other cause is the credit pathway itself: a dense feedback signal drives the code low-rank regardless of how the forward is bounded, and k-WTA does nothing about that. k-WTA plus the hypersphere normalization, alone, is not enough.

This is not a refutation of the review. The strong claim, that a sparse substrate can carry per-unit credit and abstract, is confirmed, transiently. What is refuted is the simple version of the fix: that swapping the substrate would be sufficient. It is necessary and it is not sufficient.

What this means, and what it does not

It would be easy to over-read this in either direction, so let us be exact.

It does not license a pivot. The honest move after a partial result is to not tear down the working thing. The locked dense-with-credit stack still sits at 0.50, sustained, stable across the run: it remains the program's stable abstraction reference, and nothing here displaces it. P0 is a probe of a candidate substrate, not a replacement for the one that works.

It does not say sparsity failed. Sparsity did its job: it bounded the activity and it let the code touch the ceiling, which the dense code only does with its own separate stabilizer. The peak is the sparse substrate earning its place.

What it does say is that the next move is more biological machinery, not less, and specifically, machinery aimed at the second cause. The credit pathway needs to stop driving the sparse code onto a few units. Three mechanisms target exactly that, and they are the P0.5 queue: diversity-forcing homeostasis, the boosting and duty-cycle rule from the spatial pooler, which forces every unit to take its turn winning over time, the most direct possible answer to a winner-set collapse; positive weights with no bias, the sparse-distributed-memory regime Bricken's continual learner actually runs in; and sparse-permanence credit, so the credit itself is sparse and stops collapsing onto a handful of synapses. Lateral inhibition across columns is the complementary test: diversity across the population rather than within a level.

The collapse is dual-cause: the substrate (fixed by k-WTA) and the credit pathway (not fixed by k-WTA). So the fix is more machinery, not a pivot: diversity-forcing homeostasis, positive weights without bias, sparse credit, all aimed at the second cause, the one k-WTA leaves standing. The locked dense stack remains the reference until something beats it and stays.

A note on reading the result, because it nearly fooled us. The sparse code is bistable at eighteen thousand characters (it genuinely abstracts and genuinely collapses within one run) so a single final-checkpoint score would have read the whole arm as a flat negative and missed the ceiling-touching peak entirely. That peak is the load-bearing fact. We report peak and final precisely because either one alone lies: the final says "no abstraction," the peak says "ceiling," and only together do they say the true thing, abstracts, does not retain. (Two smaller gotchas: the hypersphere normalization is mandatory, drop it and bits-per-char runs away to thirty-three, and a participation ratio of nan is the degenerate-code fallback, the no-credit arm's single winner-set.)

Lineage

Grew from the architecture review and its named empty 2×2 cell: the review's diagnosis that the abstraction line's collapses are seventy percent a substrate artifact, and its claim that the credit math is right but delivered on the wrong substrate. It reuses the apical fusion's per-unit precision-gated credit verbatim, the pure-biology sparse signature as its no-credit control, and the same transfer-abstraction and dimensionality probes that scored the line back to the temporal-pooling negatives. It grounds in sparse distributed memory (Bricken 2023, the top-k code and its mandatory hypersphere norm), active dendrites (Iyer 2022, "only the winning neurons update"), and sparse feedback alignment (Crafton 2019, rank over density).

Led to the P0.5 queue: diversity-forcing homeostasis (boosting and duty-cycle), positive weights without bias, and sparse-permanence credit, each aimed at the second cause of the collapse, the dense credit pathway that k-WTA leaves untouched; and to LV, the across-column lateral-inhibition diversity test.

Thread: abstraction, and online learning without a global gradient. A partial, decision-grade result (the empty cell is real and unstable) that sharpens the review's diagnosis into a dual-cause collapse and tells the next experiments exactly which cause to attack.