The rank-reuse tension

Theory update · 2026-06-28 · the rank-reuse tension · experiment two-head

The prize the last round named was one substrate with two heads: take the sparse abstract code that abstracts at the backprop ceiling, and put a count-native prediction head on it, so one representation feeds an abstraction head (transfer CCGP) and a prediction head (bits-per-char) at once. This builds it. A count table keyed on the P0.5 abstract winner-set predicts the next character, against the identical count machinery keyed on the raw character context, on the same frozen substrate, from eighteen thousand to one hundred thousand characters. The result is an informative middle that leans deep-negative, and it carries a real finding either way. The count head extracts real signal from the abstract code. Counting on the abstract code reads 2.578 bits-per-char at one hundred thousand characters, 0.8 to 1.0 bits below the stack's own softmax readout (3.548): the abstract code does carry next-character signal, and one substrate can feed both heads. But it stays about 0.45 bits-per-char behind raw context, and the gap holds with scale. Counting on the raw context reads 2.141, and the abstract code trails it by +0.50 at eighteen thousand and +0.44 at one hundred thousand, stable across 5.5 times the data, not a small-data artifact. The abstract code's own transfer CCGP is 0.499, above the 0.484 backprop reference: the same code abstracts well while predicting worse. The mechanism is a tension in the representation itself. The high rank that makes the code abstract makes the winner-set nearly unique per position, so it rarely repeats, so a count table has almost nothing to reuse and must back off to coarser subsets. High rank gives good abstraction and poor count-reuse. So abstraction and prediction want representations in tension, and the two-stacks split has a root in the representation, not just the wiring.

The question

The last round split the program into two engines and named the prize. A count-native voting bank predicts, low bits-per-char, because count tables combine. The sparse apical stack abstracts, transfer CCGP at the backprop ceiling, because per-unit credit carves a factorized code. Neither does the other, and giving the abstraction stack the voting machinery did not import prediction: the vote does nothing for prediction with neural columns.

That round named two ways forward, and this takes the second. Graft a count-native head onto the abstract code. Stop asking the abstract code to predict through its own softmax, and stop asking the vote to combine neural columns. Instead, use the P0.5 abstract sparse code as the context key for a count predictor, the same count machinery the voting bank runs, keyed on the abstract winner-set instead of the raw character n-gram. One substrate, two heads: the per-unit credit and boosting shape the abstract code, an abstraction head reads its transfer CCGP unchanged, and a count table on the same code reads the bits-per-char. If the abstract code is predictive, counting on it is low bits and the code keeps high CCGP, the prize in one representation.

The honest risk is the whole point of the test. Abstraction compresses: it throws away the surface detail a next-character predictor leans on. So a count head on the abstract code may predict worse than a count head on the raw context. If it does, that is a deep finding, the deepest form of the two-stacks result: abstraction and prediction want different representations, and the split is in the code itself. This run measures both, the count head on the abstract code and the same count head on the raw context, on the same frozen stack, so the verdict is honest whichever way it lands.

The arms

Four readings on the same trained P0.5 substrate, the same text8 split and bits-per-char loop, the verbatim transfer-CCGP probe, apples to apples, from eighteen thousand to one hundred thousand characters.

• COUNT-on-ABSTRACT, the test. A count table keyed on the deep abstract winner-set, with stupid backoff over the most-active subsets of that set, down to the unigram floor. The same add-alpha and backoff as the raw n-gram, so the only thing that changes is the key.
• COUNT-on-RAW, the control. The identical count machinery on the raw character context, the order-six backoff n-gram. Arm against arm, this is count-on-abstract versus count-on-raw, all else equal.
• P0.5-native, the anchor. The stack's own softmax readout, scored on the same frozen stack the count head reads, so the count head's lift over its own substrate is apples to apples.
• The abstraction head, the prize's other half. The same abstract code's transfer CCGP, on the verbatim probe, so the same representation is read for prediction and for abstraction at once.

What happened

One substrate, two heads. The count head pulls real signal off the abstract code and beats the stack's own readout, and it still trails the raw context by about half a bit, the gap stable with scale.

The numbers, on the same frozen P0.5 substrate, the same text8 split and bits-per-char loop, the verbatim transfer-CCGP probe. Bits-per-char is held-out, lower is better; the count-on-abstract arm is the backoff key, the primary number.

Read it two ways and it splits two ways.

The count head extracts real signal from the abstract code. Counting on the abstract code reads 2.578 bits-per-char at one hundred thousand characters against the stack's own native readout at 3.548, a sub-win of about one bit, and 2.953 against 3.753 at eighteen thousand, the same 0.8-bit gap. So the abstract code is not opaque to a count head. It carries genuine next-character signal that the stack's own softmax leaves on the table, and a count table on it does better than the substrate does for itself. One substrate can feed both heads: the same code that abstracts at the ceiling supports a count predictor that beats its native readout. The sub-win is real, and it is the half that says the two-heads design is not empty.

But the abstract code is measurably less predictive than raw context, and the gap is stable with scale. Counting on the raw context reads 2.141 at one hundred thousand, and the count head on the abstract code trails it by +0.44, after trailing it by +0.50 at eighteen thousand. Same count machinery, same alpha, same backoff, the only difference the key, and the abstract key costs about half a bit. The gap does not shrink from eighteen thousand to one hundred thousand, across 5.5 times the data: it is not a small-data artifact that more text would close. The abstract code is a worse context key for prediction than the raw context, robustly. The prize, count-on-abstract at or below count-on-raw, is not reached.

And the same code abstracts well while predicting worse. Its transfer CCGP is 0.499 at one hundred thousand, above the 0.484 backprop reference, the abstraction head intact and high-rank (about seventeen hundred of three thousand probe positions are distinct codes). So this is not a code that failed to abstract and so failed to predict. It is a code that abstracts well and predicts worse, on the same frozen substrate, at the same time. The two heads read the one representation and disagree about whether it is good.

The mechanism is a tension in the representation, and a table-fill diagnostic shows it directly. The count head's tables record how often the abstract code repeats across the stream. Even at seventy thousand training positions, a full abstract winner-set key is seen only three to six times: the code is nearly unique per position. The high rank that makes the code abstract, hundreds of distinct winner-sets, mixed selectivity, is exactly what makes it almost never repeat, and a count table predicts by reuse. A key that rarely recurs has almost nothing to count, so the prediction backs off to coarser subsets of the winner-set, and a coarse subset is a blunt predictor. Raw context repeats far more (a six-character string recurs across a corpus), so its count table is dense and sharp. High rank gives good abstraction and poor count-reuse, and the two pull against each other in the same code.

One substrate, two heads. The count head on the P0.5 abstract code reads 2.578 bits-per-char at one hundred thousand characters, 0.8 to 1.0 below the stack's own readout (3.548): a real sub-win, the abstract code carries next-character signal and one substrate can feed both heads. But it trails the same count head on raw context (2.141) by +0.44, after +0.50 at eighteen thousand, a gap stable across 5.5 times the data, while the same code's transfer CCGP is 0.499, above the 0.484 reference. The mechanism: the abstract code is nearly unique per position (three to six counts per key at seventy thousand positions), so it rarely repeats, so a count table has little to reuse. High rank gives good abstraction and poor count-reuse.

So the prize is not reached, and the negative is the deeper finding. Abstraction and prediction want representations in tension: the rank that abstraction wants is the rank that prediction-by-counting cannot reuse. The two-stacks split the program kept drawing has a root not just in the wiring but in the code itself.

What this means

The two-heads design is half-confirmed and half-refuted, and the refuted half is the one that teaches.

The sub-win is real and worth keeping. A count table on the abstract code beats the stack's own softmax by about a bit, so the abstract code does carry next-character signal, and the count-native head is a better readout for it than the neural readout the stack ships with. One substrate can feed both heads in the literal sense: the same code abstracts at the ceiling and supports a count predictor above its own native floor. The two-heads architecture is not empty.

But the prize, low bits and high abstraction in one representation, is not won, and the reason is structural. The abstract code is about half a bit worse than raw context as a key for the same count machinery, and the gap is stable across 5.5 times the data, so it is a property of the code, not of the data budget. The diagnostic names it precisely: the abstract code is nearly unique per position, three to six counts per key even at seventy thousand positions, because high rank means hundreds of distinct winner-sets that almost never repeat. A count table predicts by reuse, and a key that does not repeat gives it nothing to count, so it falls back to coarse subsets and predicts bluntly. The very rank that earns the transfer CCGP is what starves the count head. Abstraction and prediction want representations in tension.

That sharpens the two-stacks result into a claim about representation. Earlier the split looked like a wiring fact: the vote couples count columns and does not couple neural ones. Now it looks like a deeper fact about the code: the rank a representation needs to abstract is the rank that makes it unique, and a unique code cannot be counted. A representation cannot be maximally abstract and maximally reusable at once, so a single shared code is pulled in two directions, and the program's habit of building two stacks has a root in the representation itself.

The honest bounds are sharp. This is eighteen thousand to one hundred thousand characters and one to two seeds, so the absolute bits-per-char are scale-compressed (the voting bank reaches about 2.1 only past two hundred thousand). The robust finding is the tax: about half a bit, stable across 5.5 times the data, count-on-abstract behind count-on-raw with identical machinery. The sub-win over the native readout is robust too. What is not yet tested is the lever the diagnostic points at, so the deep-negative reading is leaned-toward, not closed.

The lever is a sparsity-versus-reuse sweet spot. If the cause is that the code is too unique to count, a coarser, less-sparse code (a smaller top-k winner-set) might repeat often enough to predict while staying abstract enough to keep its CCGP. The high-rank code wins abstraction and loses reuse; a code dialed down in rank might give up a little abstraction to buy the reuse a count head needs, and land a point where both heads are decent. That run is not done. It is the next move, and it is the test of whether the tension is a hard wall or a tunable trade-off with a usable middle.

The two-heads design is half-confirmed: a count head on the abstract code beats the stack's own readout by about a bit, so one substrate can feed both heads, the sub-win real. The prize is not won: count-on-abstract trails count-on-raw by about half a bit with identical machinery, stable across 5.5 times the data, while the same code's CCGP is 0.499 above the 0.484 reference, so the code abstracts well and predicts worse. The mechanism is the rank-reuse tension: high rank makes the code abstract and nearly unique per position, so a count table has nothing to reuse. Abstraction and prediction want representations in tension. The lever to try next is a coarser, less-sparse code, a sparsity-versus-reuse sweet spot, untested.

Lineage

Grew from one stack, two halves, which named the two paths forward after the vote failed to import prediction, and this takes the second: graft a count-native head onto the P0.5 abstract code, one substrate with two heads. It reuses that round's substrate verbatim, the P0.5 sparse apical stack from the empty cell, filled (boosting and positive weights and per-unit credit, transfer abstraction at the 0.484 ceiling), and the same count machinery the voting bank runs, so the count-on-abstract and count-on-raw arms differ only in the context key. It carries forward attention is prediction, not abstraction, which proved a richer code does not abstract better on the gradient-comparable ruler: there a richer input did not buy abstraction, here a more abstract code does not buy prediction, the same separation seen from the other side.

Led to the rank-reuse tension as a named property of the code, and to the lever the diagnostic points at: a coarser, less-sparse code (a smaller top-k winner-set) that might repeat enough to predict while staying abstract enough to keep its CCGP, a sparsity-versus-reuse sweet spot, untested as yet. And to the sharper standing read of the two-stacks split: it has a root in the representation itself, because the rank a code needs to abstract is the rank that makes it too unique to count.

Thread: abstraction, and online learning without a global gradient. An honest informative-middle result that confirms one substrate can feed both heads (the count head beats the stack's own readout) while showing the prize unreached (the abstract code pays a stable count-reuse tax), and names the deep reason: abstraction and prediction want representations in tension.