Attention is prediction, not abstraction

2026-06-28 · attention is prediction, not abstraction · experiment ABSTRACTION-CHASE

The attention track lifted transfer CCGP to about 0.60, well past the 0.484 backprop ceiling that capped every gradient-free attempt at abstraction. That number lived on one probe. This round puts it on the other. We take the context-guided attention representation and feed it as the input to the per-unit-credit readout, the one pathway proven to abstract, then measure abstraction on the same ruler the ceiling is drawn on. At 18k characters, where that readout is stable, the answer is clean and negative. The attention input scores 0.429 and 0.463 across two seeds. The plain context window, the baseline, scores 0.509 and 0.476. The attention representation does not beat the baseline and does not clear the ceiling. So the 0.60 was expressive dimensionality, not abstraction, and the wall stays exactly where it was: a credit-assignment problem. This is the program's two tracks pulling apart on purpose. Attention wins at prediction and working memory. Abstraction needs the credit pathway, and the credit pathway is unmoved by what attention feeds it.

The question

Two lines of work converge on one number with two meanings.

One line found the wall. A run of experiments carried abstraction down to a single cause: a code generalizes across contexts only when each unit gets told how to change, a per-unit credit signal. A precision-gated top-down pathway delivers that signal, gradient-free and online, and it lifts transfer CCGP to about 0.48 to 0.50, right up to the backprop ceiling at 0.484, and then it stops. That ceiling is the honest mark of how far a gradient-free local rule reaches on this task.

The other line is attention. Learn where to look, key the look on context, gate when to hold the theme. That stack beats a hand-set spread of views on prediction, online and gradient-free, and along the way it lifts transfer CCGP to about 0.60, past the ceiling, with dimensionality climbing from 19 to 32.

The catch is that the two CCGP numbers sit on two different probes. The 0.484 ceiling is measured on the credit readout's own hidden code, the gradient-comparable ruler. The 0.60 is measured on the attention stack's vote-consensus code, and that probe inflates with dimensionality: when the code spreads into more dimensions, the vote-consensus CCGP rises with it. So the 0.60 has two possible readings. It could be real abstraction, a more general code that the credit pathway could pick up and run with. Or it could be expressive dimensionality, a richer but not more abstract code, a number that evaporates the moment it is measured on the abstraction ruler. The whole program turns on which one it is.

The decisive test

Bridge the two substrates and measure on the ruler that matters.

Take the context-guided attention representation exactly as the attention track builds it: columns that each choose where to look, keyed on a working-memory bucket, trained on their own local reward with lateral inhibition for diversity, gradient-free and online, with no connection to the readout. Freeze it. Then feed its output as the input to the per-unit-credit readout, the one pathway proven to abstract. Measure the readout's transfer CCGP on its hidden code, the same probe whose ceiling is 0.484.

The two arms differ in one thing only. DE-RAW-INPUT feeds the readout the plain last-five-character window, the baseline that reproduces the known credit-pathway band. DE-ATTN-INPUT feeds it the attention representation instead. Same readout, same probe, same data, same seeds. The bridge is built to keep the attention structure intact: each column writes its attended window into its own slice of the input, so the per-column decorrelation that is the point of multi-column attention survives into the readout rather than being smeared into one vector. The only difference between the arms is which positions of the stream get read, a fixed window or the context-guided offsets.

If the attention input lifts transfer CCGP past the baseline and past 0.484, the attention structure genuinely helps credit assignment, and the gradient-free stack would clear the wall. If it lands on the baseline, the 0.60 was expressiveness, and the wall stands.

The clean test is at 18k characters, where the credit readout is stable. Two seeds.

What happened

The attention input does not climb.

Read it in three parts.

The baseline holds. DE-RAW-INPUT reads 0.509 and 0.476, squarely in the known credit-pathway band of about 0.48 to 0.50. The probe is wired right at this scale, so the comparison is fair.

The attention input does not beat it. DE-ATTN-INPUT reads 0.429 and 0.463, below the baseline on both seeds and below the 0.484 ceiling on both seeds. Feeding the context-guided attention representation to the credit pathway leaves abstraction where the plain window leaves it, or slightly under.

The feature is alive, not dead. A backprop oracle on each input dropped bits per char and reached the ceiling on both, including the attention input. So the attention representation carries real signal: a gradient readout abstracts from it just fine. The plateau is not a broken feature. It is the gradient-free credit pathway declining to turn the attention structure into a more abstract code than the raw window gives it.

A larger run at 200k characters agrees on the verdict and adds the cross-check, with one honest asterisk: at that scale the credit readout diverges, a known stability limit of the stacked pathway, so the baseline no longer reproduces its band and the literal numbers there are not interpretable. What survives the divergence is the shape. Neither arm clears the ceiling, and the dimensionality of the attention arm is actually lower than the raw arm while their CCGP is flat. In the place where the vote-consensus probe shows CCGP rising with dimensionality, the abstraction probe shows CCGP not tracking it at all. The two numbers decouple exactly the way the expressive-dimensionality reading predicts.

The verdict

Feed the context-guided attention representation to the per-unit-credit readout, measure on the ruler the ceiling is drawn on, and abstraction does not move. At 18k, where the readout is stable, the attention input scores 0.429 and 0.463, under the 0.509 and 0.476 baseline and under the 0.484 ceiling on every seed. The 0.60 the attention track reached was expressive dimensionality, a richer code, not a more abstract one. The wall stays a credit-assignment problem. Attention front-ends change what is read. They do not change whether the credit pathway can carve an abstract code from it.

This is the cleanest separation the program has drawn. The two tracks are different problems with different mechanisms, and now we know it by measurement rather than by hope.

The attention track wins at prediction and working memory. Learn where to look, key it on the higher level's state, gate when to hold the theme: that stack beats a hand-set spread of views, online and gradient-free, and it builds a high-dimensional code that the vote-consensus probe rewards.

The credit track wins at abstraction. A per-unit, precision-gated top-down signal lifts transfer CCGP to the backprop ceiling and then plateaus, on the gradient-comparable probe.

Bridging them does not add the wins together. The attention representation, handed to the credit pathway, abstracts no better than a plain window does. So the road past 0.484 is not a richer input. It is a change to the credit pathway itself, and that is now the only door left, tested directly rather than assumed shut.

The honest caveats

• The clean comparison is at 18k. The 200k run confirms the direction and carries the expressiveness cross-check, but the readout diverges there, a known stability limit and an open problem of its own. The 200k numbers are a confounded confirmation, not the headline. The decisive comparison is the stable one.
• One architecture, one probe family. The test is the context-guided attention front-end into the precision-gated credit readout, measured by CCGP. It is the same scope limit the credit-pathway results carry. It separates these two mechanisms cleanly; it does not speak for every possible front-end.
• The negative is a real one because the feature is real. The backprop oracle clears the ceiling on the attention input, so the representation is informative. The gradient-free credit pathway simply does not build a more abstract code from it than from the raw window. That is the finding, stated plainly.

The frontier

The two tracks are now cleanly apart, and that sharpens the next move rather than blunting it. The attention stack has its payoff and its open knobs: a learned context partition that grows with the data, a deeper hierarchy, a top-down prior richer than a unigram. The abstraction wall has exactly one door left. It is not the input. It is the credit pathway. The candidate is the cell the program has named but not built: per-unit credit delivered on a sparse, capped, pruned substrate instead of a dense normalized one, the combination the literature says has both the dimensionality and the abstraction. Or temporal credit, carrying the signal across steps rather than within one. This round closes off the easy hope that a better front-end would do it, and points the work where it has to go.

Lineage

Grew from a local signal clears the wall and abstraction that climbs, which proved abstraction is per-unit credit assignment and set the 0.484 ceiling this round measures against; and from where you look depends on what you understand and a gate for working memory, the attention stack whose 0.60 CCGP lift this round adjudicates.

Thread: the abstraction wall, and the attention track that tested it from outside. The verdict separates the two: attention is the prediction-and-working-memory win, abstraction is the credit-pathway win, and feeding one to the other does not combine them. The wall stays a credit-assignment problem. The frontier is the credit pathway itself, on a sparse substrate or across time.