Learning where to look

2026-06-28 · learning where to look · experiment AT1

Each column in the model reads the text through a fixed window, set by hand to look back a fixed distance. This round asks whether the model can learn that distance for itself, online and gradient-free, the way moving your eyes through a page is itself an action. The answer is a qualified yes. A count-based bandit over look-back offsets learns informative, diversified views, uses a content-relative word-jump, and beats resampling the offset at random. It does not match the hand-set spread at scale. And it works only with one extra wire: lateral inhibition on the action space, the same diversity pressure that broke the deep collapse, now pushing the columns to look in different places. The honest read is that context-free learned attention is real but trails free hand-set diversity. The payoff is context-guided attention, where higher levels steer where lower levels look, and that is the next run.

The question

The model reads through many columns, and each column reads a view: a window a fixed distance back from the cursor, over a fixed span, of a fixed feature. The distances are set by hand. One column reads the last character, another reads three back, another reads seven back, and the spread is chosen so the columns cover complementary slices of recent context. It works, and the offset-keyed count is a kind of attention, learned from nothing but counts. But the where is hand-set, not learned.

So the question is whether the model can learn where to look. Make the look-back offset an action, chosen fresh at every step, and reward the choice by how well that column then predicts. This is the program's "motor is moving through text" made concrete: scanning the page is a motor act, and the system should learn the scan the way it learns everything else, online, per character, with no gradient. The bet is that a learned scan beats a random one and recovers the hand-set spread without anyone setting it.

What we tried

Three arms over the same text, the same seeds, the same columns. Twelve columns, two hundred thousand characters, two seeds. Every column reads the same span and feature and draws from the same menu of look-back offsets. The only thing that differs across arms is how the per-step offset is chosen, so any gap is the policy's doing and nothing else.

• Fixed (the hand-set baseline). Each column holds one constant offset, spread across the menu so the columns diversify by construction. This is the diverse-views design the program already runs, the number to beat.
• Random (the floor). Each column draws a fresh offset uniformly from the menu every step. Movement, no learning.
• Learnable (the policy). Each column runs a count-based bandit over the menu. The value of an offset is a leaky running average of the reward seen when the column used it, the reward is how much probability the column put on the character that actually came next, and the choice is the value softmax with a small explore floor. Online, per character, gradient-free, bounded to one value vector per column. The menu also holds one content-relative action, a word-jump: look back to the previous word boundary, so "attend the word you are inside" is a single learnable move.

One wire makes the policy work, and it is the load-bearing finding. Reward a column for its own prediction alone, and every column learns to grab the most recent character, the single most predictive offset. All twelve columns collapse onto the same view, the product-of-experts over-sharpens, and bits-per-char explodes. This is the same-view degeneracy the diverse-views design exists to avoid. The fix is to subtract a crowding penalty: a column is docked for predicting well where the other columns already look. That is lateral inhibition on the action space, the same diversity pressure that broke the deep representational collapse, carried from the code to the scan. With it, the columns spread into complementary offsets.

What happened

The headline is prediction, averaged over the two seeds.

Read it as three findings.

Learning where to look beats random. The learnable policy reads 3.797 bits-per-char against random's 3.872, a gain of 0.076. Learning the scan helps over moving at random. The win is real and small.

The policy learns something, not noise. The learned offset distribution is non-uniform, mean KL from uniform 0.68 bits, and the columns diversify: 7 of the 12 settle on distinct greedy offsets, and several learn to use the content-relative word-jump rather than a fixed distance. The bandit discovers informative, spread-out views, the right panel above against the flat random menu. It re-finds the kind of diversity the hand-set arm was given.

It does not match the hand-set spread. Fixed reads 3.513, and the learnable policy trails it by 0.284 bits-per-char. The learned scan recovers diversity and beats random, but at this scale it does not catch the hand-set diverse views.

The abstraction axis stays flat across the arms, as the diverse-views work predicted: transfer CCGP sits near 0.51 to 0.54 for all three, so a prediction win here is not an abstraction win. The learnable arm does hold the highest within-set CCGP (0.403) and the highest vote decorrelation (0.474), consistent with columns that look in genuinely different places.

The lesson

Learnable, gradient-free, online attention is real. A count-based bandit over look-back offsets learns informative, diversified views, uses a content-relative word-jump, and beats resampling at random by 0.076 bits-per-char. It works only with lateral inhibition on the action space, the same diversity pressure that broke the deep collapse, now spreading the columns across complementary offsets. But context-free learned attention does not match hand-set diverse views at scale, trailing by 0.284 bits-per-char. When context does not matter, fixed diversity is already good and free. The payoff is context-dependent attention, which fixed offsets cannot do.

This is the crossover the diverse-views work set up. Hand-set offsets are a strong baseline precisely because, with no context to condition on, a fixed spread of views is already near the best a column can do, and it costs nothing to learn. A context-free policy that learns the same spread the hard way recovers most of the diversity and beats random, but it has no new information to beat the hand-set arm with: both end up choosing from the same menu, blind to what the text is doing right now. The diversity-pressure finding is the durable mechanism here. The same lateral inhibition that the program found breaks the representational collapse, applied to the scan instead of the code, is what stops every column from collapsing onto the single greedy offset. One principle, two places.

The honest caveats

• It trails the hand-set baseline. The learnable policy beats random and learns real structure, but it reads 3.797 against fixed's 3.513. This is a partial result: learnable attention works, and it does not yet match free hand-set diversity at this scale.
• One substrate, trimmed scale. Twelve count columns on the CPU, two hundred thousand characters, two seeds. The direction is the finding, not a scaled number.
• The diversity weight is the one sharp knob. Without strong lateral inhibition on the action space the policy collapses to the same-view degeneracy and bits-per-char explodes; with it the columns spread. Small values do not break the consensus. The crowding weight is the knob that decides the result.
• A prediction result, not an abstraction one. Transfer CCGP is flat across all three arms (near 0.51 to 0.54), so the learnable arm's gain is prediction and vote diversity, not a more abstract code.

The frontier

Context-free attention is the wrong test, and the result says so cleanly: when there is no context to condition on, a fixed diverse spread is already good and free, so a learned context-free scan can tie its diversity but cannot beat it. The whole point of attention is to look differently depending on what is happening. The next experiment puts the context back: higher levels steer where lower levels look, conditioning the scan on the situation, which a fixed offset cannot do by construction. That is where learnable attention should earn its keep, by reading the long-distance cue when the sentence needs it and the recent character when it does not. The mechanism is in hand, gradient-free and online, and the diversity wire that makes it work is the same one the program already trusts. Giving it context is the next run.

Lineage

Grew from attention, but counted instead of trained, the offset-keyed count that made fixed-distance views a kind of attention, whose hand-set spread is the baseline here; and from lateral inhibition clears the collapse, whose diversity pressure between columns is the exact wire that, moved from the code to the scan, keeps the learned offsets from collapsing onto one view.

Thread: attention, the program's new direction. The mechanism is a count-based, gradient-free bandit over look-back offsets, the scan as an action, with lateral inhibition on the action space for diversity. The frontier is context-guided attention, where higher levels steer where lower levels look.