Where you look depends on what you understand

2026-06-28 · where you look depends on what you understand · experiment AT2

The last round made attention learnable but context-free, and it trailed the hand-set views. The whole point of attention is to look differently depending on what is happening, so this round puts the context back. A higher level holds a working memory of the text, and that memory steers where each lower column looks. The result has two honest wins. At the right granularity, two context buckets, the context-guided scan beats the hand-set baseline on prediction (3.022 against 3.101 bits per char, every seed), flipping the context-free result that trailed. And the robust win is abstraction: context-keying lifts transfer CCGP from 0.494 to about 0.60 and dimensionality from 19 to 32 at every granularity tested. The attention is genuinely context-dependent, looking in different places in different contexts. One caveat is sharp: the prediction win is bucket-sensitive, and the working-memory prediction prior consistently hurts. Drop the prior. The win is on the attention side.

The question

The previous experiment made the look-back offset an action and let each column learn it, online and gradient-free. It learned real structure and beat a random scan, but context-free it trailed the hand-set diverse views, because when there is no context to condition on a fixed spread is already good and free. The verdict named its own next step: attention is supposed to look differently depending on what is happening, so a scan that ignores the situation cannot earn its keep.

So this round puts the context back. The bet is that attention should be context-dependent, and that a higher level is where the context lives. Hold a working memory of the region you are reading, and let that memory steer where each lower column looks: at a clause boundary look back to the theme, mid-word look at the recent characters. A fixed offset cannot do this. A context-free policy cannot either, because it has one rule for every situation. A policy keyed on the higher level's state can. The test is whether context-guided attention finally beats the hand-set baseline where the context-free policy did not.

What we tried

A second altitude, built from three count-native, online, gradient-free, bounded pieces.

• A working memory, one level up. Pool the lower columns into a state that persists with a slow leak: a leaky char-distribution over the characters recently realized (the running thematic mix of the region) plus the live word-phase (the distance since the last space). It holds the theme and the sentence-start while the lower level reads the end. Leaky integration is a running average, which is the online, gradient-free substrate the program already trusts.
• Context-guided attention, the core. The previous round's offset bandit gains a key. Its value table becomes context-keyed: one value vector per context bucket, where the bucket is a small quantization of the working-memory state. The column learns "in context X, look at offset Y." The update is the same gradient-free bandit, now keyed by bucket, and the same lateral inhibition on the action space that the previous round needed is kept. A fixed offset is one bucket-blind rule; a context-keyed policy can put different offset mass in different buckets, the signature of context-dependence.
• A top-down prediction prior. The working-memory state biases the next-char prediction: the leaky char-distribution blends into the pooled prediction in log space, gated by how peaked the state is. So prediction reads both the local view and the held context.

The arms run over the same text, the same seeds, the same columns, the same span and feature and offset menu, differing only in how attention is chosen and whether the prior fires. Fixed is the hand-set diverse offsets, the baseline to beat. Context-free is the previous round's single learnable policy. The operative arm is context-guided, no prior: context-keyed offsets alone, which isolates the attention guidance from the prediction prior. Eight columns, two hundred thousand characters, the mean of three seeds.

The number of context buckets is the load-bearing lever. Keying the policy on context splits each column's bandit data across the buckets, so too many buckets starve each cell while too few cannot tell situations apart. The right count is a sweep, so the round runs it at two, four, and eight buckets.

What happened

The headline is two wins on two axes, read across the bucket sweep.

Read it as three findings.

At the right granularity, context-guided attention beats the hand-set baseline on prediction. At two buckets it reads 3.022 bits per char against fixed's 3.101, a gain of 0.079, and it holds on all three seeds. This is the prediction win the context-free policy could not reach: the same hand-set spread, the same menu, but now the scan adapts to the region it is in. The previous round trailed fixed by 0.284 context-free; conditioning the scan on the higher level's state flips the sign.

The robust win is abstraction. Context-keying lifts transfer CCGP from fixed's 0.494 to 0.534, 0.604, and 0.599 across the three bucket counts, and dimensionality from 19 to about 32 at every one. The abstraction lift does not depend on getting the bucket count right: at four buckets, where prediction is slightly worse than fixed, transfer is highest at 0.604. The previous attention round left abstraction flat across its arms. Giving the scan context moves it, at every granularity.

The attention is genuinely context-dependent. The learned offset distribution differs across context buckets, mean cross-bucket Jensen-Shannon divergence 0.80 to 0.85 bits over the buckets the text exercised. The columns look in different places in different contexts, which is the thing a fixed offset cannot do by construction and a context-free policy will not do because it has one rule. This is the signature the round was built to find, and it is strong.

The lesson

Where a column looks should depend on what the level above understands. Hold a working memory of the region, key each column's offset policy on it, and the scan adapts: at two buckets it beats the hand-set baseline on prediction (3.022 against 3.101, every seed), flipping the context-free result that trailed by 0.284. The durable win is abstraction. Context-keying lifts transfer CCGP from 0.494 to about 0.60 and dimensionality from 19 to 32 at every granularity, where the context-free round left abstraction flat. And the policy is genuinely context-dependent, looking in different places in different contexts (cross-bucket offset divergence 0.80 to 0.85 bits). The previous round was the right mechanism on the wrong test. Context is what attention is for.

This is the crossover the diverse-views work set up, resolved. A fixed spread of views is a strong baseline precisely when there is no context to condition on, and a context-free learned policy can only tie that spread because it, too, has one rule for every situation. The new information is the higher level's state. Once the scan can read it, the policy has something the fixed arm never had, and it both predicts better at the right granularity and builds a more abstract code at every granularity. The same diversity wire that the previous round needed still does its job, now per bucket, keeping the columns from collapsing onto one view within each context.

The honest caveats

• The prediction win is bucket-sensitive. Two buckets beat fixed by 0.079, but four buckets read 3.114 (just above fixed) and eight read 3.084 (a slim 0.017 below it). More buckets help abstraction yet starve the bandit and lift bits per char, because keying the policy splits each cell's data. The abstraction lift is robust to the bucket count; the prediction win is not.
• The working-memory prediction prior hurts. Blending the held context into the next-char prediction consistently raises bits per char, so the full arm with the prior trails the no-prior arm. The leaky char-distribution is close to a unigram, and once the lower columns are well fed it over-biases. Drop the prior. The win is on the attention side, not the prediction prior.
• One substrate, trimmed scale. Eight count columns on the CPU, two hundred thousand characters, three seeds. The two wins and the context-dependence are the finding, not a scaled number.

The frontier

The mechanism now earns its keep: a higher level guides where a lower level looks, online and gradient-free, and it both predicts and abstracts better than the hand-set baseline. The bucket count is the open knob, and it tells a clean story: the right granularity grows with the data, because more data feeds more buckets before they starve. So the next step is to grow the context and the data together, and to make the bucket a learned partition rather than a hand-set quantization, so the higher level decides for itself how many situations are worth telling apart. The prior is parked, not killed: a top-down prediction signal that reads more than a unigram (a held theme over real concepts, not raw characters) is the version worth a fair rematch. The attention direction has its payoff. Holding context while reading is what made it pay.

Lineage

Grew from learning where to look, which made the scan a learnable action but found it trails the hand-set views context-free, naming this exact next step; from attention, but counted instead of trained, the offset-keyed count whose hand-set spread is the baseline both rounds measure against; and from lateral inhibition clears the collapse, whose diversity pressure between columns is the wire that, kept per bucket, holds the context-keyed offsets from collapsing onto one view.

Thread: attention, the program's new direction. The mechanism is a count-based, gradient-free offset bandit keyed on a working-memory state one level up, with lateral inhibition on the action space for diversity. The frontier is a learned context partition that grows with the data, and a top-down prior richer than a unigram.