Reaching back by the right cue

Win. 2026-06-26 · a new primitive · experiment AO

Content-addressable cue retrieval binds a verb to its correct-number subject across a clause: 99.96% where offset-attention's fixed position key tops out at 65%, a wall it cannot reach past. And it reproduces the human agreement-attraction illusion, for the same reason a mind makes it: the fan divides activation.

The question

Offset-attention gave the model attention out of counts: tables keyed by relative position d, weighted by each offset's information gain. It works, and it has two limits baked into the key. It reaches only a fixed window, and its informativeness decays with distance. A dependent and its antecedent can sit arbitrarily far apart (a subject and its verb across an embedded clause) and no position key reaches that far.

A reading mind solves the same problem differently. It does not count back a fixed number of words. It fires a bundle of cues (word class, number, recency) and every memory item whose features match is reactivated; the winner is the one with the highest activation. The signature law from the sentence-processing literature is

activation(item) = leaked_base(item) / FAN(cue)

where leaked_base is a recency-leaked count and FAN is how many stored items share the cue. Sharing a cue with distractors divides activation: that is similarity-based interference, the fan effect. We wanted to know whether a count model, online and gradient-free, can reach back by the right cue instead of the right offset: generalise the position key to a feature key, weight by fan instead of by offset.

What we tried

Text8, word level, top 40,000 words plus UNK, 6.79M words, one online pass, fixed seed. The probes are long-distance subject-verb agreement dependencies built straight from the corpus. A determiner pins the subject's number independently of the verb (a/an/this dog → singular; these/those/many dogs → plural); we find the first agreeing copula or auxiliary within 40 words. Every number-marked noun in between is a distractor. 60,000 probes, mean subject→verb distance 14.8 words, genuinely across a clause. The retrieval store is online and bounded: each cue keeps only its most-recent live items, counts only, no gradients. We race it head-to-head against offset-attention on the same probes and the same target.

What happened

Content addressing reaches where a position key cannot. Cue-retrieval binds the verb to a correct-number antecedent 99.96% of the time against offset-attention's 65.26%. Offset-attention's best single offset is d=1, the modal the dog is distance, and a fixed offset can only ever be right when the subject sits at exactly that distance, so it tops out near chance-plus-modal. Past its one good offset it scores a flat 0.00% at every distance band: a fixed key cannot point anywhere but its fixed offset. Cue-retrieval scores at every distance, a slope rather than a wall, strongest when the antecedent is recent and tailing off as the leak weakens it.

And the human interference signature is present, and directional. The agreement-attraction error (the verb bound to a noun of the wrong number, the classic "the key to the cabinets are…" illusion) appears only when a recent opposite-number distractor exists:

Zero where the theory says zero, nonzero exactly where it says the illusion should bite. The fan behaves as the law requires (same-number competitors carry the highest fan, opposite-number the lowest) and holding distance fixed, exact retrieval is 100% with no same-number competitor and 0% with one, so the interference is the fan dividing activation, not a distance artifact. The effect is small because the number cue's own fan protects the correct number. It is a qualitative signature, not a headline number, and it lands where it should.

The lesson

Reaching back by the right cue (a feature bundle, weighted by fan) is a strictly better combiner for a long-distance dependency than reaching back by a fixed position. A count model, online and gradient-free, binds a verb to its correct-number subject across an embedded clause where offset-attention is pinned to one offset and flat-zero past it. And because activation divides by the fan, the same mechanism that wins the binding also makes the human agreement-attraction error, and makes it in exactly the one condition a person does.

The honest limit is in the exact-instance metric: the most-recent same-number noun, not the determiner-pinned head, is the most active item, so exact subject retrieval decays with distance and collapses when a same-number competitor is more recent than the head. That is structurally correct (recency makes the intervening noun win) and it is why the agreement-level metric, the one the cognitive model actually predicts, is the faithful one. Long-distance binding is now in hand.

Lineage

Grew from attention, but counted, generalising its position key to a feature key and its information-gain weight to fan, lifting the fixed-window limit that post left open.

Thread: the right combiner, reaching past the last few words. The ideas are Lewis & Vasishth's cue-based retrieval, Jaeger, Engelmann & Vasishth's interference meta-analysis, and Anderson's fan effect.