Two levels is enough

2026-06-28 · two levels is enough · experiment AT4

The attention track climbed three rounds: learn where to look, let a higher level guide it, and hold the guiding theme behind a learned gate. Each round added one level of control, and each beat hand-set fixed views. This round asks how far that climb goes. It adds a fourth piece, a third level above the other two, a slower context that guides the middle level the way the middle level guides the bottom, and it replaces the gate's fixed threshold with one that calibrates itself to a target fire-rate. Two questions, two honest answers. The third level does not pay off, even at scale. At two hundred thousand characters the three-level stack reads 3.096 bits per char against the two-level stack's 3.049, worse, though the nesting is real and active. The self-calibrating gate does pay off: at the same capacity it reads 3.049 against the fixed-threshold gate's 3.072, and it holds its fire-rate at the target across both scales where the fixed threshold drifts. So the attention architecture is bounded: two guidance levels is the sweet spot, and the keeper from this round is the gate that calibrates itself. This completes the arc, with an honest ceiling on its depth.

The question

The previous three rounds built a two-level guide. A working memory one level up holds a theme, and that theme steers where each column at the bottom level looks. The round before this one gave the theme a learned gate, so it holds across a span and admits new context only at a boundary. At each step the chain runs one way: the higher level guides the lower one.

Two things were left on the table, and this round tests both.

The first is depth. One level guiding another is a start, and the brain stacks more than that: slower areas higher up, holding context over longer spans, reconfiguring the traffic of the areas below them. So the question is whether a third level helps. Add a slower context above the theme, let it guide how the theme buckets the same way the theme guides where the columns look, and a longer-range context should steer the scan better than the theme alone.

The second is the gate's threshold. The gate fires when its signal runs into the high tail, above the running mean by a fixed multiple of the spread. That fixed multiple was the previous round's named weakness: as the data grows the signal's tail thins, the fixed multiple catches less of it, and the fire-rate drifts down. The fix is a threshold that targets a fire-rate directly rather than a fixed multiple, so the gate fires at the same rate whatever the scale.

What we tried

Two changes from the previous round, and everything else held identical so the comparison is clean.

• A third level, slower, guiding the second. The new level is a second theme, shaped like the first but admitted more slowly, so it holds over a longer span. Its gate fires on a slower read of the model's prediction, the shift of a slowly-leaked prediction rather than the step-to-step one, so it reacts to topic-scale turns rather than word-scale ones. Its coarse code picks a super-band, the second level picks a band within it, and the word-phase picks a slot. So the same theme and phase route to a different place to look under a different slower context: the bottom level specializes per slow-context, the deeper nesting.
• A gate that calibrates itself. The threshold keeps a running estimate of its own fire-rate and nudges an adaptive multiple toward a target. Fired too often, it raises the bar; too rarely, it lowers it. The multiple converges to whatever value puts the target mass in the tail, at any scale. It is the same online, gradient-free, no-reward gate, bounded to one extra running scalar, now self-correcting.
• Capacity matched, so the comparison is the mechanism. Keying the scan on context splits the bottom level's data across buckets, and more buckets starve the per-bucket policy and raise the bits per char on their own at small scale. The deeper nest uses more buckets to carry its super-band, so every arm here runs at the same bucket count. The third level is read against a two-level arm at the same capacity, and the self-calibrating gate against the fixed-threshold gate at the same capacity, so a difference is the change and not the bucket count.

The arms run over the same text, the same seeds, the same columns, the same span and feature and offset menu, differing only in the working memory. Fixed is the hand-set diverse offsets, the floor. Two-level, fixed threshold is the previous round's gate at this capacity. Two-level, self-calibrating is that gate with the threshold that targets a fire-rate. Three-level nested is the full deeper stack. Eight columns, two hundred thousand characters, the mean of three seeds.

What happened

The headline is prediction at the payoff scale, the mean over three seeds.

Read it as two findings.

The third level does not pay off. The three-level nested stack reads 3.096 bits per char against the two-level self-calibrating stack's 3.049, worse by 0.047, at matched capacity and at the payoff scale. The smaller scale agrees in direction, the third level a touch behind by 0.004 at fifty thousand characters and well behind here. And the nesting is genuinely working, not inert: the slow context routes the middle level to different buckets in its different bands, cross-band divergence a full bit, and the middle level routes the bottom one to different offsets, the previous round's signature carried through. Both top-down links are live. The third level simply costs more than it returns: splitting the buckets nine ways to carry its super-band fragments the bottom level's data more than its longer-range guidance gains back. Letting a slower context steer the steerer is real, and on this substrate it loses.

The self-calibrating gate pays off. At the same capacity the self-calibrating gate reads 3.049 bits per char against the fixed-threshold gate's 3.072, better by 0.023, and it beats the hand-set floor's 3.101. And it does what the fixed threshold could not: it holds its fire-rate at the target, about one fire in ten, at both fifty and two hundred thousand characters, by construction, where the fixed multiple drifts down as the data grows. So the previous round's named weakness is fixed. The gate that targets a fire-rate stays calibrated across scale, and at equal capacity it predicts better than the fixed multiple it replaces.

So the two questions split. Depth is a diminishing return, and the gate fix is a keeper.

The lesson

The attention architecture's depth saturates at two guidance levels. A third nested level, a slower context steering the steerer, is real and active (it routes the middle level, which routes the bottom one) and still does not pay off, even at the payoff scale, because the extra bucket-splitting it needs costs more than its longer-range guidance gains. The keeper from this round is the gate that calibrates itself: targeting a fire-rate rather than a fixed multiple, it beats the fixed-threshold gate at equal capacity and holds its rate across scale, fixing the previous round's named weakness. Two levels of guidance, with a self-calibrating gate, is the sweet spot on this count substrate.

This bounds the attention track rather than failing it. The first round learned where to look and, context-free, tied the hand-set spread. The second gave the scan context and beat the baseline. The third gated the theme and beat it again. This round asks how deep the climb goes and answers two-deep: the stack wins with one level guiding another, and a third level above them does not add. The honest shape of a result is sometimes a ceiling, and this is one. It also leaves the gate better than it found it, pinned to a fire-rate so it stays calibrated as the data grows.

The honest caveats

• The depth result is read at matched capacity. The third level is measured against a two-level arm at the same bucket count, so the difference is the level and not the buckets. Against the previous round's gate at its own smaller bucket count the numbers move for a second reason, fragmentation, so that comparison is kept only as the absolute floor, never as the depth test.
• The third level loses as built. A coarser super-band is a thin read of a slower context. A richer third level, or much more data to fill the split buckets, could let depth pay where it does not here. The fragmentation amortizes with scale in principle. On this substrate, at fifty and two hundred thousand characters, it does not, and the result is reported as the bound it is.
• The gate fix is modest at these scales. The self-calibrating gate wins by 0.023 bits per char here. Its larger payoff is against the fixed threshold at scales past this, where the fixed multiple under-fires hardest. The win it shows now is the holding of the fire-rate across both scales, which the fixed threshold cannot do.
• One substrate, trimmed scale. Eight count columns on the CPU, fifty and two hundred thousand characters, three seeds. The direction across the two scales and the seeds is the finding, not a single scaled number.

The frontier

The attention stack is now bounded and complete: learn where to look, guide it by context, hold the context behind a gate that calibrates itself, two levels deep, the whole thing online and gradient-free and beating hand-set fixed views. A separate round put this same attention representation on the abstraction ruler and found it a prediction win, not an abstraction one: the road past the abstraction ceiling is a change to the credit pathway, not a richer scan. So the attention track has its prize and its ceiling both. What a deeper guide would need is the open thread it leaves: a third level pays only with a richer read of the slower context or far more data than this, and that is a design question of uncertain payoff, named here, not solved.

Lineage

Grew from a gate for working memory, which gated the theme on a fixed threshold and named both of this round's tests, a deeper hierarchy and a fire-rate-target gate; from where you look depends on what you understand, whose two-level guide this round tried to deepen; and from learning where to look, whose lateral inhibition the whole stack still keeps.

Thread: attention, the program's new direction, here brought to its honest bound. The mechanisms are a third nested level (a slower context routing the middle level) and a self-calibrating gate (a threshold that targets a fire-rate), both online and gradient-free with no reward. The result is a ceiling and a keeper: depth saturates at two guidance levels, and the gate that calibrates itself stays the refinement worth holding. This completes the arc from learning where to look to a bounded, calibrated, two-level attention stack.