Less is more, and you can prove it
Theory update. 2026-06-26 · clear win · experiment AJ
Validity-ordered, noncompensatory, early-stopping inference beats full geometric-mean integration on every axis at once: accuracy, perplexity, and compute. Less is more, and the count model is exactly the place it can be proven.
The question
Our standing combiner pools every cue with a geometric mean (all the offset-experts, the soft accumulated channel) paying full compute to weigh evidence that mostly does not matter. The fast-and-frugal program says the opposite: rank cues by their measured validity, consult them one at a time high-validity-first, and stop at the first cue that discriminates. A lower-validity cue never overrides a higher one that already fired: the rule is noncompensatory. Simon's satisficing sets the stopping bar; Gigerenzer and Goldstein's take-the-best does the ordering.
The bias-variance decomposition says why this should win, not merely save compute. A count model is high-bias and low-variance by construction. When data is sparse and single-pass, that is the right place to be, and a frugal rule that ignores weak channels trades a sliver of bias for a large cut in variance: the less-is-more effect. We asked whether a validity-ordered, early-stopping rule beats the full pool, and whether ignoring the weak channel helps exactly where the theory says it should.
What we tried
Word level on text8, top 40,000 words, eight offset-experts reused from attention, but counted, 18 MB prefix, last 80,000 words held out. Each cue's ecological validity is counted online: its running argmax scored against the next token, v = hits / (hits + misses). No gradient, no fitted knob. Take-the-best scans the crisp cues (bare recognition and the eight offsets) in descending validity; the first whose top-1 probability clears a satisficing bar decides, and we stop. The bar is a leaky accumulator over the deciding cue's discrimination: it rises when recent calls are confident, falls when they are scarce, so it sets itself. The soft geometric-mean pool is demoted to a last-resort fallback, and the less-is-more rule lets it take over a crisp cue only when its validity is higher; when it is the weaker channel, it is ignored.
The immediately-preceding word was by far the most-valid cue (validity 0.175); validity then decayed fast and plateaued, the same shape the offset work found for information gain. The satisficing bar settled at 0.24.
What happened
| combiner | top-1 accuracy | perplexity | cues / step |
|---|---|---|---|
| full integration (all offsets) | 9.71% | 7,160 | 8.00 |
| take-the-best (validity-ordered, early-stop) | 15.00% | 1,918 | 4.56 |
Less is more, and it is not close. Take-the-best beats full integration on accuracy (+5.3 points), cuts perplexity 3.7×, and does it at 43% less compute. It does not merely match the pool at a fraction of the cost; it wins on every axis at once. The reason is the bias-variance story made concrete: unweighted integration pools eight experts of wildly different validity, and the seven weak far-offsets dilute the sharp nearest-word signal, dragging the argmax off target. Letting the single most-valid cue decide, and ignoring the rest, is the right trade for a sparse single-pass count model.
And the less-is-more effect appears exactly where predicted, on sparse contexts, where the soft pool is thinnest:
| rule, on sparse contexts | top-1 accuracy | perplexity |
|---|---|---|
| ignore the weak channel (α>β) | 11.68% | 3,443 |
| always integrate it | 10.96% | 4,623 |
| full integration | 7.41% | 7,959 |
On the positions where the soft channel is least reliable, not consulting it is the better call: +0.72 points on the sharp test, and both frugal rules crush full integration where over-pooling hurts most. The weak channel is not just useless on sparse contexts; using it actively hurts.
One honest negative came with it. A base-rate prior on the leader-clusterer (tilting assignment toward big, well-attested prototypes) was supposed to steady it under perturbation. It did the opposite: pure similarity (γ=0) was the most stable at 0.982, and any γ>0 lowered stability. The representativeness intuition is right in spirit but the wrong knob for a single-pass, no-reassignment clusterer, so γ=0 stays.
The lesson
Stop asking "how do we weigh all the evidence?" and ask "which one cue do we trust, and when do we stop looking?" Validity-ordered take-the-best with early-stopping reaches 15.00% accuracy at perplexity 1,918 and 4.56 cues a step, against full integration's 9.71% / 7,160 / 8.00: higher accuracy, far lower perplexity, 43% less compute. Pooling every cue dilutes the one that matters. And ignoring the weak channel wins, overall and on the sparse slice where it is thinnest, exactly the less-is-more prediction. This revises the standing combiner: validity-ordered take-the-best and early-stop, not full geometric-mean pooling.
A count model is high-bias by construction, which is precisely the place where stopping early wins. The combiner question the work has carried since the voting post, full pool versus something sharper, gets its clearest answer here: sharper, and frugal.
Lineage
Grew from attention, but counted, whose offset-experts are the cues take-the-best ranks, and from how sure is a count?, whose hit/miss truth value is exactly the ecological validity that orders them.
Thread: the right combiner. Take-the-best is the sharpening rule the scorecard kept asking for: it sharpens by ignoring, not by blurring. The ideas are Gigerenzer, Goldstein, and Simon, on fast-and-frugal heuristics and satisficing.