What survives scale

2026-06-26 · Scaling study · 15 MB to 3 billion chars

At scale, more data absorbs any mechanism that just re-predicts what local counts already know. Only the mechanisms that do what counting can't survive.

The question

Every post here measured an idea on a few megabytes. That is the right size to find out whether an idea works, and the wrong size to find out whether it lasts. A mechanism can look like a win at 15 MB for the dullest of reasons: the local counts have not yet seen enough text, so anything that fills the gap looks clever. Pour in more data and the gap closes on its own. The mechanism was renting its win from data starvation.

So we re-ran every idea at thirty to two hundred times the data and asked one question of each: which verdicts change when data is no longer the bottleneck? Two overnight batches did the work. The first scaled the single mechanisms to half a billion words and three billion characters. The second re-ran the synthesis experiments at scale, each on its own right-axis metric. The point was never to find a new win. It was to learn which of the old wins were real.

What we tried

The corpus was the 1-Billion-Word benchmark, 526 million words and 3.03 billion characters, normalized to a 27-symbol space, plus enwik9 for the article-boundary truth and the Darwin, Shakespeare, and Bible register files for retention. Everything online, the house rule held without exception: counting, leaky accumulators, online leader-clustering, no gradient descent, no k-means, no SVD, no batch pass that revisits the data.

The trick was to refuse the single headline number. Bits-per-char would have flattered the counts and buried everything else. So each experiment was re-measured on the axis it was built to win: boundary F1 for the event model, unseen-context perplexity for the similarity backoff, calibration error for the confidence value, held-out generalization for constructions, the forward-minus-reversed gap for the trajectory memory. An idea scaled or it did not on its own terms.

What happened: the dichotomy

The results split cleanly into two piles, and the line between them is sharp enough to name.

Held or grew with scale. These do something counting structurally cannot.

The event model is the standout. Bayesian surprise did not merely survive; it grew, from F1 0.154 at 36 MB to 0.447 at 960 MB, while per-token surprisal stayed pinned to the floor at 0.032. Fourteen times better at finding real article boundaries, and the gap widened with every order of magnitude. The similarity backoff tripled its lead, from a twenty-fold to a fifty-six-fold perplexity win on contexts the counter has never seen. Calibration, construction generalization, and trajectory direction all held their wins intact. Even the plain char counter has not finished: order-5 was still dropping at three billion characters. Only order-4 saturated, at 1.95.

Vanished at scale. These were competing with the local counts on prediction the counts already own.

The topic prior was the clearest casualty. At 15 MB a global topic saved a third of a bit per word; at 864 million words it saved nothing, exactly nothing, because the local counts had by then seen the contexts the topic used to cover. The noise→concept shift inverted: at scale the model is already about 99.9% concept-reliant before any noise is added, so the headline 86%→95% climb turns out to have been a small-data artifact. Sleep's gain flipped sign: there is less lossy memory to consolidate when the tables are dense, though the rare-context tail still benefits. And the retention edge collapsed from twenty-one-fold to barely better than the baseline, which we mark soft: the register corpus is genuinely too small to be a real scale test, and the regime, not the data alone, moves it.

The lesson

At scale, "more data" absorbs any mechanism that competes with local counts on already-seen prediction. A mechanism survives scale only if it does something counting structurally cannot: generalize to the unseen, detect discourse structure, report calibrated confidence, encode direction, or retain under bounded memory. And every survivor is about representation and signal, not better prediction of the seen, the same frontier as the open question of a count-native combiner that sharpens without blurring.

The dividing line is not how clever a mechanism is. It is whether it is doing the counts' job or its own. Topic ignition, the noise gate, sleep consolidation: each, underneath, was a smarter way to predict a word the local context could already reach, and so each was always going to be outrun by more data. The survivors all sit somewhere the counts cannot go: a boundary is not a word, calibrated doubt is not a guess, a direction is not a content, an unseen pair has no count to refine. That is the whole map. Build toward the things counting cannot do, and let the data own the rest.

Honest caveats

A few measurements came back incomplete, and the bet is that a flat or broken big-scale result is still data. The similarity-hybrid rare-slice and the ray-cortex rare-slice both returned nan: zero probes landed in the rare bucket at scale, so those slices say nothing either way. Analogy (AD) did not truly scale: its fixed-size profile ran at 16 MB, so its "raw beats smoothed" verdict is unchanged but untested at volume. The retention corpus, as noted, is too small to be a real scale test, which is why its row is marked soft. And char order-6 was skipped outright: the table is a 42 GB float array before the int64 bincount temp doubles it, past the machine's memory; that one needs a GPU or a sparse path, and is owed a rerun.

Lineage

This is a meta-post. It summarizes the experiments it re-ran, and points back to each.

Grew from the wins that survived (what the model thinks is happening, the standout that grew with scale; use the map to read, not to walk, whose unseen-context win tripled; how sure is a count?; grammar is just counting, made productive; and you can't write your signature backwards) and from the mechanisms that dissolved into the counts: when the whole room agrees on a topic, when the letters lie, it leans on the idea, what an agent learns while it dreams, and learning the new without losing the old. The scaling rails come from more data helps, all the way to a gigabyte and the level that reaches past the last few words.

Thread: scale, and the right combiner. The survivors all point at the same frontier the scorecard named: a count-native combiner that sharpens the signal without blurring the representation. Scale did not name a new direction. It cleared away everything that was not it.