Keeping track of what's going on

Negative. 2026-06-26 · the control ate the result · experiment AM

A persistent situation model (event chains plus who/where/topic slots) does not predict over long spans. The half-bit "everywhere" win was pure smoothing repair: a static frozen unigram beats the live situation by −0.07 bpw on the 99% of text that isn't backoff. The situation helps only the same 0.9% backoff slice three mechanisms have now landed on. Measure a top-down prior against a static prior, never against no prior.

The question

A story is more than its sentences. When you read she three pages after a name, you know who. When a scene opens back at the cabin, you know where. You carry a running model of the situation (who is on stage, where, what is happening) and it quietly shapes every word you expect next. The dream on this thread has been to give a count cortex that same standing model: not just to notice when the topic changes, but to predict with a sense of what is going on.

The event model got the noticing (Bayesian surprise found article boundaries beautifully) but its persistent slot only helped predict on about 1% of the text, the words where the local n-gram had already run out of evidence. Ignition found the same line for a global topic state. Two roads, one law: a top-down prior pays off only where local prediction fails. That 1% nagged. Maybe the slot was just too crude. So we built two of the discourse literature's best ideas into the substrate, online, no backprop, and asked the blunt question: does a real situation model predict over long spans, or does it just rediscover the 1%?

What we tried

Two ideas from how people read. Narrative event chains (Chambers and Jurafsky): people learn that arrested follows charged; count the ordered event pairs that share a protagonist, score them by surprising co-occurrence, let the current event predict the expected next. With no parser, an "event" is the concept cluster of a content word and "shared protagonist" is the recency of a shared entity-ish cluster. Typed situation slots (Zwaan's event-indexing model): not one slot but several (who is active, where we are, the slow background topic) each a leaky accumulator nudging the word distribution, sorted by cheap surface signals (protagonists are rare and high-IDF; places follow in/at/near). Both feed one soft top-down prior, and this time we mixed it in everywhere, not just the backoff slice. If a situation model is real, it should help generally.

What happened

It looked like a triumph. Blending the situation prior in at every step cut bits-per-word by half a bit on the 99% of text the slot had never touched. After three experiments of "+0.001," here was "+0.55." It was a trap. A sparse smoothed trigram table is badly calibrated on its tail, and blending it against almost any broad word prior repairs that, whether or not the prior knows anything about the situation. So we ran the only control that matters: a static prior, the plain corpus unigram, frozen and situation-blind, mixed in the exact same way.

The frozen unigram cut more bits than the live situation. Tracking who and where and what was happening made next-word prediction −0.07 bits worse than tracking nothing. The half-bit was smoothing repair, top to bottom. None of it was understanding. And where the situation did beat the frozen prior, by a real +0.06 bits, was exactly the same 0.9% backoff slice the event model found last time. The marquee mechanism, the Chambers-Jurafsky event chain, barely predicted at all and hurt on the slice that mattered: without a parser and real coreference, "event = content-word cluster" is too lossy. The dimension that "helped" most was the slowest, broadest topic slot, the closest thing to the static unigram, the best smoothing-repairer, not the best tracker.

The lesson

Three experiments now, three different mechanisms (a recency histogram, a Bayesian-surprise slot, a typed schema-driven situation model) and all three land on the same small island: a top-down prior helps a count predictor only where local context has run out, and only there. Stacking more structure onto the prior did not move the line. The methodological rule, written down: measure a top-down prior against a static prior, never against no prior. A sparse smoothed model is so hungry for tail mass that any broad prior flatters it, and you will mistake repair for comprehension every time if you skip the control.

The global-coherence frontier is still out there, but it is now clear it will not be crossed by bolting more persistent state onto a predictor whose every easy word is already decided locally. Whatever carries understanding across a long span has to change what we do on the 99%, not just rescue the 1%.

Lineage

Grew from what the model thinks is happening and when the whole room agrees on a topic, the same 1%-backoff law, now found a third time with richer machinery and a tighter control.

Thread: global coherence (open). The ideas are Chambers and Jurafsky's narrative event chains and Zwaan and Radvansky's event-indexing model.