A baseline on dialogue
2026-06-28 · a baseline on real conversation · experiment DIALOG-BASELINE
The generation track has a producer and a measuring stick for invented words, but it never read real conversation. This round gives it one. The locked model streams strict-online over the whole DailyDialog training split, 5.24 million characters, and is then measured on held-out dialog it never saw. Two honest readings. Dialog is a bounded, repetitive distribution the model fits stably: held-out bits per char drops with data, from 3.767 at 200 thousand characters to 3.549 at the full split, with no drift, and it sits 0.542 bits below the same model's text8 reference of 4.090. Everyday talk is easier than encyclopedia text for a model trained on talk. But the second reading is a warning. Over a long pass on the narrow dialog register, general grammaticality erodes: a minimal-pair grammar score falls from 64.8 percent at 200 thousand characters to 51.1 percent at the full split, word-order phenomena holding at 100 percent while agreement and the others sink to chance. The free-run samples are gibberish with the right shape: correct spacing, the turn marker recurring, no words. This is the starting line, measured, not a new mechanism.
The question
The generation track has built a producer and the instruments to grade it, but every measurement so far was on controlled vocabularies and synthetic meanings. What it lacked was a reading on real conversation: where the model stands when it reads human dialogue, and a stick to watch that number move. So this round establishes a baseline. It is measurement, not a new architecture. One locked model, the depth-three apical stack settled above the backprop ceiling, streamed online over a real dialogue corpus, then measured on held-out dialogue and on a grammar battery. The question is plain: how well does the model fit real conversation, how does that compare to the encyclopedia text it has always been measured on, and what does training on a narrow everyday register do to its general grammatical sensitivity.
What we tried
The corpus is DailyDialog, a manually written set of everyday multi-turn conversations: chit-chat about food, work, weather, relationships. It is cleaned to the model's alphabet, lowercase letters and space, every other character mapped to one space. Turns are segmented with an in-alphabet marker the model can see and learn, a rare all-letter word standing in for the turn boundary, so the model reads one flat stream with the boundaries visible in it. The training split is 5.24 million characters; the test split is held out and never trained on.
The model is the locked configuration, driven exactly as the scale runs drive it: ids streamed in id space, featurized on the fly in memory-safe blocks, trained strict online, one pass, batch of one, in compiled chunks. The locked defaults are reused verbatim. Nothing about the model changes; only the corpus does.
Three measurements read the trained model, all on the same read-side contract a producer is scored through.
- Held-out dialog bits per char, over a slice of the test split the model never saw. This is the new standing measuring stick for the generation track, the second after text8.
- A text8 reference, the same trained model scored on a held-out slice of encyclopedia text, so the comparison is clean: only the evaluation corpus changes, the model and the stick stay fixed. This answers how much easier or harder real dialog is than text8 for this model.
- A minimal-pair grammar battery, the first such score for this model on any corpus. It scores hand-built pairs of a grammatical and an ungrammatical sentence by per-character surprisal: a pair is correct when the model is less surprised by the grammatical one. The check is run at 200 thousand characters and again at the full 5.24 million, to read what a long pass on the narrow register does to grammar.
A handful of free-run samples close it out, the intelligibility starting line to watch improve.
What happened
The two readings sit on two axes, one reassuring and one a warning.
| the locked model on real dialog, single seed | held-out dialog bpc | text8 reference bpc | grammar macro |
|---|---|---|---|
| 200 thousand chars | 3.767 | 64.8% | |
| 5.24 million chars (full split) | 3.549 | 4.090 | 51.1% |
Read it as two findings.
Dialog is a bounded distribution the model fits stably. Held-out bits per char drops with data, from 3.767 at 200 thousand characters to 3.549 at the full 5.24 million, and it keeps dropping with no drift. This is the opposite of the long fresh text8 stream, where held-out bits per char rises on a never-repeating feed. Dialogue is narrow and repetitive, the same everyday vocabulary and the same turn markers over and over, so more of it makes the model better at it rather than washing it out. And on the same stick, the dialog-trained model reads 3.549 on held-out dialog against 4.090 on text8, so real conversation is 0.542 bits per char easier than encyclopedia text for a model trained on conversation: narrow vocabulary plus the repeated turn markers.
Training on the narrow register erodes general grammar. The grammar macro falls from 64.8 percent at 200 thousand characters to 51.1 percent at the full split. The split tells the story: word-order sensitivity holds at 100 percent, the model still confident by a wide margin that a filler-gap sentence beats its scrambled twin, while subject-verb agreement, determiner-noun agreement, and negative-polarity licensing sink to or below the 50 percent chance line. The longer the model trains on everyday talk, the better it fits that talk and the less it reads as a general grammar. Fitting a narrow distribution and holding broad grammatical sensitivity pull against each other, and over a long pass the fit wins.
The free-run samples are the intelligibility starting line, and they are honest. At a sampling temperature of 0.8 the model produces gibberish with the right shape: the space density is right, and the turn marker recurs, but the strings are not words. Prompted with "i think that ", it continues "o yli e e cilk int ee wefr be wot et eoke". The structure is there, the words are not. That is where generation on real dialog begins.
The lesson
Give the generation track a real-conversation measuring stick, and it reads two ways. Dialogue is a bounded, repetitive distribution the locked model fits stably: held-out bits per char drops with data, 3.767 to 3.549, no drift, and it is 0.542 bits easier than the same model's text8 reference of 4.090. That is the reassuring half, the mirror image of the fresh-text drift. The warning half is specialization: over a long pass on the narrow register, general grammar erodes, the minimal-pair score falling 64.8 percent to 51.1 percent, word order holding at 100 percent while agreement sinks to chance. And the producer's free-run output is gibberish with the right shape, correct spacing and the turn marker recurring, no words yet. A baseline, honestly: the starting line on real conversation, and a stick to watch it move.
This is a measurement, and it places the generation track on real conversation for the first time. The reassuring reading and the warning reading are the same finding seen twice. A bounded distribution is one the model can fit and keep fitting, which is why held-out bits per char drops on dialog where it rises on a fresh stream, and why dialog comes out easier than text8. But fitting a narrow distribution hard is exactly what costs the broad grammatical sensitivity, because the register the model is fitting does not exercise the full range of grammar the battery probes. The specialization that buys the dialog fit is the same specialization that erodes the grammar score. It is the same family as the program's drift theme, read from the other side: there the worry is a model washed out by an endless fresh feed, here it is a model narrowed by a long pass on a small register.
The honest caveats
- A single seed. The whole baseline is one run at seed zero. The numbers are a starting point, not a distribution.
- The grammar battery is an analogue, not the full suite. The score comes from a hand-built set of minimal pairs over a handful of phenomena, the in-process stand-in for the published battery, not the published battery itself. It reads grammatical sensitivity through character surprisal, and its honest blind spots are known: a character counter cannot see scope, so the negative-polarity phenomenon was never going to clear chance. The 64.8 percent to 51.1 percent fall is a within-experiment comparison, the same battery on the same model at two data points.
- The grammar reference is a different model. A count-band model on text8 scored 60.2 percent on this same battery, but that is a different model on a different corpus, context only. The finding here is the change within this model as it over-trains, not a head-to-head against that 60.2 percent.
- Generation quality is gibberish, by design at this scale. The free-run samples have the right shape and no words. This is the intelligibility floor the generation work has to lift, stated plainly, not a producer being claimed as fluent.
The frontier
The generation track now has a real-conversation baseline and a standing stick to move it. The two readings name the two next moves. The bounded-distribution stability is the encouraging surface to build the interactive loop on: a model that fits dialogue and keeps fitting it is the right substrate for a producer that reads turns and replies. The grammar erosion is the warning to design around: a long pass on a narrow register trades general sensitivity for fit, so the producer that learns to converse will need a way to hold its broad grammar while it specializes, the same anti-drift question the program already carries, met from the specialization side. And the gibberish-with-the-right-shape samples are the number to watch: correct spacing and a recurring turn marker today, intelligible turns the goal. The stick is in place. The next run is the loop that moves it.
Lineage
Grew from internal self-feedback teaches production, the birdsong loop that opened the generation frontier; producing on purpose, the separate inverse that gave the track a working producer; and what it learns and what it can say back, the coverage-and-playback instrument that measured the comprehension-production gap this baseline now reads on real text.
Thread: generation, and the drift theme it shares a spine with. The reassuring half is the mirror of fast, and at scale, where a long fresh stream made held-out bits per char rise; here a bounded distribution makes it drop. The warning half, specialization erodes generality, is the same family as the drift the two mechanisms that compose work fights, read from the narrow-register side. The frontier is the interactive loop, and a producer that holds its grammar while it learns to converse.