The honest scorecard

The benchmarks

A living scorecard: the locked model measured across four angles (compression, abstraction, continual learning, efficiency) against a SOTA transformer and the plain baselines both. Near 2.1 bits per character and plateauing, above the backprop ceiling on abstraction with no gradient at all. Re-run and redrawn on every architecture update.

2026-06-28 · the benchmark suite · re-run on every architecture update

A model is only as honest as the company it keeps. So we put the locked cortex next to the thing everyone means by "a language model" (a large transformer trained for many epochs with backpropagation) and next to the plain baselines a count model should be measured against, across four angles at once. The result is a clear, unflattering, useful picture. On raw compression we are far behind a SOTA transformer, and behind a plain n-gram, and that gap is real. But the comparison that matters is not "us versus a thing trained a thousand times over the corpus." It is "us versus a transformer held to the same rule we live under" (one online pass, no epochs, no going back) and on the axes that rule is for, abstraction geometry and learning from little data, the picture turns. On the program's home axis, the abstraction of the learned code, we sit above the gradient/backprop ceiling, with no gradient at all.

This page is a living scorecard. Every number here is produced by one re-runnable suite, stored with a version label, and redrawn on every architecture update, so as the model changes, the bars and the progression lines move with it. v1 is the locked architecture (the depth-three apical stack, settled above the ceiling).

How to read it

There are two regimes on every chart, and keeping them apart is the whole point.

• Our regime: online, single pass, no epochs, gradient-free, bounded memory. The cortex lives here. To place it fairly on prediction we run two baselines on the identical data: an order-six count n-gram (the count floor a count-shaped model should be measured against) and a small gradient char-Transformer (the gradient anchor at our data scale, trained with backprop, early-stopped on held-out). This is the fair fight.
• The other regime: multi-epoch, backpropagation, the whole corpus seen hundreds of times, hundreds of millions of parameters. This is where the cited SOTA lives (Transformer-XL, the Compressive Transformer). We cite it, never run it, and we mark it as a different regime everywhere, because comparing a single online pass to a thousand backprop epochs on raw bits is not a fair fight; it is a category difference, and pretending otherwise would be dishonest in the opposite direction.

Every baseline is labelled RUN (we ran it here) or CITED (a published number, attributed).

Angle 1 · Compression: how far behind are we, honestly

The standard char-LM axis: bits per character on text8, lower is better. This is the calibration. Where does kinogaki-cortex sit versus the field on raw next-character prediction? The answer comes in two halves: a cited ladder that places us against the published rungs, and an apples-to-apples head-to-head we ran ourselves on identical data. Both are on bits per character (bpc) on text8, the one axis where the comparison is fair. (These numbers update as the model improves; this is the calibration as of the current build.)

The short version, before the detail: we sit near 2.1 bpc. We clear the no-skill ceiling (4.755) comfortably, but a plain order-six n-gram beats us at every scale and a small gradient transformer overtakes us once it has enough data. As a raw next-character compressor the locked cortex is behind a large transformer and behind a count table, and we will not dress that up. The honest reading is that the locked architecture is not built to be the best next-character predictor; it is built to learn an abstract, generalizable code, and prediction was never the axis it won on (the program found this directly: voting and diverse views improve prediction, per-unit apical credit improves abstraction, and they are different jobs).

The cited ladder, units tagged

One rung per row, easiest at the top, frontier at the bottom. The unit is tagged on every row, because the denominators differ and do not convert cleanly. text8 bpc is over a 27-symbol alphabet (a to z plus space). enwik8 bpb is bits per byte over a 256-symbol byte alphabet, a harder and different corpus. Perplexity is per BPE sub-word token, a different axis again. Read across a denominator and the comparison breaks, so the ladder keeps them separate and only the text8-bpc rows are strictly one-to-one with us.

We clear the no-skill ceiling comfortably and sit near 2.1 bpc. The simple-gradient rung (a 2012 LSTM at 1.54) is the nearest named neural step above us; Transformer-XL at 1.08 is the landmark transformer on our exact corpus; the wall on text8 is about 1.0. The enwik8 and Chinchilla rows are on a harder byte corpus and a giant-budget compressor, plotted as a different corpus and a different sport, never on our rung.

The apples-to-apples head-to-head

The ladder mixes regimes, so we also ran a fair fight. Three learners on the identical text8 slices, the identical bpc loop, the same held-out evaluation of 20,000 characters at each scale: a plain order-6 backoff n-gram (the count floor), a gradient char-Transformer (805K params, a nanoGPT-class net trained with backprop, early-stopped on held-out), and our model (BLEND-NORM, 12-column, online, single-pass, gradient-free). Same data, same metric, nothing hidden.

Read the scaling story straight. Our model plateaus: 2.234 to 2.112 to 2.109, flat from 1M to 10M. The n-gram keeps falling (1.991 to 1.829) and so does the transformer (2.074 to 1.938). Past about 1M characters we do not turn more data into better prediction, while both baselines do.

A plain order-6 n-gram beats us at every scale, and the gap widens with data: even at 200k, and by 10M it is 0.28 bpc ahead. On raw next-character prediction, a dumb count table beats our voting machinery. The gradient transformer starts worst (it overfits 200k), then overtakes us by 1M and pulls clear: it scales the fastest of the three.

The plateau is a fixed-capacity artifact: a growable bank breaks it

The plateau is a hypothesis about capacity, and that makes it testable. The fixed bank is twelve voting columns, the same twelve at every scale. The n-gram is not fixed: as the data grows its context table grows with it, and a longer context is a sharper prediction, so it keeps converting data into lower bpc. So the question is sharp. If the flat spot is a fixed-bank artifact, a bank that grows should break it.

We built the grow-then-prune the program had named, on the identical text8 split and the identical bpc loop. The growable bank starts identical to the fixed one, then grows a longer-context voting column whenever the held-out gain decays (the columns have saturated), caps the count, and prunes the least-used column at the cap. Online, count-native, gradient-free, single pass, under a memory budget.

The fixed bank plateaus and the growable bank does not. Past 1M the fixed bank moves 2.112 to 2.109, a drop of 0.003, flat. The growable bank moves 2.004 to 1.875, a drop of 0.129, still falling. At 10M the growable bank reads 1.875 against the fixed bank's 2.109, 0.233 bpc ahead, within 0.047 of the n-gram floor at 1.829. So the plateau was a fixed-capacity artifact, not a wall. The growable bank restores the n-gram-like scaling slope by doing what the n-gram does: it adds longer-context views as data arrives (its longest look-back span climbed from five to eight), under a cap. It grew where there was data to earn it and not at 200k, where it correctly stayed at twelve columns and tied the fixed bank.

This does not change the honest placement. The growable bank reaches about the n-gram (1.875 against 1.829), not past it, and it stays about 0.85 bpc behind text8 SOTA near 1.0. And prediction is not our headline axis. What it changes is the reading of the plateau: it is not a hard limit of the substrate but a property of a fixed bank, and a count-native, online, bounded substrate that grows keeps learning from data where the fixed one saturates.

With repetition: the fair axis

One asymmetry in the head-to-head is worth naming, because the fair fix is free. The gradient transformer is trained multi-epoch: it sees the corpus many times, early-stopped on held-out. Our learned stacks see each character once. The fair comparison gives the learned stacks the same thing a multi-epoch gets, more than one exposure, in the form our rule permits: re-reading the same bounded slice, online, no reset between passes. Re-reading is one online, single-datum, gradient-free update each time, not a gradient epoch, so it stays inside the rule.

It helps, and it does not overfit. On the apical learned stack, re-reading a fixed 200K slice five times lowers held-out bpc from 3.86 to 3.76 with the train-versus-held-out gap flat, no overfit, and it beats fresh data at matched compute: 200K read five times reads 3.762 against a million fresh characters' 3.987, because the fresh single pass drifts while re-reading stays in a stable basin. And the floor goes lower with a gentler learning rate. The 3.76 was measured at the stack's locked rate (0.005), which is too aggressive: it crams the slice on the first pass and then runs flat. Drop the rate to 0.001, the sweet spot of a U-shaped sweep, and re-reading compounds across all ten passes to 3.637 against the locked rate's 3.762, 0.125 bits better at matched exposure, still overfit-free. The full results are in re-reading the same text and gentle learning compounds.

But it does not move the headline, for one clean reason: the count models are re-reading-invariant. A count table is complete after one pass. Read the same text again and the second pass adds the same counts, normalized away, so the n-gram and our ~2.1 voting bank are already single-pass-fair, have no learning rate to tune, and re-reading does nothing for them. Repetition is a real, fair lever for the learned stacks, which are not our best predictor, and it leaves the ~2.1 headline and the growth that breaks the plateau exactly where they stand. It is a prediction lever only: transfer CCGP holds flat under re-reading (about 0.53, above the 0.484 ceiling), so it does not touch abstraction.

The honest verdict

Prediction is not our axis. We clear no-skill comfortably and are competitive at tiny data, but on bpc the fixed bank sits about one bit behind the simple-gradient rung. We will not dress that up: a count n-gram beats us, and a small gradient transformer beats us once it has enough data. The fixed bank also plateaus, and the growable bank breaks that plateau, reaching about the n-gram, so the flat spot was fixed capacity rather than a wall, though the placement against SOTA is unchanged.

The model's bet is what neither the n-gram nor the tiny transformer has: the abstraction the CCGP probe tracks (it rises with depth and sits about 0.5 to 0.6, above the 0.484 gradient ceiling), the online, bounded, gradient-free regime, and the attention, generation, and working-memory machinery the rest of this program builds. The bpc plateau is the stability and bounded-capacity limit made concrete on a public yardstick, and the growable bank is the bounded-capacity half of it dissolved: add longer-context views as data arrives, under a cap, and the substrate keeps learning instead of saturating. The same capacity story drives the sparse-substrate redesign, where delivering the per-unit credit on a sparse, capped substrate is what holds abstraction stable.

What is comparable is bpc on text8: the ladder above and this head-to-head. What is not comparable is MMLU and the knowledge and reasoning frontier. A 27-character count learner has no world-knowledge mechanism and does not compete there, in either direction.

Angle 2 · Abstraction: the axis we win on

This is the program's home axis, and the one where the comparison flips. The metric is transfer CCGP, the cross-condition generalization of the learned representation (Bernardi/Fusi), measured on held-out text the model never trained on. Higher means a more abstract, factorized, transferable code.

The locked model's code reaches 0.537 transfer CCGP, above the 0.484 ceiling that a backprop-trained net of the same shape reaches, and it does it with no gradient anywhere, by a per-unit precision-gated apical credit signal. The abstraction also climbs with depth (0.373 at level one, 0.481 at level two, 0.537 at level three) the rising-altitude code the program spent the whole abstraction line chasing. Against the floors this is unambiguous: a random untrained net scores 0.28, the raw input 0.29, and our learned code clears the gradient mark.

So on the axis that asks "did the model build an abstract, generalizable representation?", the gradient-free cortex is competitive with, and slightly above, a gradient net: the one place the apples-to-apples comparison to backprop lands in our favor. That is the headline of the suite.

Angle 3 · Continual learning: where the count organ wins, and the abstraction organ does not

The continual-learning angle is about forgetting under a real distribution shift, and it carries an honest split the program insists on. We stream three genuinely different registers in one pass, no replay (Darwin's Victorian science, Shakespeare's verse, the King James Bible) and measure backward forgetting: how much an earlier register's cost degrades after the later ones flood in. Lower is better.

The win here belongs to the count substrate, and it is a decisive one: under a real register shift with a memory budget, the brain-inspired count model (two-timescale importance, ART vigilance, broadcast) forgets about 21 times less than a flat recency cache, at equal-or-better peak. That is kinogaki-cortex's genuine non-forgetting result.

But it belongs to the count organ, not the abstraction organ, and the suite says so out loud. The locked abstraction stack is a trained feedback-alignment net, and trained nets forget: it loses +0.113 bits here, less than a plastic LSTM's catastrophic +0.866, but it is not a non-forgetting model and we will not claim it is. Non-forgetting is the count substrate's job; abstraction is the apical stack's job. The honest version of "does kinogaki-cortex forget?" is: the part of it that counts barely forgets, the part of it that builds the abstract code forgets like the trained net it is, and these are different organs doing different work. The progression line for this angle tracks the count substrate's retention, the program's actual continual-learning capability.

Angle 4 · Efficiency and compute: the early-data regime

The last angle is about what each bit costs, and it is the counterweight to Angle 1. The cortex is online, single-pass, and bounded-memory: it sees each character once, updates once, and lives under a fixed budget, no epochs, no replay buffer over the whole corpus, no gradient. A SOTA transformer's bits are bought with a training budget orders of magnitude larger.

The head-to-head in Angle 1 is run under exactly this rule: our model sees each text8 slice in one online pass. The fixed locked config's bpc improves with more fresh data only in the ~1M-character window; past it, on a long fresh non-stationary stream, the locked config drifts: held-out bits-per-char rises and the deep code collapses, the plateau the head-to-head plots (2.234 to 2.112 to 2.109, flat from 1M to 10M). The growable bank lifts the bounded-capacity half of this (it keeps dropping to 1.875 by adding longer-context views under a cap), still in one online pass. The cheap-and-online efficiency story holds early; staying stable over a long stream is an open continual-learning problem (the count organ's ~21×-less forgetting, Angle 3, is one of the mitigations).

So the raw-bits gap and the compute gap point the same way and partly cancel: we are behind on bits, and far ahead on what those bits cost to learn. The honest framing is not "we beat the transformer" (we do not) it is "we are a different, much cheaper kind of learner, and the axes where cheap-and-online is the goal are the axes we hold."

The honest one-paragraph read

How far are we from a large transformer LLM, and where does the gap matter? On raw compression we sit near 2.1 bpc on text8: above the no-skill ceiling, behind a plain order-six n-gram at every scale, and overtaken by a small gradient transformer once it has enough data, against a SOTA transformer's ~1.0. That gap is real and unspun. The fixed bank plateaus past ~1M characters while both baselines keep falling, but a growable bank breaks that plateau (2.004 to 1.875, near the n-gram), so the flat spot is fixed capacity, not a wall, and the placement against SOTA holds either way. The gap matters if you want a raw compressor; it matters much less as a verdict on the learning principle. On the axes our regime is for (the abstraction of the learned code, where we sit above the gradient/backprop ceiling with no gradient at all, and continual learning, where the count substrate forgets ~21× less than a recency cache) the gradient-free, online, bounded cortex is competitive with or ahead of a gradient learner. So: behind on bits and plateauing there, even or ahead on abstraction and on cheap continual learning, and honest that the non-forgetting win is the count organ's while the above-ceiling abstraction is the apical stack's. We are not an AGI breakthrough and not a fluent model; we are a validated, inspectable, brain-shaped learner that is strong exactly where its constraints are the point.

Progression: the scorecard grows

These charts start with one point (v1, the locked architecture) and gain a point on every architecture update, so each headline metric can be watched over time. As the model changes, the suite is re-run with a new label and every chart redraws itself; benchmarking is a single command, by design.

v1 (2026-06-28, locked architecture): head-to-head bpc 2.109 at 10M (the corrected BLEND-NORM pool, online single-pass) · transfer CCGP 0.537 (above the 0.484 ceiling) · count-substrate retention 0.96. Compression is tracked by the head-to-head in Angle 1; abstraction and retention are plotted above.

To add the next version, the suite is re-run with a new label and every chart redraws itself, so the four progression lines above are how a single headline metric is tracked across the program's architecture history.

Lineage

Grew from the architecture locked above the ceiling (the v1 model this measures), the counter that beat the neural net and non-forgetting (Angle 3's count-substrate win), and abstraction that climbs (Angle 2's above-ceiling code and the prediction/abstraction split that explains Angle 1).

Thread: measuring the program honestly. The promise is a gradient-free, online, bounded learner that is real where its constraints are the point, and this is the scorecard that holds it to that, across four angles, against the state of the art and the plain baselines both, refreshed on every change.