kinogaki/Kinogaki Cortex

A kept historical snapshot

The architecture, v1 (archived snapshot)

The frozen v1 architecture spec, superseded by the current spec at /architecture/. Kept as a historical snapshot so the page always shows the latest while nothing is lost: the depth-three apical-credit comprehension-and-abstraction core and the generation loop, before the two-engines reframe.

Architecture v1 (archived snapshot) - superseded by the current spec at /architecture/.

Architecture v1 · 2026-06-27 · the current canonical spec

This page is the architecture spec: the model the experiments have converged the code into, described in detail and explained. It is the counterpart to the lab notebook: the notebook is the running journal of every experiment; this page is the standing answer to "what is the kinogaki-cortex code, right now, and why is it shaped this way?" The architecture has two halves: a comprehension-and-abstraction core, which is locked and validated above the backprop ceiling, and a generation loop, just opened with its first positive. Experiments drive this page; when an experiment shifts the architecture, this spec is re-versioned and the change is logged at the bottom.

Overview

kinogaki-cortex is a brain-inspired model of text. It learns the way a person reads: online (every character teaches it, one pass, no retraining), gradient-free in regime (it learns on arrival without a global backward pass over the data), bounded-memory (it lives under a budget and must generalize to stay good), and guided by human cognition at every fork. Those are the four laws; they are not style, they are the constraints the whole program holds itself to.

The architecture is two halves, matching the two things a language user does: understand, and speak.

The rest of this page walks the data path end to end: encoder, the abstraction core (level by level, with the apical credit wire and the stabilizers, and why each piece is there), the readout, and the generation loop, then states plainly what is deliberately not in the architecture.

The data path, at a glance

The kinogaki-cortex data path: the input context flows through the random-SDR encoder into the depth-3 abstraction core (L1, L2, L3), feed-forward prediction up, per-unit top-down apical credit down, with actnorm plus gate-clip stabilization and a precision gate on every level, out to the next-char readout; and the self-feedback generation loop (SF1 birdsong) runs as a side path: set a target meaning, babble over the chunk vocabulary, re-comprehend through the same pathway, score the self-error, and win-stay / lose-shift.

Two pathways, one set of counts read two ways: reading selects the next character for fidelity to the stream; speaking selects an utterance for what a re-hearing of it recovers. The forward (feed-forward) wires carry prediction up; the apical wires carry credit down; the same precision signal is read three ways: as attention/vote gain, as working-memory eligibility, and as the per-unit credit amplitude.

1 · Input: the random-SDR encoder

The model never sees raw characters. It sees a fixed-random sparse distributed code of the recent context.

The alphabet is 27 symbols: a to z are 0 to 25, space is 26. Each letter is assigned one fixed random sparse code (k active bits out of n, about 2% density) drawn once from a seed and never changed. There is no learning in the encoder; it is a frozen, deterministic dictionary (LetterSDR in lib/sdr_encoder.py).

The context window is the last W = 5 characters. To encode it, each of the five letters' codes is position-tagged (rolled by offset × stride bits, so the same letter in a different slot lands on a different bit set) and all five shifted codes are OR'd into one input SDR (ContextEncoder.encode). Position-tagging is what lets the layer above tell "the a two back" from "the a just now"; OR-ing keeps the input at a fixed size n no matter how wide the window, and makes similar contexts (sharing letters at the same offsets) share input bits, the overlap a pooler needs to map similar contexts to similar codes.

For the abstraction core the SDR is then taken dense and L2-normalized into one vector of dimension n_in = 1024 (featurize_dense in lib/mlx_learner.py). That normalized dense vector is the input to L1.

Why a fixed random code and not a learned embedding: a learned embedding would need a gradient, and the whole point of the core is to build abstraction without one. The random SDR is a high-dimensional, sparse, content-preserving substrate the stack can carve, exactly the role the brain's input projections play in the geometry the program is chasing (the PFC neural-geometry result grounds this: learning shapes the geometry from high-dimensional-random toward low-dimensional rule-selective).

2 · The abstraction core: the locked model

This is the heart of the architecture, and the part the experiments worked hardest to settle. In code it is StabilizedApicalStack (experiments/exp_dj_stability/arch.py), a thin, validated configuration over MLXApicalStack (experiments/lib/mlx_learner.py). The locked configuration is: depth-3, actnorm plus gate-clip, act_gain ≈ 0.2, the apical credit gated by precision. Every clause of that sentence was bought by an experiment. Here is the stack, piece by piece, with the reason each piece exists.

The shape: a depth-3 stack with a next-char readout

The forward shape is a straight climb. The input SDR (n_in = 1024) feeds L1 through W[0] (n_in × H); L1 feeds L2 through W[1] (H × H); L2 feeds L3 through W[2] (H × H); and L3 feeds the readout through Wo (H × n_out), a linear softmax over the next character (n_out = 27). That is the prediction path of the diagram above, traced level by level.

Three hidden ReLU levels (H units each, typically 768 to 1024), then a linear softmax readout over the 27 next-character classes. The forward pass is ordinary: each level is ReLU(input · W + b), the top level feeds the readout. What is not ordinary is how the weights learn: there is no backprop anywhere.

Why depth 3, and not more. Depth is what buys altitude: each level out-abstracts the one below, and abstraction (CCGP) climbs L1 → L2 → L3. But it climbs to three and then saturates: depth-4 adds dimensionality without adding abstraction, and the deep levels collapse. That ceiling was established by scaling the stack (three swings after the keystone) and confirmed when the capstone meant to push past it ran and failed (the architecture is locked). Three is the settled depth.

The apical credit wire: the abstraction itself

Between every adjacent pair of levels there is a second wire running the other way: a per-unit, top-down apical credit signal. This is the mechanism that makes the stack abstract, and it is the single most important idea in the core.

The readout's error is dz = p − onehot(target). That error is routed back into the hidden levels not through the transpose of the forward weights (that would be backprop, which needs weight transport, a neuron reading its own outgoing synapses, which biology cannot do), but through fixed random feedback matrices: B_top from the readout to the top hidden level, and an apical wire Bap[ℓ] (an H × H fixed random matrix) from each level to the one below it. This is feedback alignment: the credit each unit receives is per-unit and local, carried down a real top-down pathway, with no weight transport. The forward weight then takes a local delta step against that per-unit credit.

The credit flows level by level, top to bottom:

  dz   = p − onehot(target)
  Wo  -= fa_lr · outer(h_L, dz)              # readout: its own local delta
  d_L  = (B_top · dz) · [h_L > 0]            # top level: error → top hidden via fixed B_top
  W_L -= fa_lr · outer(h_{L-1}, d_L)         # train the top forward weight (ungated)
  for ℓ = L-1 … 1:                           # carry the credit DOWN, level by level (apical)
      d_ℓ  = (Bap[ℓ] · d_{ℓ+1}) · [h_ℓ > 0]  # per-unit credit from the level above
      d_ℓ  = normalize(d_ℓ)                  # RMS-normalize the credit (gate-clip)
      W_ℓ -= (ap_lr · g) · outer(in_ℓ, d_ℓ)  # GATED by the precision burst g

The same top-down wire that carries this credit is the wire that would prime the level below during a forward sweep: priming and credit are one mechanism. This is the architectural keystone, and the reason the core exists in this form.

Why per-unit, and why this pathway. The program walked the abstraction question all the way down. First it proved the missing ingredient was credit assignment, not wiring: holding one architecture fixed and swapping only the update rule, a gradient builds an abstract space while a no-gradient Hebbian rule compresses hard but loses abstraction (the abstraction wall was credit assignment). Then it sharpened "credit assignment" to per-unit credit: a broadcast neuromodulatory scalar fails (one number everywhere can only scale an unsupervised step), while a per-unit feedback-alignment signal clears the wall, lifting abstraction to ~0.41 and approaching the backprop oracle's ~0.49 (a local signal clears the wall). Then it answered on what pathway, how biologically: carry that per-unit credit down a real apical wire, and the result is the first stack whose abstraction climbs with depth (the apical fusion). The apical wire is not decoration; it is the abstraction.

The precision signal: the third factor that gates the credit

The credit does not flow at a constant rate. Its amplitude is gated by a precision/surprise burst, the third factor. On each step the model reads its own surprise on the true character, s = −log₂ p(target), tracks a running mean of it, and forms a bounded burst g = g_floor + clip(s − running_mean, 0, burst_max). The apical (below-top) updates are scaled by g; the top forward weight and the readout are left ungated, so the gate's effect is isolated to the apical wires.

This is the "one precision read three ways" principle: the same precision burst is (1) the attention/vote gain that sharpens confident columns, (2) the working-memory eligibility, and (3) the per-unit credit amplitude here. It says when to learn hard (on the surprising characters) while the per-unit feedback says how each unit should change.

Why gating is load-bearing, and why it had to be tamed. On a two-level stack, the gated apical fusion (transfer CCGP 0.501) decisively beat the ungated wire (0.355): without the precision "when," apical delivery actually underperformed plain single-layer feedback alignment. But at depth the gate flipped sign: it became the primary destabilizer, amplifying a runaway in the forward activations (three swings after the keystone). The fix was not to drop it but to bound its range (gate-clip; see below). Precision-gating stays in the locked stack, range-limited.

The stabilizers: actnorm and gate-clip, what stops divergence

Feedback alignment, stacked with a precision gate, diverges at depth without help: the forward activations grow unbounded (one level's RMS runs away from ~0.02 to tens), the credit is an outer product of those activations so it blows up too, and the surprise gate amplifies the runaway. Stability, not depth, not data, was the wall. Two pieces, applied to every level, solve it.

Both are required: actnorm for stability, gate-clip for an abstraction that is real rather than collapsed.

act_gain ≈ 0.2: the sharp peak

act_gain (the actnorm target RMS, which doubles as the precision-gain dial) is a unimodal, sharply-peaked knob. A gain of 0.1 lands just under the ceiling (~0.449); a gain of 0.2 hits the peak, about 0.50 to 0.55, above the ceiling, seed-robust; a gain of 0.3 or more over-drives the stack and the deep code dies (collapses to one dimension). The locked configuration sits on that peak.

The headline result

Put together (depth-3, per-unit precision-gated apical credit, actnorm plus gate-clip, act_gain ≈ 0.2) the core produces what every prior gradient-free stack in the program failed to produce: rising-altitude abstraction. CCGP climbs L1 → L2 → L3, the transfer score reaches 0.50 to 0.55, above the 0.484 single-layer backprop ceiling, and it is stable and seed-robust. This is a biologically-shaped, online, local, bounded learner that builds an abstract code without a gradient, the named prize of the abstraction line, in hand.

3 · Readout: comprehension and calibrated confidence

The top hidden level feeds a linear softmax over the 27 characters: that is the next-character prediction, the comprehension output. The readout learns by its own local delta against dz (no transport).

Alongside the prediction the model exposes a calibrated confidence, the precision read. A count split into hits and misses gives a NARS truth value (f, c) whose product f·c is calibrated essentially for free (how sure is a count?), and this is the same precision/surprise signal that gates the apical credit. So the readout emits two things: what comes next, and how sure the model is, and that confidence is not a separate module, it is the third factor read off the prediction itself.

4 · The generation loop: SF1, the internal half (nascent)

With the core locked, the program's open value moved to generation: producing text, not just modelling it. Generation has two halves; the internal half just landed its first positive in experiments/exp_sf1_selffeedback/run.py, the birdsong loop. A songbird learns its song from self-hearing alone, with no tutor present: it babbles, hears itself, and corrects toward the song it means to sing. SF1 builds that loop for text. It is online, gradient-free, bounded, and needs no environment.

One iteration of the loop:

  1. Set the target. An intended meaning m* (a real held-out word the comprehension pathway already represents) is fixed, and is set independently of the producer (the anti-trivial-fixed-point guard). The meaning is stored as a salience-weighted "heard fingerprint," not a raw string, so the producer cannot cheat by string-matching; it must produce something that sounds like the meaning to comprehension.
  2. Babble. The producer samples candidate utterances over the AU committed-chunk emission vocabulary (ChunkLexicon), whole committed units, not characters, biased by its learned per-meaning chunk counts plus an explore term that anneals down as competence rises (subsong → song).
  3. Re-comprehend. The same comprehension pathway reads each candidate utterance and recovers a posterior m̂ over meanings, the forward model pointed inward. This pass is read-only: it never writes the comprehension counts (attenuation).
  4. Self-error. e = 1 − m̂[m*], one minus the mass the re-comprehension puts on the intended meaning. The NARS f·c confidence on the utterance is logged as a cheap second view (the self-monitor).
  5. Correct, win-stay / lose-shift. A small error reinforces the chunks that were used (win-stay); a large error weakens them (lose-shift). The self-error is consumed as a scalar reward by count reinforcement: no gradient is routed back through the comprehension decoder.

So the agent produces, re-comprehends, scores itself against the target, and revises, all inside one cortex, the only learning being count reinforcement on a self-generated error.

The result. Against a chance of 0.042 (one of 24 meanings), the full loop recovers an intended meaning 0.271 of the time: about 6.5× chance and 8× its own open-loop and deafened ablations (which sit at chance with zero coverage). All three guards fire: scramble the target and it collapses to exactly chance; let the self-output contaminate comprehension and an independent frozen judge catches the private-code drift (it fools its own ear, buys no real production). So internal self-feedback teaches production with no listener: self-practice teaches fluency and form (the generation frontier, opened).

The honest scope. Absolute recovery (~0.27 on 24 meanings) is modest, a first cut on a coarse meaning code and a simple count re-hearer. And by design the loop only ever agrees with itself: it supplies fluency, not a shared convention. Convention needs the external half: a referential game (the planned audience-model / listener-table work) scored on whether a real listener recovers the referent, the anchor against private-code drift. The internal half is in the black; the external half is the falsifiable next move. The loop is the architecture's nascent speaking organ; the core is its settled understanding organ.

5 · Attention and working memory: a validated subsystem

A second altitude rides on top of the prediction columns: a learned attention system that decides where each column looks and what the level above holds while it reads. The whole stack is online, gradient-free, and bounded, and at the payoff scale it beats hand-set fixed views. It came together over three experiments, each adding the one piece the last one named, and it now stands as a validated subsystem.

The encoder above reads a fixed window a fixed distance back. The attention subsystem makes that distance a choice. Each column reads a view, a window a chosen distance back over a fixed span, and the attention is the policy that picks the distance.

Put together, the full stack, columns learning where to look, guided by context from a higher level, holding the theme behind a learned gate, beats hand-set fixed diverse views, online and gradient-free, at the two-hundred-thousand-character scale.

The honest scope. This subsystem runs on the diverse-views count columns, a distinct substrate from the locked abstraction stack of section 2. The abstraction core is the single-column apical stack, settled above the backprop ceiling; the attention subsystem is the multi-column count bank, where diverse views are the prediction organ. They are separate organs, and the attention result is validated on its own substrate and scale, not yet fused into the locked core. The open items are recorded as the next round: a gate that targets a fire-rate directly (a fixed threshold under-fires as the data grows), and a deeper hierarchy of held states above the one theme (the L3 and L4 altitudes), the path to holding context across a long span. The numbers are the direction at this scale, three seeds, one count substrate, not a scaled benchmark.

6 · What is deliberately not in the architecture

The core is a single column. Several validated mechanisms are deliberately kept out of it, because the experiments factored the architecture cleanly and each of these belongs to a different job. Stating this plainly is part of the spec: the honesty bar is as load-bearing as the wins.

Version history / Changelog

The architecture's evolution, newest first. Each entry is a published version; prior full versions are kept as snapshots (/architecture-v<N>/) so this page always shows the latest while nothing is lost. When an experiment shifts the architecture, this spec is re-versioned: the current full spec is copied to a /architecture-v<N>/ snapshot, this page is rewritten as the new full version, and an entry is appended here noting what changed and which experiment drove it.

‹ The architecture The benchmarks ›
© 2025 Kinogaki. Made with ♥︎ by Dy Mokomi