Phrases that rhyme share their counts

Negative. 2026-06-26 · pooling works, the addressing blurs · experiment AP

Permutation-bound phrase addresses let similar phrases pool their counts (beating a floored literal on 67% of never-seen-in-form phrase probes) while keeping word order (×2.29 degradation under scramble, where a bag is ×1.00). But FlyHash crosstalk loses the tail: aggregate perplexity 1206 vs the literal's 831, and exact recall 423 vs 2.2. The fix is to use it as a backoff layer under the literal table, not a replacement.

The question

The phrase level suffers from sparsity. A literal n-gram counts continuations at the exact string "a b c," so a phrase seen once has a near-zero count even when dozens of similar phrases ("the b c," "a b d") were common. Each literal phrase is its own island; the counts never pool. The VSA fix is to give similar phrases overlapping addresses so their counts pool, while still keeping order, so "abc" ≠ "cab" and the thing is a sequence, not a bag. We wanted to know whether a count model can buy that generalization without losing what makes a literal n-gram good.

What we tried

Each word gets a fixed random ±1 atom hypervector. Encode the n-gram as one order-preserving address by binding shifted atoms (addr = ρ²(A) ⊛ ρ(B) ⊛ C) so position rides inside the vector and binding is similarity-distributive: phrases that share most shifted atoms get close addresses. Then FlyHash (Dasgupta's fly-olfactory circuit): a sparse expansive random projection blows the address up to 4000 buckets and a top-16 winner-take-all keeps the strongest, lighting up overlapping bucket sets for nearby inputs. We count the next token at every active bucket; counting at shared buckets is exactly count-pooling across similar phrases, no backprop, no factorization.

Text8, 2.39M words, top-10,000 get an atom. We held 20% of distinct context-phrase strings out of training, so the held phrases were never seen in this exact form: the literal must floor them. Three predictors on the same one-pass training: literal phrase-count, order-blind bag-of-context, and perm+FlyHash.

What happened

The mixed truth, stated plainly. Per probe, the VSA model beats the floored literal two times out of three, and beats the order-blind bag on every axis. The pooling is real: a phrase with zero literal count lands on buckets that common similar phrases filled, and inherits their continuation. And it stays a sequence: scramble the context word order and perm+FlyHash degrades ×2.29 while the bag is exactly invariant (×1.00). The ρ-shifted binding genuinely encodes order, and FlyHash crosstalk did not wash it out.

But on aggregate perplexity it does not beat literal (1206 vs 831), and on seen phrases it is a poor exact-memory (423 vs 2.2). The literal's unigram floor is a well-tuned fallback that never spikes; FlyHash crosstalk is the opposite: on the third of probes where the wrong similar phrases dominate the shared buckets, it puts low mass on the true token, and those spikes drag the geometric-mean perplexity up. The model wins the majority vote and loses the tail. The same crosstalk that pools rare phrases also blurs common ones. FlyHash is a generalizer, not a store.

The lesson

Similar phrases can share their counts without becoming a bag: pooling is real and order survives, the niche the construction was built for. But the addressing that pools the rare phrases also blurs the common ones, and on aggregate perplexity that loses the tail. The shape of the fix: use perm+FlyHash as a backoff layer, not a replacement: literal counts when the exact phrase has evidence, fall to the FlyHash address only when the literal count is zero. That keeps the 2.2 ppl on seen phrases and the 67% generalization on unseen ones.

Lineage

Grew from finding phrases the way you'd guess them and grammar is just counting, made productive, the phrase level and its sparsity problem, and from the VSA mining of reading structure back out of a sum.

Thread: representations, the right combiner. The ideas are Kanerva's hyperdimensional computing and Dasgupta, Stevens & Navlakha's FlyHash.