Test-time training: a pre-alpha experiment

The castle stays put. You only get a small bag of extra bricks.

I have been running a local, falsifiable stand-in for the shape Ilya Sutskever has been willing to say in public: experience arrives as a stream, the system has to change, later work should get easier, and the person should not be wiped. Safe Superintelligence still has no paper. The only technical window is the Dwarkesh interview of 25 November 2025 and a short correction on X: scaling the current thing “won’t stall. But something important will continue to be missing.” The missing object, in his words, is a continual learner — not a finished oracle — and a mid-trajectory signal he analogises to a value function. He has not named an architecture.

That vacuum filled quickly. On 12 August 2026, @iruletheworldmo posted that SSI’s product is test-time training: a subset of weights updated on the fly, “instead of a context window you get actual learning,” a small reasoner beating larger frozen runs. A follow-up called TTT “the third axis we are scaling” and pointed at Yu Sun’s TTT-E2E paper as further reading. That paper is Stanford / Astera / academic NVIDIA. Its authors are not SSI staff. Treating the rumour as a recipe is how you spend a week proving the wrong lab’s homework.

I did not try to guess SSI’s stack. I asked a narrower question a 3090 can kill: if you freeze a small language model, keep a 1,024-token attention window, and write a recoverable bag of fast weights D into three MLP down_proj layers as a long prefix streams by, does that bag remember this stream after the window has evicted it?

The recipe is In-Place-style, not a clone of TTT-E2E. The frozen core is Qwen3-0.6B-Base. The bag is a dense delta on layers 12, 18 and 24. An outer rule φ (width-5 conv + projection) is meta-trained on PG19, then frozen. At evaluation we walk a 4,096-token prefix, crop attention to the last 1,024, and score next-token NLL on the last 512 — tokens that never write D. The number is delta_nll = NLL_with_D − NLL_frozen. More negative means the extra bricks made the last page easier.

§What the private ledger already held

We wrote this campaign down as we went, on purpose: a living experiment tree, a results file that only quotes numbers from run logs, a fuckups file, and a canvas that kept getting rewritten. The public version of that ledger is this note. The useful process facts, compressed:

A TinyLM fixture printed delta_nll = 0 after real CUDA trains. That object was killed. We ported the same recipe to Qwen anyway, rather than scale a dead toy.

The paper-scale inner step η = 3.0 slammed a Frobenius cap on almost every write (~99.6% clip). Those runs were gated as invalid, not as science. A search mutator eventually found the first legal keep by shrinking η by three million, to 3×10⁻⁶, and opening the step cap to 10⁻³. Champion commit 3654e3c: mean delta_nll = −0.271126 on eight PG19 val-4k last-512 slices (nll_ttt = 4.411367, nll_frozen = 4.682494, clip 4.2%). That number reprinted exactly when we rebuilt D from scratch for the controls.

A GSM8K gold-NLL sentinel on 100 test items, after walking one 4,096-token prefix, moved by +0.00379 against a 0.10 band. Reset recovered the empty-bag score exactly. That is a short-prefix homework check, not the 128k stream in the design doc, and we did not run exact match.

We also learned a process lesson the hard way: after we intended to switch the search mutator, the previous paid provider stayed preferred in the live path and kept getting called. The fix was to remove that path entirely, not to leave an opt-in.

None of that yet answered the question Ilya actually posed. A last-512 win on Project Gutenberg prose can be “I remember this walk in the woods,” or it can be “I got good at old-book brown bricks.”

§The two controls

B3, dump the bag. Walk the prefix, then D.reset(), then score the same last 512. If the gain survives, something other than D leaked — a longer window, a stuck hook, a mutated base weight.

B2, swap bags. Walk book A, take book B’s exam wearing D_A. If the wrong bag still helps, D is a style adapter. This is the control the TinyLM branch never lived long enough to run.

Both used the champion φ, discarded leftover training D, and rebuilt the bag with the champion η and cap. We did not call the trainer’s compute_delta_nll helper: that path defaults to a 10⁻⁴ cap and only does matched-versus-frozen on the same book.

Reset Δ was 0.0 on every book. Frozen NLL and reset NLL were the same float, 4.682493507862091. B3 keeps. The extra bricks were really doing the helping.

Shuffle mean Δ was −0.252954. That is 93% of the matched gain. Every circular pair was still ≤ −0.18. On val books 23 and 36 the wrong bag was slightly better than the right one.

B2 is killed. The −0.271 is real, resettable D. It is not this-book memory. It is “I have been reading this kind of prose.”

§What this is not

It is not evidence that SSI uses TTT. Success here would not have been that either. Ilya named a problem — stream, change, later easier, don’t wipe the person — and a mid-trajectory value signal he has not let us implement. @iruletheworldmo named a product and pointed at a public paper from another lab. Those are different objects.

It is not a 128k NonceFacts run, not eight-shot GSM8K exact match, and not the critic / “how do I feel about this game” piece. We locked that on purpose. A style adapter does not get a value head.

It is also not a claim that test-time training is empty. A bag that writes, resets, and transfers across books is doing something. The cheap explanation is domain adaptation into a 0.6B that was not already a Gutenberg specialist. The expensive explanation — instance-specific memory of evicted tokens — failed the swap test.

The control I would want before believing a TTT last-N win is the same one: put the bag on a different document. If it still helps, you have not built a deployment-time learner. You have built a small LoRA that likes the house style.

Numbers from prototype/results/b2-b3-pg19-controls.json. The living tree is ROADMAP-EXPERIMENT-GRAPH-TREE.md in the same repo. Next open node is whether a 4k prefix is even better than 2k on the same eight books, which is a length check, not a resurrection of the this-stream claim.