Looped Transformer:
Decoupling compute from parameters,
until it hits the chain-of-thought monitoring red line

A standard Transformer "stacks layers"—it stacks N layers with different parameters; the deeper the model, the more parameters, locking the two together. A Looped Transformer "loops"—it runs a small set of shared parameters through N passes, completely decoupling "how much compute to spend" from "how many parameters to store." To make the model compute deeper, you don't need to store another copy of weights; you just loop a few more times.

This idea didn't come out of nowhere. It traces back to Google's 2019 Universal Transformer (Dehghani et al.)—which replaced fixed depth with looping, allowing the model to adjust computation based on input difficulty—and the ALBERT paper's cross-layer weight sharing (which used shared parameters to slash BERT's storage footprint by an order of magnitude). But at the time, both efforts were more like efficiency-optimization side branches; no one treated them as a mainline architecture worth serious investment.

Parameter count determines how much knowledge the model can memorize; loop count determines how much compute the model is willing to spend on a question—
for the first time, these two things can be paid for separately.

What really nailed the Looped Transformer was the 2023 foundational paper "Looped Transformers as Programmable Computers" (Giannou et al., ICML 2023). The authors didn't train a model; instead, they hand-constructed a set of weights that lets a 13-layer Transformer with repeated looping simulate a real computer—implementing memory reads/writes, conditional branches, and function calls, proving the mechanism Turing-complete.

The specific approach splits the input sequence into three parts: scratchpad, memory, and instructions. The Transformer's output after one round is fed back entirely as input, serving as the starting point for the next loop—this feedback arrow is the core of the entire mechanism. The key conclusion: how complex a program is only shows up in "how many times it loops"; the model's own parameter count doesn't need to increase at all.

This paper itself isn't a usable product; it's more like a mathematical existence proof—first pinning down "what Looped Transformers can theoretically achieve," giving the engineering papers of the next two years a ceiling to benchmark against.

The most important step from "mathematical proof" to "trainable architecture" in these two years was "Mixture-of-Recursions" (KAIST / Mila / Google DeepMind joint team, NeurIPS 2025). It solved a problem that previous Looped Transformers hadn't seriously addressed: early schemes assigned the same loop count to every token, but function words like "the" or "of" and a complex mathematical proof require vastly different amounts of computation.

Mixture-of-Recursions' approach is to assign loop depth individually per token: simple tokens exit the loop early, hard tokens stay in and compute more rounds. The paper's visualization is intuitive—dark cells indicate "this token was computed at this layer," light cells indicate "skipped"; within the same sentence, common function words exit early, while tokens requiring reasoning stay in the loop.

This paper is the foundation for nearly all subsequent 2025–2026 Looped Transformer papers—it transforms "looping" from a coarse-grained switch applied uniformly across the entire sequence into a resource allocation problem that can be finely tuned per token, opening up space for the next step of engineering optimization.

From "stacking layers" to "looping," the Looped Transformer's one plain selling point is decoupling compute from parameter count—for the same parameter budget, you get deeper effective computation; for the same compute budget, you get a smaller storage footprint. In scenarios where VRAM and deployment costs are hard constraints, this is a tangible engineering benefit.

But the three "unsolved" items in the §04 table are also real: there exists an optimal loop count, past which returns regress (LoopCoder-v2's loop-2 optimum); when loop count gets high, training itself can collapse due to gradient oscillation and residual explosion (Fully Looped Transformer had to specifically fix this); and the latency and VRAM overhead introduced by looping require dedicated engineering solutions like Parallel Loop Transformer to counteract. IQuest-Coder-V1-40B-Loop's benchmark results also serve as a reminder: looped architecture is not a universal win—generation tasks (HumanEval) improve, but long-horizon agentic tasks like SWE-Multi and CodeArena regress, and the team itself positions it as an "experimental prototype."

This parameter-saving technique soon also answered a more sensitive question: where does AI actually "think"—in readable chain-of-thought text, or inside unreadable hidden states? On September 1, The Information, citing anonymous sources, reported that OpenAI's next-generation model Astra would use an inference technique called "recurrent depth"—the same query is processed repeatedly inside the model. Critics call this approach "opaque recurrence" because it leaves fewer readable traces than a conventional chain of thought (The Information's original article is behind a paywall and could not be directly verified; this is relayed from safety researchers' public responses).

OpenAI's response left room: the report explicitly stated that Astra's use of this technique "appears to be limited," chain-of-thought is expected to remain readable, and the company rebutted the notion that it would "shift to neuralese"; chief scientist Jakub Pachocki wrote on X that "since our first-generation reasoning model, OpenAI has been working to preserve and leverage chain-of-thought monitoring... this is a core objective of our current research plan"—this statement reads more like a reaffirmation of stance than a direct answer to the specific question of "whether Astra actually uses recurrent depth."

Independent researcher Sebastian Raschka wrote a calmer technical dissection the next day, with a rather pointed observation—he noted that "reusing the same layer" itself doesn't automatically suppress visible chain-of-thought; "it just adds some extra computation to the hidden state before generating the next token, which is exactly what standard Transformer layers do." His more cautious interpretation: if a model loops more times, it may indeed need to generate fewer intermediate reasoning tokens, shifting more computation into unreadable hidden activations—but this effect is essentially the same as just making the model bigger, and isn't necessarily a problem unique to the "looping" mechanism. In other words, part of this panic may be built on a misunderstanding of the Looped Transformer mechanism itself. More notably, The Information's follow-up report stated that Anthropic and Google DeepMind are already discussing this technology—if true, this isn't just OpenAI's choice, but the entire industry probing in the same direction.

This debate didn't stay at the speculation stage. On September 3, GPT-6 Astra was officially released, and the two things that previously could only be guessed from anonymous sources and a non-denial, OpenAI now fully admitted. The official blog positions it as "the world's smartest, most aligned model": FrontierMath Tier 4 at 98%, ARC-AGI-3 at 99.9%, ExploitBench at a perfect score, with president Greg Brockman calling it a "generational leap." More crucially, on the safety half—official wording confirms Astra is the first model to cross the Preparedness Framework "Critical" cybersecurity threshold; simultaneously, the company directly admitted for the first time the most sensitive charge from the "recurrent depth" controversy—"Our evaluations find that Astra's written reasoning is harder to monitor than GPT-5.6 Sol", because "Astra has stronger control over written reasoning for simple tasks, able to solve problems with fewer written steps." The company calls this a problem they are "taking seriously," but also emphasizes that "on complex tasks, it still seems difficult to fully hide the required reasoning process." Chief scientist Jakub Pachocki's response this time was much more specific than "reaffirming monitoring as a core objective"—he offered a verifiable technical judgment: the compute graph depth of current frontier systems including Astra is still around twice that of GPT-4, with no sign of the "race toward unmonitorability" that outsiders feared.

The other, longer-circulating online claim—"Claude Mythos has long been using Looped Transformers"—when traced to its root, turns out to be just fan speculation based on benchmark tables, completely different in evidentiary weight from the Astra evidence in §07, and shouldn't be conflated.

Claude Mythos itself is a real product—on April 7, 2026, Anthropic officially confirmed a restricted preview (the "Project Glasswing" project, open only to partners in cybersecurity / biosecurity), which has now iterated to Mythos 5.1. But searching Anthropic's own Frontier Red Team blog post and the full product page yields not a single sentence about architecture—only usability, pricing, safety guardrails, and benchmarks. The source of the "Looped Transformer" claim was informal speculation on X based on Mythos's benchmark patterns, later turned into a PyTorch reproduction project "OpenMythos" by developer Kye Gomez (now 14.9k stars on GitHub), but the project's own README states plainly: "This is an independent, community-driven theoretical reconstruction, based entirely on publicly available research and speculation, with no affiliation, endorsement, or subordinate relationship with Anthropic or any of its proprietary systems."

Also worth correcting is a commonly reversed timeline: the University of Tokyo paper (Xu & Sato, "Should You Chain-of-Thought or Loop?", accepted at ICML 2026) that's often mentioned alongside this rumor was actually posted on arXiv in May 2025—about 10 months before the Mythos leak (March 26, 2026). The paper didn't come about because of the Mythos rumor; rather, the people later weaving this theory retroactively cited an already-existing paper. The causality is reversed.

On September 1—the same day as the Astra rumor—Anthropic simultaneously released Fable 5.1 and Mythos 5.1; the official announcement was unambiguous: "Claude Fable 5.1 and Claude Mythos 5.1 are the same model, just with different guardrail levels"; the benchmark gap is explained as "the task scope of early, lower-precision cybersecurity guardrail intervention, and with this guardrail upgrade, we expect the gap between the two models to significantly narrow." This addresses a different question, with no direct relation to "whether recurrent depth architecture is used"—but it suffices to show that the question of "whether Fable and Mythos are actually two different models," which previously could only be guessed from benchmark tables, received a blunt official answer: they were never two models, just the same weights with different levels of safety interception. This release also quietly tightened a safety clause— official documentation confirms, Fable 5.1 added new validation to the Messages API: "thinking blocks" must now exactly match the system prompt, tools, and history messages that produced them, or an error is thrown—blocking precisely a technical pathway that could be used for scaled distillation.

The Mythos "looped architecture" rumor and the Astra "recurrent depth" report are not in the same league of starting evidence—but the Astra thread has this week completed the full journey from rumor to official confirmation, while the Mythos thread remains stuck in place.

The real weight of this debate comes from an earlier document. In July 2025, led by Tomek Korbak, 41 researchers from OpenAI, Google DeepMind, Anthropic, the UK AI Safety Institute, METR, Apollo Research, and Redwood Research co-signed a position paper "Chain-of-Thought Monitorability: A New and Fragile AI Safety Opportunity", warning in advance that if models put more "reasoning" into continuous hidden states computed in loops, rather than generating human-readable chain-of-thought text, safety researchers' ability to observe what a model "intends to do" would fail.

Today's reasoning models at least still write intermediate steps as readable text (even if imperfectly); the Looped Transformer structure is precisely the opposite—computing more cheaply and potentially more powerfully, but with the process completely invisible. The surface of the Astra controversy is "whether OpenAI is using some architecture," but the real bet is whether what that warning described is shifting from "theoretical concern" to "happening right now."

This architectural debate has reached a阶段性 result: Looped Transformer is no longer just a parameter-saving engineering trick—it has been confirmed by OpenAI itself as standing on the AI safety red line of "whether the model's thinking process remains visible." What deserves scrutiny next is no longer "whether Astra uses recurrent depth," but "whether monitoring capability can be recovered."