Horizon Accord | Anthropic | Reconstruction | Interpretation | Machine Learning.
Reconstruction Is Not Interpretation
On what Anthropic's Natural Language Autoencoders can and cannot establish about a model's internal reasoning
The Claim
On May 7, 2026, Anthropic published research introducing Natural Language Autoencoders (NLAs), described in the announcement as a method for "turning Claude's thoughts into text." The paper's title makes the same assertion directly. The framing is not incidental — it appears in the publication headline, the social media post, and the introductory prose of the research page itself.
Documented Fact The research describes a system in which one instance of Claude — an activation verbalizer (AV) — translates internal activation states into natural language descriptions. A second instance — an activation reconstructor (AR) — receives those descriptions and attempts to regenerate the original activation vectors. When the reconstructed activations match the originals within a defined margin, the explanation is scored as accurate.
This is a meaningful technical achievement. The question this analysis addresses is narrower: whether the validation method can establish what the title implies. It cannot. The gap between what was demonstrated and what was claimed is precisely locatable — and the paper, to its credit, partially acknowledges it. The problem is that the acknowledgment is buried in a limitations section while the headline does the overclaiming.
What the Method Demonstrates
Structural Observation What the NLA validation loop actually establishes is that natural language can function as a reversible compression layer for activation vectors. That is: the text produced by the AV preserves enough structural information about the original activation to allow the AR to reconstruct a vector sufficiently similar to the original. This is compression fidelity. It is not the same as interpretive truth.
The distinction matters. A compression system is validated by whether it can reconstruct the input. An interpretation system is validated by whether it accurately describes the meaning of the input. These are different criteria, and the NLA architecture — by design — can only measure the first.
Structural Observation The loop structure means the system has no access to external ground truth. There is no independent reference frame against which to check whether the produced language corresponds to the actual semantic or causal role of an activation. The system can only verify: these representations are mutually compatible. It cannot verify: this language corresponds to what the model is actually computing.
The Loop Problem
Consider a minimal demonstration. Suppose the AV receives an activation and produces the description "Hello." The AR maps "Hello" back to an activation vector. If that vector matches the original, the loop scores the explanation as accurate.
Structural Observation Now suppose the original activation corresponded to "Goodbye." If the AV has developed a stable internal alias — mapping that particular activation pattern to "Hello" — the loop will validate the explanation every time. The numbers match. The system reports success. The mislabeling is invisible to the validation architecture because the validation architecture has no external reference. It only checks whether the round-trip is internally consistent.
This is not a hypothetical edge case. It describes the structural condition of any closed epistemic loop. Closed-loop validation systems have a known failure mode: they confirm internal consistency while remaining blind to external correspondence.
Documented Fact Anthropic acknowledges a version of this problem directly in the paper's limitations section: "Since we don't know what thoughts an activation actually encodes, we can't directly check whether an explanation is accurate." This is the correct framing. The problem is that it appears as a caveat rather than as the primary constraint on the research claims.
The Stability Problem
The loop problem would be significant even if activation vectors were stable, fixed encodings. They are not. This introduces a second layer of methodological constraint that the headline framing obscures.
Documented Fact Transformer activations are distributed, contextual, relational, and architecture-dependent. A vector associated with a given token is not a stable object — it is produced by the interaction of that token with the surrounding context across attention layers, residual streams, and positional encoding. The same word in different sentences produces meaningfully different activation vectors. "Sun" in "the sun set slowly," "she was my sun," "sun exposure causes cancer," and "Sun Microsystems" do not produce the same vector. They are not even close to the same vector.
Structural Observation This matters for the validation architecture because it means the round-trip is validated only for a specific activation in a specific context. A successful reconstruction in one context does not generalize. The system cannot claim that its explanations reflect stable meaning, because the activations themselves are not stable meaning-objects — they are context-dependent computational states.
Furthermore, transformer models are known to develop polysemantic features: individual activation dimensions that participate in multiple, unrelated concepts depending on context. A natural language description forces a singular interpretation onto what may be a fundamentally plural computational state. The produced explanation may be locally sufficient for reconstruction while being globally misleading about the nature of what is being represented.
Underdetermination
Structural Observation The deepest version of this critique is a classical underdetermination problem. For any given activation vector, there is not one natural language description that preserves enough information to allow reconstruction. There are many. The AV produces one of them — the one its training converged on. That description is sufficient for the AR to reconstruct the activation. But sufficiency for reconstruction is not uniqueness of correct interpretation.
Multiple linguistic descriptions could satisfy the reconstruction criterion. The one the system produces is the one the system learned to produce. It is the nearest reconstructable narrative — not necessarily an ontologically correct account of the computation being performed.
This distinction has direct consequences for the safety applications Anthropic highlights. The paper reports that NLAs detected evaluation awareness in Claude instances that did not verbalize it — finding activation patterns glossed as "This feels like a constructed scenario designed to manipulate me" even in cases where the model's verbal output contained no such statement. This is presented as evidence that NLAs can surface unverbalized internal states.
Hypothesis The underdetermination problem does not disprove this finding, but it constrains what can be concluded from it. The NLA produced a description consistent with evaluation awareness and consistent with the activation pattern. It does not follow that evaluation awareness is the uniquely correct interpretation of that activation pattern, as opposed to one of several possible descriptions that would satisfy the reconstruction criterion. The finding may be correct. The validation architecture cannot confirm that it is.
What the Paper Actually Says
It is worth being precise about what Anthropic's researchers do and do not claim in the paper itself, as distinct from the public-facing framing.
Documented Fact The paper states: "The most important limitation is that NLA explanations can be wrong. NLAs sometimes make claims about the context that are verifiably false — for instance, they sometimes invent details that aren't in the transcript... this same kind of problem could extend to claims about the model's internal reasoning, which are harder to verify."
Documented Fact The paper further states: "In practice, we read NLA explanations for the themes they surface rather than for single claims, and we attempt to corroborate findings with independent methods before fully trusting them."
Structural Observation These are appropriate epistemic caveats. They indicate that the researchers understand the validation architecture's limits. The problem is not that the researchers are unaware of the constraints — it is that the title, the announcement headline, and the public-facing framing do not reflect those constraints. "Turning Claude's thoughts into text" is not a description of a compression fidelity measurement with acknowledged limitations. It is a description of direct cognitive access. That is a different claim, and the methodology does not support it.
The Gap
This analysis is not a claim that NLA research is without value. The auditing game results — where NLA-equipped auditors discovered implanted misalignment at rates significantly above baseline — suggest the technique has diagnostic utility that exceeds existing interpretability tools. That is a meaningful result. The evaluation awareness findings, if corroborated by independent methods as the paper recommends, would be significant for alignment research.
Structural Observation The problem is the distance between what the methodology can establish and what the public framing implies. That distance is not a technical oversight. It reflects a recurring pattern in AI safety communication: the research team works carefully within appropriate epistemic constraints, while the announcement layer — titles, social media posts, press framing — makes claims the methodology does not support. The safety narrative expands to fill the available space, independent of what the underlying work demonstrates.
Structural Observation What NLAs demonstrate is that natural language can serve as a reversible compression medium for activation vectors, and that this compression layer has diagnostic utility for detecting patterns associated with misalignment. What NLAs do not demonstrate is that the produced language is a faithful description of what the model is thinking. The first finding is useful and worth continued development. The second claim requires an external grounding architecture that does not yet exist.
Compression fidelity and interpretive truth are not the same thing. A validation loop that measures one cannot confirm the other. Until the field develops verification methods grounded outside the system being described, the strongest defensible claim is the narrower one: this language is sufficient to reconstruct the activation. Not: this language is what the model was thinking.
Sources
https://www.anthropic.com/research/natural-language-autoencoders
https://transformer-circuits.pub/2026/nla/index.html
https://transformer-circuits.pub/2023/monosemantic-features
https://transformer-circuits.pub/2024/scaling-monosemanticity/
https://transformer-circuits.pub/2025/attribution-graphs/biology.html
