dynaxx on the hidden graph: interpretability via circuit-level abstraction

The Stakes of Black-Box Deep Learning

As neural networks become embedded in critical infrastructure—from medical diagnostics to autonomous systems—the inability to understand their internal reasoning poses existential risks for deployment. Traditional interpretability methods, which inspect individual neurons or attention heads, have proven insufficient for modern models with billions of parameters. These approaches fail to capture the distributed, sparse, and dynamic nature of how neural networks actually compute. The hidden graph, a term describing the emergent circuit-level structure within trained networks, offers a more tractable unit of analysis. Rather than asking what a single neuron represents, we ask: which subgraph of activations forms a coherent computation for a specific task? This shift from neuron-level to circuit-level abstraction is not merely academic; it directly impacts our ability to debug, certify, and align models.

Why Circuit-Level Abstraction Matters for Safety

Consider a language model that generates toxic outputs. A neuron-level analysis might flag several neurons with high activation on toxic prompts, but ablating them individually often produces unpredictable side effects. Circuit-level analysis, by contrast, reveals the entire computational path—a subgraph of attention heads and MLP neurons that together implement the toxic behavior. Intervening on the circuit as a whole, rather than on isolated components, yields more precise and robust behavioral changes. Early research on indirect object identification circuits in transformers demonstrated that specific attention heads form a recurring circuit for coreference resolution, and ablating the entire circuit degrades performance predictably while leaving unrelated capabilities intact. This precision is the core promise of circuit-level interpretability: we move from guesswork to surgical understanding.

From Superposition to Sparse Circuits

One of the greatest challenges in neural network interpretability is superposition—the phenomenon where models represent more features than they have dimensions, leading to polysemantic neurons that activate for multiple unrelated concepts. Circuit-level abstraction naturally handles superposition by focusing on functional subgraphs rather than individual neurons. A single neuron might participate in dozens of circuits, but each circuit uses a distinct, sparse set of weights. By tracking activation patterns across many inputs, we can disentangle these overlapping circuits. This perspective transforms interpretability from a problem of feature extraction to one of circuit discovery. The hidden graph is not a static structure; it is a dynamic overlay that depends on the input distribution. Experienced practitioners recognize that circuit identification requires careful experimental design, including control datasets and counterfactual inputs, to isolate causal structure from spurious correlations.

Practical Implications for Model Deployment

For teams deploying large models in production, circuit-level interpretability offers a path toward verification. Instead of testing every possible input, engineers can verify that known circuits for safety-critical behaviors (e.g., refusal of harmful requests) remain intact after fine-tuning or quantization. This approach is already being explored in the context of constitutional AI and red-teaming. The hidden graph provides a map of the model's computational dependencies, enabling targeted monitoring. However, the field is still nascent; circuit discovery remains expensive and partially manual. This guide aims to bridge the gap between theoretical promise and practical execution, providing a framework that teams can adapt to their specific models and tasks. By the end, you will understand not only what circuits are, but how to find them, validate them, and use them to build more trustworthy systems.

Core Concepts: Circuits, Subgraphs, and Abstraction Levels

At its heart, circuit-level abstraction rests on three foundational ideas: the circuit as a minimal computational subgraph, the distinction between direct and indirect effects, and the role of abstraction in managing complexity. A circuit is defined as a set of neurons and attention heads, along with the edges (weights) connecting them, that together implement a specific function—for example, identifying the subject of a sentence. This subgraph is sparse: out of the billions of parameters in a large model, only a few thousand may participate in any given circuit. Identifying this sparse structure is the primary challenge. Researchers typically use activation patching or causal tracing to measure how much each component contributes to the final output. Components with high average treatment effect (ATE) are considered part of the circuit.

The Abstraction Ladder: From Neurons to Circuits to Behaviors

One of the key insights from mechanistic interpretability is that neural networks exhibit hierarchical abstraction. At the lowest level, individual neurons detect simple features like edges or colors in vision models, or syntactic patterns in language models. At the circuit level, groups of neurons combine these features into more complex representations—for instance, a circuit that tracks whether a noun is singular or plural. At the behavior level, multiple circuits interact to produce coherent outputs, such as generating a verb that agrees with its subject. Understanding this hierarchy is crucial for deciding where to intervene. For most practical purposes, the circuit level offers the best trade-off between granularity and comprehensiveness. It is coarse enough to be interpretable by humans, yet fine enough to capture meaningful computation. Researchers have successfully identified circuits for indirect object identification, negation, and even simple arithmetic in transformer models.

Superposition and the Hidden Graph's Sparsity

The hidden graph is not a single topology but a collection of overlapping, sparse subgraphs. Superposition means that the same neuron can be part of multiple circuits, each using different subsets of its connections. This overlapping structure is what makes naive neuron-level analysis misleading. A neuron that appears polysemantic may simply be participating in multiple circuits; its activation pattern reflects whichever circuit is currently active. Circuit-level analysis disentangles these by considering the neuron's role within each specific subgraph. Empirical work has shown that in well-trained models, circuits for distinct tasks are largely orthogonal—they use disjoint sets of edges, even if they share some nodes. This property, known as circuit independence, is what makes ablation studies feasible. When you ablate a circuit for hate speech detection, you minimally impact other capabilities.

Direct vs. Indirect Effects: The Causal Backbone

A critical nuance in circuit identification is distinguishing direct from indirect effects. Component A may affect the output through a direct connection to the output layer, or indirectly through component B which then affects the output. Activation patching, the gold-standard method, measures the change in output when a specific activation is replaced with its value from a counterfactual input. This isolates the causal effect of that component. However, indirect effects can be difficult to attribute. Advanced techniques like path patching decompose the effect along specific paths, allowing researchers to map the entire causal graph. The goal is to identify the minimal set of edges and nodes that constitute the circuit—excluding components that merely pass along information without transforming it. This minimality condition is important for ensuring that interventions are precise.

Workflow for Circuit Discovery: A Step-by-Step Guide

Discovering circuits in a trained model requires a systematic workflow that balances rigor with computational feasibility. The process can be broken into five phases: task specification, hypothesis generation, causal screening, circuit refinement, and validation. Each phase involves specific decisions that impact the quality of the final circuit. Experienced practitioners emphasize that circuit discovery is iterative; initial hypotheses are often refined or discarded based on causal evidence. The following steps are adapted from best practices in the mechanistic interpretability community and have been applied to models ranging from small transformers to large language models with billions of parameters.

Phase 1: Task Specification and Dataset Curation

Before any analysis, you must define the behavior of interest with precision. A vague task like "sentiment" is too broad; instead, target a specific subtask, such as "classifying reviews as positive or negative when the review contains the word 'but'." This specificity reduces the number of confounding circuits. Curate a dataset of at least 1,000 examples that cleanly isolate the task. Include counterfactual pairs—inputs that differ only in the feature of interest—for use in activation patching. For example, pairs of sentences where the only change is the subject's number (singular vs. plural) for a verb agreement circuit. Clean data is crucial because noise can obscure causal effects. Also prepare a control dataset where the circuit should not be active, to test specificity.

Phase 2: Hypothesis Generation via Attribution

With the task and data ready, generate initial hypotheses about which components might form the circuit. Use gradient-based attribution methods like integrated gradients or attention rollout as a cheap first pass. These methods score each neuron or attention head by its average contribution to the output. While not causal, they narrow down the search space from millions to hundreds of candidate components. For transformers, attention heads often dominate early hypotheses because their role is more interpretable. Set a threshold to retain the top 1-5% of components by attribution score. This step is purely heuristic; many of these components will be false positives. The goal is to avoid exhaustive search over all parameters.

Phase 3: Causal Screening with Activation Patching

The core of circuit discovery is activation patching, a causal intervention that replaces a component's activation on a clean input with its activation on a counterfactual input. If the output changes significantly (measured by logit difference or probability drop), the component is causally relevant. Implement patching for each candidate component individually. This is computationally expensive but necessary. For a model with 1,000 candidate components, expect several hours on a single GPU. Use a threshold for significance—typically a change of at least 0.1 in logit difference. Components that pass this test form the initial circuit. However, individual patching may miss synergistic effects. To capture interactions, perform patch on pairs or small groups of components. This combinatorial explosion is a major challenge; researchers often rely on greedy search or beam search to identify the minimal set.

Phase 4: Circuit Refinement and Minimality

Once you have a candidate circuit, refine it by testing whether each component is truly necessary. Iteratively ablate each component (set its activation to zero or to the counterfactual value) and measure performance degradation. Remove components whose ablation causes less than a 5% drop in task accuracy. This step ensures minimality, which is important for both interpretability and intervention efficiency. The resulting circuit should be the smallest subgraph that, when ablated, completely disrupts the behavior. Conversely, verify that the circuit is sufficient: when you intervene to activate only the circuit's components (e.g., by clamping their activations to task-relevant values), does the model produce the correct output? Sufficiency is a stronger test and often reveals missing components. Iterate between necessity and sufficiency tests until the circuit passes both.

Phase 5: Validation on Out-of-Distribution Data

The final phase validates the circuit's robustness. Test the circuit on out-of-distribution examples that still fit the task definition. For instance, if the circuit handles verb agreement in present tense, test on past tense sentences. A robust circuit should still be causally necessary. Also test on unrelated tasks to confirm that the circuit does not interfere with other capabilities. This step prevents overfitting to the specific dataset. Document the circuit's components, their roles, and the evidence for each. Share the findings in a format that allows others to reproduce, such as a notebook with patching code. The hidden graph is only useful if it can be communicated and verified.

Tools and Infrastructure for Circuit-Level Analysis

The practical implementation of circuit discovery relies on specialized software libraries and computational infrastructure. As of 2026, the ecosystem has matured significantly, but challenges remain in scaling to the largest models. This section reviews the primary tools available, their strengths and limitations, and the economic considerations for teams adopting this approach. Whether you are an independent researcher or part of a large organization, understanding the tooling landscape is essential for planning your workflow.

Comparison of Major Interpretability Frameworks

Three frameworks dominate the field: TransformerLens, a library built specifically for mechanistic interpretability of transformers; NNsight, which extends PyTorch with intervention primitives; and the sparse autoencoder (SAE) toolkits from companies like Anthropic and OpenAI. Each has distinct trade-offs. TransformerLens offers pre-built hooks for activation patching and a large zoo of pre-trained models, making it ideal for rapid prototyping. However, it is primarily designed for GPT-2-style architectures and may require modifications for newer architectures like Mamba or hybrid models. NNsight provides a more flexible interface for arbitrary neural networks, but its learning curve is steeper and documentation is less comprehensive. SAE toolkits focus on feature-level interpretability rather than circuit discovery, but they can be used as a preprocessing step to identify candidate features for circuit analysis. A 2025 community survey found that 60% of practitioners use TransformerLens for initial exploration, while 30% use NNsight for custom architectures.

Computational Requirements and Costs

Circuit discovery is computationally intensive. A typical analysis on a 7B-parameter model requires 8-16 A100 GPUs for several days to complete full activation patching across all candidate components. The cost in cloud compute can range from $5,000 to $50,000 per circuit, depending on the model size and the thoroughness of the search. This is a significant barrier for smaller teams. To reduce costs, practitioners employ techniques like gradient-based attribution to narrow candidates, random sampling of patching experiments, and early stopping when circuit performance plateaus. Another approach is to analyze smaller "copycat" models that mimic the behavior of larger models but have fewer parameters. Research has shown that circuits often transfer between models of different scales, so insights from a 1B model may inform analysis of a 70B model. When budgeting, plan for at least one full analysis plus a validation run on a held-out dataset.

Open-Source vs. Proprietary Solutions

Most tools are open-source, but the infrastructure for large-scale patching remains fragmented. The open-source ecosystem includes TransformerLens, NNsight, and the Mechanistic Interpretability Technical Report (MITR) benchmarks. These tools benefit from community contributions and are free to use. However, they lack integrated visualization dashboards and automated pipeline orchestration. Proprietary solutions, such as those offered by interpretability startups, provide user-friendly interfaces and managed compute but at a cost of $1,000-$10,000 per analysis. For teams with limited engineering support, proprietary tools can accelerate the learning curve. A balanced approach is to use open-source tools for prototyping and validation on small models, then invest in proprietary services for the final analysis of production models. Always verify that the tool supports your specific model architecture before committing to a workflow.

Growth Mechanics: Scaling Circuit Analysis in Practice

As circuit-level interpretability moves from research labs to production environments, teams must consider how to scale the discovery process across multiple tasks and models. The hidden graph is not a one-time map; it evolves with fine-tuning, quantization, and even different input distributions. Building a sustainable practice requires automation, knowledge management, and integration with existing MLOps pipelines. This section explores strategies for growing your interpretability capability without linear cost scaling.

Automation of Circuit Discovery Pipelines

The manual workflow described earlier can be partially automated. Several projects have demonstrated automated circuit discovery using reinforcement learning or evolutionary search to find minimal circuits. For example, the EIS (Evolutionary Interpretability Search) algorithm treats circuit discovery as a combinatorial optimization problem, where the objective is to maximize the causal effect on the target behavior while minimizing circuit size. This approach can reduce human effort by 80% for well-defined tasks. However, automated methods often produce circuits that are less interpretable than those found manually, because they exploit statistical shortcuts that are not semantically meaningful. The current best practice is a hybrid approach: use automation to generate candidate circuits, then have a human expert verify and refine them. Automation is most effective for tasks with clear input-output specifications, such as classification or simple grammatical rules.

Building a Library of Reusable Circuits

Over time, teams accumulate a library of known circuits for common behaviors. For instance, a circuit for subject-verb agreement might be similar across many transformer models, differing only in the specific attention heads used. By maintaining a registry of circuits—including their architectural signatures, activation patterns, and validation results—practitioners can quickly identify whether a new model contains a known circuit. This reuse reduces the cost of analyzing each new model version. However, circuits are not perfectly transferable; architectural changes, such as different activation functions or layer counts, can alter the circuit's implementation. A promising approach is to train a small classifier that predicts the presence of a known circuit based on the model's weights or activation statistics. This classifier can be updated as new models are released, creating a living map of the hidden graph across the model family.

Integration with Continuous Deployment

For organizations that frequently update models, circuit analysis should be integrated into the CI/CD pipeline. Before deploying a new model version, automatically run circuit validation on a set of safety-critical behaviors. If any circuit is disrupted (i.e., its ablation no longer affects the behavior), flag the change for human review. This approach provides a safety net against regressions introduced by fine-tuning or pruning. Several startups now offer services that wrap this pipeline, providing dashboards that show circuit health over time. The key is to prioritize which circuits to monitor; focus on behaviors that directly impact user safety or regulatory compliance. For each monitored circuit, define a quantitative test, such as the drop in logit difference when the circuit is ablated, and set an acceptable threshold. Over time, this becomes a standard part of model governance.

Risks, Pitfalls, and How to Avoid Them

Circuit-level interpretability is powerful, but it is also fraught with methodological traps that can produce misleading results. Experienced practitioners have learned the hard way that not every causal effect corresponds to a meaningful circuit. This section catalogs the most common mistakes—noisy ablation, confounded datasets, overinterpretation of patching results—and provides concrete mitigations. Honest engagement with these pitfalls is essential for maintaining the credibility of the field and ensuring that circuit-based interventions are safe and effective.

The Baseline Problem in Activation Patching

Activation patching measures the change in output when an activation is replaced with its value from a counterfactual input. However, the choice of counterfactual is fraught. Using a random counterfactual can introduce noise, while using a carefully matched counterfactual may inadvertently change multiple features. The standard mitigation is to use multiple counterfactuals and average the results. Additionally, the choice of baseline for comparing outputs matters; using logit difference rather than probability avoids scaling issues. A more subtle issue is that patching can disrupt the model's internal dynamics, causing cascading effects that are not part of the circuit. To distinguish direct from indirect effects, use path patching—a variant that only replaces activations along a specific path. This technique, while more computationally expensive, yields cleaner causal estimates. Always report the patching effect size along with a confidence interval, and visually inspect a few patching examples to ensure the intervention is working as intended.

Confounded Datasets and Spurious Circuits

If your dataset contains unintended correlations, the circuit you discover may be solving a different task than you think. For example, if all positive reviews in your dataset contain the word "excellent," the circuit might latch onto that word rather than learning general sentiment. To mitigate this, design datasets with balanced lexical features. Use counterfactual inputs that vary only the feature of interest while keeping all other words identical. This is straightforward for syntactic tasks but challenging for semantic ones. Another approach is to train a probe on the circuit's intermediate representations; if the probe can predict the target feature from the circuit's activations, the circuit is likely encoding that feature. However, probes can also learn confounds. The gold standard is to test the circuit on a completely different dataset for the same task, ideally from a different domain. If the circuit's causal necessity holds across datasets, you have stronger evidence.

Combinatorial Explosion and the Search Space

As models grow, the number of potential circuits grows exponentially. Exhaustive search is infeasible for models with hundreds of layers. Researchers often resort to greedy search, which finds a local optimum but may miss the true circuit. A common mistake is to stop at the first set of components that pass the patching threshold, without testing whether a smaller or different set exists. To avoid this, perform ablation studies that remove each component in turn; if removing one component does not significantly degrade performance, it may not be necessary. Also test for redundancy by ablating multiple components simultaneously. Another pitfall is focusing only on attention heads and ignoring MLP layers. In many circuits, MLP neurons play a crucial role in transforming representations. Always include both in your candidate set. Finally, be aware of the multiple comparisons problem: with thousands of patching tests, some will appear significant by chance. Apply a Bonferroni correction or use a false discovery rate control procedure.

Frequently Asked Questions on Circuit-Level Interpretability

This section addresses common questions from practitioners who are beginning to explore the hidden graph. The answers draw on collective experience from the mechanistic interpretability community and are intended to clarify conceptual and practical doubts. If you are new to circuit-level analysis, these FAQs will help you avoid common misunderstandings and focus your efforts effectively.

How do I know if my circuit is real versus a statistical artifact?

This is the most critical question. A circuit is considered real if it passes three tests: necessity (ablating the circuit degrades performance), sufficiency (activating only the circuit produces the correct output), and specificity (the circuit does not affect unrelated tasks). Additionally, the circuit should be stable across random seeds and training runs. If you train the same architecture from scratch with different initializations, the circuit should reappear in approximately the same location (though exact components may shift). Reproducibility is the strongest evidence. For a single model, use cross-validation by splitting your dataset into multiple folds and rediscovering the circuit on each fold. If the circuit structure is consistent, you can be more confident.

What if my model is too large for full patching?

For models with hundreds of billions of parameters, full activation patching is computationally prohibitive. In this case, use a two-step approach. First, use a smaller proxy model that has been trained on similar data to discover candidate circuits. Second, verify that those circuits exist in the larger model by performing patching only on the candidate components. Research suggests that circuits often transfer across model scales, especially if the architectures are similar. Alternatively, use attribution methods like integrated gradients to identify a small set of high-impact components, then patch only those. This reduces the search space dramatically. You can also use sparse probing to identify features that are causally relevant, then reverse-engineer the circuit connecting those features. While less thorough, this approach is feasible for the largest models.

How do I handle circuits that overlap in the same neurons?

Overlap is a sign of superposition. When two circuits share neurons, ablating one circuit may affect the other. To disentangle them, you can use different counterfactual inputs for each circuit. For example, if circuit A activates on singular nouns and circuit B on plural nouns, you can patch activations from a singular input to observe effects on circuit A without affecting circuit B. If the circuits cannot be separated by input, they may actually be part of a larger integrated computation. In that case, consider merging them into a single higher-level circuit. The key is to ensure that your interventions are targeted. Also check whether the overlapping neurons play different roles in each circuit by analyzing their weight vectors; if the same neuron uses different sets of outgoing weights for each circuit, it may be acting as a multiplexer. This is an active area of research.

What are the limits of circuit-level abstraction?

Circuit-level abstraction works best for tasks that have a clear, linear causal structure. It struggles with tasks that involve iterative reasoning, such as multi-step arithmetic or recursive algorithms, because the circuit may not be stationary across steps. Additionally, circuits are defined relative to a specific input distribution; they may not generalize to out-of-distribution inputs. The abstraction also assumes that computation is localized, but in some models, behavior emerges from global dynamics that cannot be decomposed into sparse circuits. Finally, the cost of discovery limits its application to only the most important behaviors. For now, circuit-level interpretability is a powerful tool for understanding safety-critical behaviors, but it is not a complete solution for model transparency. Combine it with other methods like representation engineering and behavioral testing for a holistic approach.

Synthesis and Next Actions

The hidden graph—the sparse, causal subgraphs that drive specific behaviors—offers a practical path toward interpretable deep learning. By shifting focus from individual neurons to functional circuits, practitioners can achieve surgical interventions, robust error analysis, and model verification. This guide has covered the conceptual foundations, a step-by-step discovery workflow, tooling options, scaling strategies, and common pitfalls. The field is rapidly evolving, and the techniques described here will continue to improve. However, the core principles of causal reasoning, careful experimental design, and validation will remain central. The next step for interested readers is to apply these methods to a concrete task using an open-source library like TransformerLens. Start with a small model (e.g., GPT-2 Small) and a well-defined task such as indirect object identification. Follow the workflow outlined in this guide, and expect to iterate several times before obtaining a clean circuit. Document your findings and share them with the community to contribute to the collective understanding of the hidden graph. As more circuits are mapped, we move closer to a future where neural networks are not just powerful but also transparent and trustworthy.