What you’ll learn

• Some motivation for why you should care about interpretability
• overview of current techniques, mostly on interp for language models
• open problems that might make for good projects

Slides with links

miv.name/teaching/MLSS-2026

Talk Outline

• Introduction
• “Classical” Interpretability, where it fails
• Transformer notation & review
• mechanistic interpretability

The Problem:

Gradient descent finds a set of parameters which minimize the training objective, but it’s just a point in a billions-dimensional space of possible models. We don’t build models the way we build cars or planes.

and when we give them agency, they misbehave – often in ways we don’t see coming.

What can we do?

• Can we understand how a neural network is performing computation?
• Can we use that understanding to control AI systems?

What does it even mean to “understand” a neural network anyway?

Input Attribution

When training a linear classifier with Lasso, you can just look at the weights to see what inputs mattered. With a deep neural net, this is not so easy!

• “Local Interpretable Model-Agnostic Explanations” (LIME), from Riberio et al (2016): locally around a sample, make a linear model that behaves like your big model

• “SHapley Additive exPlanations” (SHAP), from Lundberg et al (2017): much like LIME, but with nicer game-theoretic guarantees

miv.name/transformer-demo/classifier

Saliency maps

What part of the input is most relevant for the decision the network is making?

Feature visualization:

• Pick a neuron in an image model
• adjust an image to make that neuron activate more strongly
• cool pictures!

“Neural correlates of interspecies perspective taking in the post-mortem Atlantic Salmon: An argument for multiple comparisons correction” (Bennett et al)

• LIME/SHAP: a locally accurate representation of your bigger model. Useful when building a relatively simple classifier
• Saliency maps: which part of your input was used to make the decision? However, easily misleading
• Feature Visualization: Cool pictures, but neurons can be polysemantic. Unclear how to make this apply to LLMs.

Key issue: everything so far is merely correlational.

Let \mathcal{V} be a finite vocabulary (typically |\mathcal{V}| > 10^5):

• q_i \in \mathcal{V} is a “token” – usually a word or subword
• s = [q_1, \ldots, q_n] is a sequence of tokens
• E: \mathcal{V} \times \mathbb{Z} \to \mathbb{R}^{d_m} is a learned embedding into a residual stream of dimension d_m

"The quick red fox jumped over the lazy brown dog"

\Rightarrow

[
  "The", "quick", "red", "fox", "jumps",
  "over", "the", "lazy", "dog"
]

\Rightarrow \begin{bmatrix} 3, 17, 42, 8, 123, 56, 3, 78, 9 \end{bmatrix} \in \N_{d_v}^{9}

\Rightarrow X \in \R^{9 \times d_m}

A Language Model learns the conditional probability:

\mathbb{P}\left[ q_{n+1} \mid [q_1, \ldots, q_n] \right]

i.e. given a sequence s, predict the distribution over possible next tokens. By sampling from the distribution over the next token and appending it to the sequence, we can generate text.

Attention Heads

\mathbb{A}: \R^{n \times d_m} \to \R^{n \times d_m}

[\mathbb{A}(X)]_i = \sum_{j=1}^n \sigma(\mathcal{A}(X))_{i,j} W_{OV} x_j

where \sigma is the softmax function applied row-wise:

\sigma(B)_{i,j} = \frac{e^{B_{i,j}}}{\sum_k e^{B_{i,k}}}

and the “attention pattern” \mathcal{A}(X) \in \R^{n \times n} is a “measure of similarity” between the embeddings.

• \mathcal{A}(X)_{i,j} decides how much to move from j to i
• W_{OV} x_j decides what to move from j

Attention patterns

So where do we get \mathcal{A}(X) from? \mathcal{A}(X)_{i,j} = x_i W_{QK} x_j^T + M_{i,j} where M \in \R^{n \times n}, \quad M := \begin{cases} 0 & i \leq j \\ -\infty & i > j \end{cases}

Masking is necessary to prevent the model from “cheating” by looking at future tokens.

The learnable parameters of an attention head are thus given by the matrices W_{QK}, W_{OV} \in \R^{d_m \times d_m}. In practice, we learn low-rank approximations to both of these.

Demo:

miv.name/transformer-demo/index

A primary object of study in interpretability is the residual stream: the set of activations at various layers and sequence positions.

• starts as per-token embeddings (with positional information), no information from nearby tokens
• MLPs transform these, individual sequence positions at a time
• Attention heads move data from previous tokens to the present token

If W_U converts a final-layer residual vector into a next-token prediction, why not apply it to an intermediate one? This is exactly the logit lens, introduced by nostalgebraist (2020):

\texttt{logits}^{(\ell)} = X^{(\ell)} W_U \in \mathbb{R}^{n_c \times d_v}.

Tuned Lens

• Residual connections exist, so there is some level of stability in representations across layers. But, this stability is imperfect

• enter the Tuned Lens (Belrose et al., 2023): learn a small affine translator mapping intermediate to final-layer basis before unembedding

• the lenses tell you what the model predicts, but not why or how
• “explanations” are always of the form “probability distribution over tokens”
• set the foundation for later work – the residual stream is clearly carrying some cool stuff!

miv.name/transformer-demo/index

We’ve been watching: the logit lens shows what the model predicts at each layer.

"The Eiffel Tower is located in _____"

• Back in 2020-2022, it was pretty much a miracle that LLMs could answer such a question!
• Somewhere, in the weights of the model, this information must be stored.
• Somewhere, this information is recalled and put into the residual stream to increase the probability of the next token being "Paris"

The trick: a controlled swap

A debugging move you’ve all done: splice a value from a known-good run into a broken run to find where it diverges – git bisect, inside one forward pass.

However, this is trickier than it looks! Naively, given this one example, you might as well just find the (un)embedding of the token “Paris” and patch that in.

Causal tracing: bisect everywhere

Do the surgical swap at every site (\ell, t) \to a heatmap of causal importance.

From one part to a mechanism

What the swap bought us:

• causal, not just correlational
• a repeatable loop: localize \to ablate / edit \to verify
• direction matters: denoising tests sufficiency, noising tests necessity (not symmetric)

Language models are just Markov chains, right?

Try the following:

• generate a random string, random number, anything – something that could not have shown up in the training data
• ask a language model to repeat it

Language models learn in-context. Even in the simplest possible case, you can think of them learning an n-gram during a single forward pass!

Given a sequence:

A B C D E ... A B C D _

How does a language model correctly predict E?

Two steps:

• a Previous Token Head copies from t-1 onto t the information that “the last token was x”
• the actual Induction Head, given the present token q_n, attends back to positions t which have stored that q_{t-1} = q_n, and copies the actual q_{t} information to the “present” to predict q_{n+1}

“In-context Learning and Induction Heads”, Olsson et al (2022)”

From one circuit to the general idea

The induction head was our first circuit: a small group of components that together implement a behavior.

What makes a subgraph a “circuit”?

A circuit C should be:

• faithful – running just C reproduces the behavior on the task
• minimal – remove any piece and it breaks
• interpretable – each component has a human-understandable role

Olah et al., 2020; Elhage et al., 2021

A bigger circuit: Indirect Object Identification

“When Mary and John went to the store, John gave a drink to ___”

The model repairs itself

IOI also has backup name-mover heads: dormant heads that take over when the primary name-movers are ablated.

Finding circuits automatically

Hand-analysis doesn’t scale. Automate the search over the graph:

• ACDC (Conmy et al., 2023): from the full graph, patch each edge (= path patching) with a corrupted activation, then greedily prune the edges that don’t matter
• Edge attribution patching (EAP / EAP-IG) (Syed et al., 2023; Hanna et al., 2024): approximate every edge’s effect with one backward pass – much faster than ACDC

Did we actually find it?

Validate a candidate circuit C by ablation:

• faithful: C alone does the task
• complete: the complement \bar{C} alone fails it
• minimal: no smaller subset is faithful
• generalizes: holds across the task distribution

Atoms

Circuits over heads and neurons hit a wall:

• a single head can carry several unrelated signals
• entire MLPs are far too coarse
• individual neurons are polysemantic

Problem 1:

The model has ~no incentive to have it’s internal representations align with your canonical basis vectors (neurons)!

Problem 2:

The Johnson–Lindenstrauss lemma (Johnson & Lindenstrauss, 1984):

n vectors can be embedded in \mathbb{R}^{d_m} with pairwise near-orthogonality

|\langle u, v\rangle| \le \epsilon

as long as

d_m = O(\log n / \epsilon^2)

The number of “almost orthogonal” directions grows exponentially in the dimension, not linearly.

The model can store more features than it gets orthogonal directions!

miv.name/transformer-demo/superposition

Leads us to the current paradigm: the residual stream contains dense vectors, but these secretly contain many features, which we would like to find

Going back in time a bit: Alain & Bengio, 2016 introduce linear probes:

Given some known feature (that you have labels for) of the dataset, you can train an affine model to predict the feature given only the target model’s activations. Whether or not it works might tell you something about whether the model is using this known feature.

For example:

Your instructor said " there will be no homework for this course " and the students were glad.
0    0          0    1 1     1    1  1  1        1   1    1      1 0   0   0        0    0

def probe(text: list[str]) -> list[bool]:
  output: list[bool] = []
  # count the quotes
  n_quotes: int = 0
  for x in text:
    if x == '"':
      # if a quote, increment count and append True
      n_quotes += 1
      output.append(True)
    else:
      # otherwise, True if there have been an odd number of quotes
      output.append(bool(n_quotes % 2))

Linear Probes provide a supervised way to figure out which directions represent a feature. However:

1. A model with billions of parameters has probably got more features than you’d like to write detectors for
2. Humans are not very good at knowing what a language model is computing – if we were, language models would be software, not neural networks!

What is an autoencoder?

Here we have the reverse problem: our data has already been packed into a high dim space. We need to unpack it.

Given some intermediate activation x, learn an autoencoder

z = \mathrm{ReLU}\!\bigl(W_{u} x + b_{u}\bigr) \in \mathbb{R}^n, \qquad \hat{x} = W_{d}\, z + b_{d} \in \mathbb{R}^{d_m}.

with loss

\mathcal{L} = \lVert x - \hat{x}\rVert_2^2 + \lambda \lVert z \rVert_1 .

• The rows of W_u (or columns of W_d) are the dictionary items – the actual features we care about
• the L_1 penalty on the activations z is what gets us sparsity

See: Bricken et al., 2023, Cunningham et al., 2023, Templeton et al., 2024.

Automated Interpretability

Just give a language model examples of text where the feature activates, and ask it to explain what’s going on!

Variants and Extensions

Lots of variants of SAEs exist and are an active area of research, but the key idea of “decomposing activations into a sparse set of features” is the core.

See:

• neuronpedia.org
• miv.name/transformer-demo/index

Sparse Autoencoders let us decompose activations in an unsupervised way, but intervening on these activations can be brittle. What if we could decompose the weights instead?

Given some network f_\theta : \mathbb{R}^d \to \mathbb{R}^d taking inputs x \sim \mathcal{D} \subset \mathbb{R}^d with learned parameters \theta = \{W^1, \ldots, W^L \}, SPD learns:

• rank-one components C_{\ell,i} = u_{\ell,i} \cdot v_{\ell,i}{}^T which approximate the model weights at a layer \ell
• gates g_{\ell,i}: \mathbb{R}^{d} \to [0,1] whose sparsity is encouraged

How do we learn this?

Stochastic Reconstruction Loss

Given some input x, we have the “original” forward pass f_\theta(x), and we want our “masked” forward pass with weights \hat{\theta} to give the same output.

1. For each component C_{\ell,i}, sample stochastic mask m_{\ell,i} \sim \mathcal{U}[g_{\ell,i}(x), 1]
2. Construct replacement model parameters \hat{\theta} \hat{\theta} = \{\hat{W}^1, \ldots, \hat{W}^L \} \qquad \text{where} \qquad \hat{W}^\ell = \sum_{i} m_{\ell,i} C_{\ell,i}
3. Compute divergence: \mathcal{L}_{\text{recon}} = \bigg\| f_\theta(x) - f_{\hat{\theta}}(x) \bigg\|

Stochastic Parameter Decomposition:

Slice up (decompose) a network's weights (parameters) into rank-1 components so that only some of them are needed for any given computation, and the rest can be set to arbitrary magnitudes (stochastically)

More on parameter decomposition methods:

goodfire.ai/research/interpreting-lm-parameters

1. come up with a feature that you think a language model might compute
2. train a linear probe to see if an LLM of your choice computes this feature
3. where in the network does the probe work best?
4. what happens when you ablate this feature? Can you control the behavior in a meaningful way?
5. are there SAE features which correlate (data-wise, or via cosine similarity) with your feature?

Surveys and orientation

Read one survey first to get the lay of the land, then a problem list to find something to work on.

Courses and hands-on tutorials

Tooling stack