AI Interpretability’s Dead Salmons (Notebook)#
Author: Valentina Staneva
Adapted from an exploratory notebook by Jesse Hartman.
pip install --quiet marimo
import marimo as mo
!gdown --id 1I4nRhvYoGciCW_KPBkz8rzswlXshD5f8
/usr/local/lib/python3.12/dist-packages/gdown/__main__.py:139: FutureWarning: Option `--id` was deprecated in version 4.3.1 and will be removed in 5.0. You don't need to pass it anymore to use a file ID.
warnings.warn(
Downloading...
From: https://drive.google.com/uc?id=1I4nRhvYoGciCW_KPBkz8rzswlXshD5f8
To: /content/salmon.png
100% 278k/278k [00:00<00:00, 28.8MB/s]
Exploring Statistical Fragility#
A visual companion to “The Dead Salmons of AI Interpretability” (Méloux, Dirupo, Portet & Peyrard, 2025) — which revives the famous dead-salmon fMRI cautionary tale and argues that modern AI interpretability methods are vulnerable to the same class of statistical false-positive failures: feature attribution, probing, sparse autoencoders, concept directions and mechanistic circuit search can all produce confident-looking explanations for randomly initialized models on random data.
This notebook allows to see the failure modes the paper describes, not just read about them. Each section below rebuilds one of those traps as an interactive marimo demo: drag a slider, move a threshold, watch noise dress itself up as a finding. The voxel demo immediately below is the warm-up; the modern interpretability case studies start further down.
Move the sliders below to see how the false-positive rate and Bonferroni threshold change with the experiment shape.
The gist, in plain English#
Imagine 20,000 people each flip a coin 200 times, and you look for someone whose flips match a pattern you picked in advance. Even though every coin is fair, with that many people you’ll find about 1,000 whose flips “match” by pure luck.
That’s what happens here — except the “people” are voxels (tiny 3D pixels of a brain scan) and the “pattern” is a fake task. No voxel has any real signal. But if you accept anything with “p < 0.05” as a finding, you’ll announce ~1,000 discoveries that are entirely noise.
Multiple Testing Problem!
The Bonferroni correction raises the bar: a finding only counts if
its p-value beats 0.05 / 20,000. Apply it and the false alarms almost
entirely disappear.
📝 Note: Bonferonni Correction is a rather conservative approach to address the multiple testing problem, and it reduces the power of the test. There are other “trade-off” approaches which control the false discovery rates.
Does this happen with real data? Yes.#
We just showed pure noise producing false positives. The math doesn’t care whether the “noise” is synthetic — any time you run many tests with no real effect, ~alpha of them will flag as “significant”.
Below we pull a classic cars dataset from vega-datasets and
p-hack it: split the cars into two groups by coin flip (no real
difference between groups), then run a t-test on every numeric
feature. Repeat many times. By design every test is null — yet ~5%
of them will come up “significant”.
cars.json — 406 cars ·
features tested: Miles_per_Gallon, Cylinders, Displacement, Horsepower, Weight_in_lbs, Acceleration
Tests run: 500 random splits × 6 features
= 3,000 t-tests, all null by construction.ML/AI Interpretability: the same trap, in modern clothes#
Everything above was classical statistics dressed for fMRI and for cars. The same failure mode shows up across modern ML interpretability tools. Below: three case studies where confident-looking explanations turn out to be artifacts of high-dimensional noise, not of anything the model actually learned or the data actually contains.
Feature Attribution#
Saliency maps, Grad-CAM, SHAP, Integrated Gradients — the promise is: “here are the parts of the input the model cared about.” Adebayo et al. (2018) showed that many popular attribution methods produce nearly identical heatmaps even when you randomize the model’s weights. That means the pretty overlay is mostly a projection of the input, not evidence of what the model learned.
Below: we train a tiny linear classifier on synthetic 16×16 shapes
(circles vs squares), then compute input-times-gradient saliency for
both the trained weights and a fresh random model. The “hot” regions
line up on the shape in both cases — because |W·x| is near zero
wherever x is near zero, regardless of what W is.
Probing#
Probing classifiers are the “did the model encode X?” tool: train a linear probe on a model’s internal activations to decode some property, and take success to mean the model “knows about” that property.
Hewitt & Liang (2019) showed the trap: a probe can succeed even when the model is not using the decoded information for its output. A high-dimensional projection preserves almost everything fed into it — a probe can recover a feature from the activations even if the downstream readout completely ignores it.
Below: synthetic features flow through a fixed random hidden layer. A
linear readout is trained on y = sin(1.5·x1) + noise — only x1
matters. For every feature we measure:
Probe R² — can a linear probe decode
xⱼfrom the hidden activations?Ablation impact — when we replace
xⱼwith its mean, how much does the model’s output change?
The probe says “yes, it’s all in there!” The ablation says “no, only
x1 is actually used.” Both are true at the same time.
probing_n_nuisance = mo.ui.slider(
start=2, stop=20, step=1, value=5, label="nuisance features"
)
probing_n_nuisance
Sparse Autoencoders#
Sparse autoencoders fit on the model’s activations with an L1 penalty produce a dictionary of features deemed to be “interpretable” (Cunningham et al., 2023). Each feature is then hand-labeled (“this one fires on Arabic script”, “this one on car windshields”) and taken as evidence of the model’s internal conceptual vocabulary. However, Heap et al., 2025 showed that a SAEs can recover “interpretable” components even in randomly initialied transformers.
Fit the same optimizer to pure Gaussian noise and you still get a
dictionary of crisp, human-nameable “features.” Below: 2000 random 8×8
noise patches → MiniBatchDictionaryLearning with 32 atoms. Each tile
in the gallery is one learned atom. Several will look spatially
localized, oriented, blob- or edge-resembling — all the properties a
real SAE atom would have. The structure is in the optimizer, not the
data.
Concept-Based Explanations#
TCAV (Kim et al. 2018) and its descendants frame interpretability as “we found a direction in activation space aligned with a human concept, and the model is sensitive to that direction — therefore the model uses that concept.” In high-dimensional activation spaces, the trap is the same one we’ve seen all along: any random direction, out of enough random directions, will appear “significantly” aligned with the model’s output.
Below: we treat the Phase-1 shape classifier’s weight vector attr_W
as the “model’s decision direction” in 256-dim input/activation space,
then generate N random unit-vector “concepts” and compute each
concept’s cosine similarity with the model’s decision direction.
Under the null (random concepts), cosine similarity is approximately
N(0, 1/256) — we z-score and two-sided-test each concept, then feed
the p-values into the same hunter widget. Naive p < 0.05 produces
~5% of concepts looking “significant.”
Mechanistic Interpretability#
The deepest corner of the interpretability tree: find circuits — small subsets of neurons that implement a specific model behavior. At small scale, with cherry-picked examples, this can be genuinely illuminating. At scale, with automated search across many candidate circuits and many candidate behaviors, it runs straight into the same wall: brute-force search across enough noisy candidates will “discover” circuits that look convincingly specialized for any target behavior you pick, even when the neurons are random and the behavior is a coin flip.
Below: we define a “target behavior” by flipping a coin for each of 400
random input vectors — so the labels have no underlying relationship to
the inputs at all. We then search through N candidate two-neuron
“circuits” (random weight matrices feeding a random linear readout) and
two-sample t-test each circuit’s output across the random labels.
Feed the per-circuit p-values into the hunter widget; the naive
threshold announces dozens of “discovered circuits” for a behavior
that doesn’t exist.