What is Feature Visualization?
Feature Visualization is a family of interpretability methods that characterize what a neural-network unit, channel, layer, class, or feature direction responds to by inspecting high-activation examples or synthesizing inputs that optimize a selected activation objective.
Quick Facts
| Specification | Official Specification |
|---|
How It Works
Choose the feature and objective
Identify the checkpoint, preprocessing, layer, unit or direction, spatial aggregation, and scalar objective. Maximizing a pre-softmax class logit differs from maximizing a probability, which can rise by suppressing alternatives. Feature Visualization documents objectives for neurons, channels, layers, logits, and classes, together with the optimization choices that shape their outputs.
Constrain optimization with an input prior
Freeze model weights and update only the input or generator latent. Pixel-space optimization often needs L2 or total-variation penalties, frequency controls, jitter, rotation, scaling, clipping, or a learned generative prior. These choices suppress some artifacts but also restrict what can be found. Report the objective, regularizers, augmentation distribution, step size, iterations, seeds, and stopping rule.
Validate semantics beyond a compelling image
Compare multiple optimized restarts with top and bottom natural examples, negative directions, neighboring units, and class controls. Quantify activation on held-out labeled concepts when possible and test whether an intervention changes relevant behavior. A recognizable picture establishes that an input can drive the feature; it does not prove the unit represents only that concept or that the concept causes a prediction.
Key Characteristics
- Targets a declared neuron, channel, layer, class logit, or feature direction
- Uses high-activation dataset examples or optimization-generated inputs
- Freezes model parameters while changing the input or generator latent
- Depends strongly on regularization, augmentation, initialization, and objective choice
- Needs diverse restarts and natural-example controls to expose multiple facets
- Shows feature preferences but does not by itself establish causal decision use
Common Use Cases
- Inspecting visual patterns that maximize convolutional channels or class logits
- Comparing feature complexity across early and late neural-network layers
- Diagnosing whether a unit responds to an object or a correlated background cue
- Generating hypotheses for network dissection, ablation, or robustness tests
- Auditing how optimization priors change the apparent meaning of a feature
Example
Loading code...Frequently Asked Questions
What is the difference between feature visualization and attribution?
Feature Visualization asks what inputs strongly activate a selected model feature, often by retrieving examples or optimizing an input. Attribution starts from a particular observed input and estimates which parts contributed to an output or activation. They answer different questions and can be used together.
Why do unregularized feature visualizations look noisy?
Neural networks can respond strongly to high-frequency or adversarial patterns that lie outside natural-image statistics. An unconstrained optimizer exploits those directions. Frequency penalties, total variation, transformations, clipping, and learned priors alter the search space, but each also biases the result.
Should feature visualization maximize a class probability or a logit?
A class probability can increase because competing classes are suppressed, even if evidence for the target barely rises. A pre-softmax logit more directly measures target evidence, though it still needs regularization and controls. State the objective exactly and compare alternatives when conclusions depend on it.
Does one generated image reveal a neuron's true meaning?
No. A feature may be polysemantic or respond to several distinct patterns, and one optimization run can converge to a single facet. Use multiple seeds, diversity penalties, positive and negative directions, natural high-activation examples, neighboring-unit controls, and quantitative concept tests.
Can Feature Visualization prove that a feature causes a prediction?
No. It demonstrates inputs that drive a selected activation under the chosen objective and prior. To test decision relevance, intervene on the feature or corresponding activation, measure a predeclared output metric, and check specificity, sufficiency, necessity, and behavior on held-out examples.