What is Explainable AI (XAI)?
Explainable AI (XAI) is the field of methods and practices for producing evidence or representations that help a defined audience understand selected AI-system outputs, behavior, or mechanisms for a stated purpose.
Quick Facts
| Specification | Official Specification |
|---|
How It Works
Define the explanation contract first
State who needs the explanation, which model output or mechanism is in scope, what action the audience may take, and what evidence would make the explanation adequate. NISTIR 8312 separates explanation delivery, meaningfulness, explanation accuracy, and knowledge limits. An engineer, auditor, affected user, and domain expert can require different vocabulary and detail for the same system.
Choose a method that answers the question
Intrinsic models expose understandable structure directly; post-hoc methods analyze an already trained model. Local explanations address one prediction, while global explanations summarize behavior over a declared population. Gradient attribution, SHAP, LIME, Grad-CAM, examples, counterfactuals, probes, and mechanistic interventions produce different evidence. Their outputs are not interchangeable merely because each is labeled XAI.
Evaluate the explanation and the system separately
Measure fidelity to the model, stability across seeds and meaning-preserving perturbations, sensitivity when the prediction changes, sparsity or complexity, coverage, and task usefulness with representative users. research on rigorous interpretability evaluation distinguishes application-grounded, human-grounded, and functionally grounded tests. Model accuracy, calibration, fairness, privacy, robustness, and safety remain separate requirements.
Key Characteristics
- Starts from a declared audience, question, decision, and model output
- Includes intrinsic transparency and post-hoc local or global explanation methods
- Can target input features, examples, concepts, internal components, or behavior
- Requires fidelity, stability, sensitivity, coverage, and usability evidence
- Treats explanation artifacts as versioned outputs tied to a model and data scope
- Does not make a model fair, causal, correct, safe, or compliant by itself
Common Use Cases
- Debugging whether a model relies on intended signals or brittle shortcuts
- Explaining a specific prediction to an operator, reviewer, or affected user
- Comparing model versions with stable explanation and behavior tests
- Supporting audit evidence for a bounded decision process
- Generating hypotheses for data repair, model redesign, or causal experiments
Example
Loading code...Frequently Asked Questions
What is the difference between explainability and interpretability?
Usage varies. A practical distinction treats interpretability as how directly a person can understand a model or representation, while explainability includes processes that produce reasons or evidence about an otherwise opaque system. Teams should define their terms, audience, and acceptance test instead of relying on the label alone.
What is the difference between local and global explanations?
A local explanation addresses one prediction or a narrow neighborhood, such as feature contributions for one applicant. A global explanation summarizes behavior across a declared population, such as response curves or rules. Aggregating local explanations can support global analysis, but sampling, interactions, and subgroup coverage determine what it represents.
Does an intuitive explanation prove that it is faithful?
No. A heatmap, rule, or natural-language rationale can look convincing while being insensitive to model parameters or omitting decisive interactions. Test it against the model with perturbations, randomization, held-out cases, repeated seeds, counterfactuals, and task-specific fidelity metrics before using it as evidence.
Can Explainable AI prove fairness, safety, or causality?
No. Explanations can reveal hypotheses about sensitive features, shortcuts, or failure paths, but fairness needs subgroup and policy analysis, safety needs hazard and behavior testing, and causality needs an identified causal design. A feature attribution explains a configured model output, not necessarily the real-world cause of an outcome.
How should an XAI method be selected?
Start from the decision and stakeholder question, then choose the narrowest method whose assumptions fit the model, data, modality, and access level. Predefine fidelity, stability, usefulness, privacy, latency, and failure criteria. Compare at least one alternative and retain the underlying model and outcome evidence alongside the explanation.