What is Bayesian Neural Network?
A Bayesian Neural Network is a neural network model that places a prior distribution over selected parameters or functions, updates that distribution with observed data, and forms predictions by marginalizing over the resulting posterior.
Quick Facts
| Specification | Official Specification |
|---|
How It Works
Define priors and likelihoods on meaningful scales
A prior over weights induces a prior over functions, but simple independent weight priors can produce unintuitive function behavior. The likelihood must match the target: categorical for classes, Gaussian only when its residual assumptions are defensible, and a suitable count, survival, or heavy-tailed model otherwise. Wilson and Izmailov emphasize Bayesian marginalization over single-point weights; this does not remove the need to inspect the implied function prior and data model.
Approximate the posterior without hiding approximation error
For weights w and data D, Bayes' rule gives p(w|D) proportional to p(D|w)p(w). Variational methods optimize a tractable distribution, Laplace methods fit a local Gaussian around a MAP solution, and sampling methods attempt to explore the posterior directly. Bayes by Backprop provides a scalable variational construction, but its benchmark findings do not establish that every variational BNN captures multiple modes or tail uncertainty.
Validate computation, prediction, and decisions separately
First diagnose the inference algorithm: optimization convergence, chain mixing where applicable, sensitivity to priors, and stability across seeds. Then evaluate posterior predictive Log Loss or Brier Score, calibration, interval coverage and width, task quality, and representative shift slices. Posterior predictive checks ask whether simulated data reproduce decision-relevant structure; they are not a substitute for untouched out-of-sample evaluation or external safety controls.
Key Characteristics
- Combines a neural likelihood with explicit prior assumptions
- Represents selected weights, functions, or outputs as random quantities
- Forms predictions by integrating or averaging over a posterior approximation
- Usually requires approximate inference at modern neural-network scale
- Can express parameter uncertainty but can still be misspecified or overconfident
- Requires computational diagnostics and predictive validation under deployment conditions
Common Use Cases
- Producing uncertainty-aware regression or classification predictions
- Incorporating defensible prior knowledge in data-sparse scientific models
- Supporting review or abstention policies when evidence is weak
- Driving information-aware exploration or active data collection
- Comparing posterior predictive behavior across model and prior choices
Example
Loading code...Frequently Asked Questions
How is a Bayesian Neural Network different from a standard neural network?
A standard training pipeline usually returns one parameter estimate. A BNN defines a prior and likelihood, infers a posterior over selected unknowns, and averages predictions over that posterior. In practice the posterior is approximate, so the distinction does not guarantee better uncertainty.
Do Bayesian Neural Networks put distributions over every weight?
Not necessarily. Some methods model all weights, while scalable systems may use a last-layer, subnetwork, low-rank, function-space, or local posterior. The modeled subset and all fixed components must be documented because they determine which uncertainty the predictor can express.
Are Bayesian Neural Networks always better calibrated?
No. Calibration depends on the prior, likelihood, inference approximation, data, and evaluation distribution. A BNN can be poorly calibrated or confidently wrong under misspecification and shift. Compare proper scores and reliability against deterministic and ensemble baselines.
Why is exact inference difficult in Bayesian Neural Networks?
A modern network has a high-dimensional, non-linear parameter space with symmetries and multiple plausible regions. The posterior normalizing integral and posterior predictive integral are generally unavailable in closed form, making exact enumeration or integration infeasible.
When should I use a Bayesian Neural Network?
Use one when a defensible probabilistic model and posterior-aware decisions justify the added inference and serving cost. Establish a deterministic or ensemble baseline first, then require measurable gains in predictive quality, calibration, decision utility, or data acquisition.