What is Gibbs Sampling?
Gibbs Sampling is a Markov Chain Monte Carlo method that updates one variable or block at a time by drawing exactly from its full conditional distribution given the current values of all other variables.
Quick Facts
| Specification | Official Specification |
|---|
How It Works
Compose full-conditional transitions
For target pi(theta_1, ..., theta_d), a coordinate update draws theta_j from pi(theta_j | theta_-j). Cycling through coordinates constructs a transition that leaves the joint target invariant. A complete sweep is one iteration only if that convention is stated consistently.
The UC Berkeley Gibbs sampler notes demonstrate the update with a correlated bivariate normal and show why conditional dependence controls movement. Gibbs is often described as a special Metropolis-Hastings case whose exact conditional proposal is always accepted.
Choose coordinates, blocks, and conditional solvers
Scalar updates are easy when every full conditional belongs to a standard distribution, but highly correlated parameters may move only in small coordinate-wise steps. Blocking related parameters can reduce autocorrelation while increasing the cost or difficulty of each conditional draw.
If a full conditional cannot be sampled exactly, a Metropolis-Hastings, slice, or other transition may be nested inside the sweep. This Metropolis-within-Gibbs construction must preserve the conditional target and record rejections; it is not exact Gibbs for that block.
Diagnose the joint chain and every estimand
A chain can reproduce each conditional update correctly and still fail to traverse joint modes. Run multiple chains from dispersed valid states, inspect joint and marginal traces, and account for label switching rather than averaging incompatible labels. Report rank-normalized split R-hat, bulk and tail ESS, and Monte Carlo standard error.
Validate with a known joint distribution, posterior predictive checks, or an alternative sampler. Increasing sweeps cannot repair an incorrect conditional, incompatible update order, hidden constraint, or model that does not identify the quantity of interest.
Key Characteristics
- Updates coordinates or blocks from full conditional distributions
- Accepts exact conditional draws without rejection
- Uses systematic or declared random scan schedules
- Can embed Metropolis steps for intractable conditionals
- Exploits conjugacy and conditional model structure
- May mix slowly under strong posterior dependence
Common Use Cases
- Conditionally conjugate hierarchical Bayesian models
- Latent-variable models with tractable conditional blocks
- Missing-data and data-augmentation procedures
- Discrete assignments in mixture and graphical models
- Baseline sampling for validating more complex transitions
Example
Loading code...Frequently Asked Questions
Why are Gibbs Sampling proposals always accepted?
An exact full conditional already includes the target's relative probability for that coordinate given all others, so the corresponding Metropolis-Hastings ratio reduces to one. Approximate conditional draws do not inherit this property automatically.
What is one Gibbs Sampling iteration?
Under systematic scan, one iteration commonly means a complete sweep through every coordinate or block. Under random scan it may mean one selected update. Reports must state the convention before comparing iteration counts or ESS.
Why can Gibbs Sampling mix slowly?
When parameters are strongly correlated, one-at-a-time conditionals may permit only small movement across the joint posterior. Near-deterministic constraints, multimodality, and label switching can make the problem worse.
What is Metropolis-within-Gibbs?
It replaces an unavailable exact full-conditional draw with a valid Metropolis-Hastings transition targeting that conditional. Rejections must be retained, and proposal tuning and diagnostics are required for each embedded block.
Should Gibbs Sampling be preferred whenever conditionals are available?
Not automatically. Compare mixing and effective samples per unit time with blocked Gibbs, HMC, NUTS, or other valid methods. Exact coordinate updates can still be statistically inefficient for correlated or multimodal targets.