What is Gradient Descent?
Gradient Descent is a first-order optimization method that iteratively updates parameters by subtracting a positive step size times the objective gradient, using local slope information to seek a lower objective value.
Quick Facts
| Full Name | Gradient Descent Optimization Algorithm |
|---|---|
| Created | Proposed by Augustin-Louis Cauchy in 1847 |
| Specification | Official Specification |
How It Works
Separate differentiation from optimization
Backpropagation or automatic differentiation computes derivatives of a composed model and loss. Gradient descent decides how to turn those derivatives into a parameter update. Plain gradient descent uses the full gradient; stochastic and mini-batch methods estimate it from samples. Momentum, Adam, and other optimizers transform gradient history or scale coordinates, so they are related first-order methods rather than synonyms for the basic update.
Control step size, noise, and conditioning
A learning rate that is too large can oscillate or diverge, while a very small one wastes computation. Poorly scaled features and ill-conditioned curvature produce slow zig-zagging. Mini-batch size changes gradient variance and hardware efficiency, so learning rate and batch size cannot be tuned independently. Warmup, decay, normalization, momentum, clipping, and preconditioning address different symptoms and should be justified by monitored runs.
Verify optimization without confusing it with generalization
Track training loss, validation metrics, gradient norms, update-to-parameter ratios, non-finite values, and throughput. Use a finite-difference gradient check on small deterministic cases to catch derivative bugs. A stationary point is not necessarily a global minimum, and lower training loss does not guarantee better unseen performance. Release decisions require both optimization evidence and held-out task evaluation.
Key Characteristics
- Uses first-order derivative information to construct local parameter updates
- Depends on a step-size rule and objective geometry for descent or convergence guarantees
- Can use full-data, single-sample, or mini-batch gradient estimates
- Is distinct from backpropagation, which computes the gradient
- May approach a stationary point without finding a global optimum
- Requires numerical, optimization, and held-out task diagnostics
Common Use Cases
- Fitting linear or logistic models by minimizing empirical risk
- Updating neural-network parameters from backpropagated gradients
- Optimizing embedding and matrix-factorization objectives
- Testing optimizer implementations with deterministic convex functions
- Training large models with distributed mini-batch gradient estimates
Example
Loading code...Frequently Asked Questions
Does moving against the gradient always reduce the loss?
Only an infinitesimal negative-gradient direction is guaranteed to be locally descending when the gradient exists. A finite update also depends on step size and curvature. With noisy mini-batch gradients, individual updates may increase full-data loss even when the method improves it in expectation.
What is the difference between batch, stochastic, and mini-batch gradient descent?
Batch gradient descent uses the full dataset for each update. Basic stochastic gradient descent samples one example, while mini-batch SGD averages a subset. Their update costs, variance, parallel efficiency, and convergence behavior differ; mini-batch is common in deep learning because it maps well to accelerators.
How is gradient descent different from backpropagation?
Backpropagation efficiently computes derivatives through a computational graph. Gradient descent consumes those derivatives to update parameters. A model can use backpropagation with Adam or another optimizer, and gradient descent can optimize objectives whose gradients were derived analytically or numerically.
Does gradient descent find the global minimum?
For suitable convex objectives and step-size conditions, strong convergence guarantees are possible. Neural-network losses are non-convex, so practical methods usually seek a useful low-loss stationary region rather than prove global optimality. Initialization, noise, conditioning, schedule, and architecture all affect the result.
How do you debug a gradient-descent training run?
Start with a tiny deterministic batch, verify analytical gradients against centered finite differences, and confirm loss decreases for a small step. Then monitor non-finite values, gradient and update norms, learning-rate changes, training versus validation metrics, and throughput before adding distributed or mixed-precision complexity.