What is Overfitting?
Overfitting is a failure of generalization in which a model or model-selection process adapts to finite training or validation data so specifically that it performs worse than a less adapted alternative on new data from the target distribution.
Quick Facts
| Created | Concept formally established in statistical learning theory |
|---|---|
| Specification | Official Specification |
How It Works
What overfitting is relative to
Overfitting is relative to a dataset, learning procedure, selection process, metric, and target distribution. A model can generalize within the sampled population yet fail under deployment shift, or perform poorly everywhere because it underfits. Diagnose the relationship between empirical risk and an independent estimate of target risk rather than assigning the label from training accuracy alone.
Why a generalization gap is only a signal
Lower training loss than validation loss is expected because the optimizer saw the training data. The size and trend of that gap become useful only when preprocessing is fit on training data, related entities stay in one split, temporal order is respected, metrics match the task, and uncertainty across splits or seeds is reported. The scikit-learn guidance shows how leakage can make validation results optimistic rather than reveal overfitting.
Model-selection overfitting
A validation set becomes part of the optimization loop when teams repeatedly choose features, hyperparameters, checkpoints, prompts, or thresholds from its score. Performance on that set then becomes optimistically biased. Use a final untouched test set or nested cross-validation when the same finite sample drives model selection; research by Cawley and Talbot documents why selection-criterion variance can create substantial bias.
Capacity is not parameter count alone
More flexible models can fit spurious patterns, but a large parameter count or zero training error does not establish poor generalization. Modern overparameterized models can interpolate training data and still generalize because architecture, optimizer, initialization, data augmentation, margins, and other explicit or implicit regularizers shape the learned solution. Double-descent research shows why the classical single U-shaped complexity curve is not universal.
How to choose a remedy
Fix the diagnosed failure, not the label. Improve the split or remove leakage when evaluation is invalid; collect more representative data when coverage is weak; reduce capacity, add regularization, augment data, or stop earlier when variance is the issue; and change the target or features when the model is learning the wrong signal. Keep an untouched test set for the final estimate and monitor production drift after release.
Key Characteristics
- Defined by worse expected performance on new target-distribution data, not by training loss alone
- Often appears as a persistent train-validation gap under a representative, leakage-free evaluation
- Can affect parameters, hyperparameters, feature selection, checkpoints, prompts, or thresholds
- Depends on data volume, noise, inductive bias, optimization, and evaluation design as well as model capacity
- Can be hidden or mimicked by leakage, duplicate entities, temporal mistakes, and distribution shift
- Requires uncertainty and slice analysis because one split or aggregate score can be misleading
Common Use Cases
- Reading training and validation learning curves without treating one fluctuation as a diagnosis
- Selecting model capacity, regularization strength, or an early-stopping checkpoint
- Designing grouped, temporal, stratified, or nested validation for the deployment unit
- Auditing fine-tuned language models for task, safety, and memorization regressions
- Separating optimization failure from data leakage and production distribution shift
Example
Loading code...Frequently Asked Questions
What is overfitting in machine learning?
Overfitting is a generalization failure: a model or selection process adapts to finite development data so specifically that it performs worse than a less adapted alternative on new examples from the target distribution. Low training error can accompany overfitting, but it is not sufficient to prove it.
Does a training-validation gap always prove overfitting?
No. A persistent gap on representative, leakage-free splits is useful evidence, but label noise, group or temporal leakage, preprocessing mistakes, metric variance, and distribution shift can change the gap. Confirm the split, report uncertainty, and compare learning curves before choosing a remedy.
What is the difference between overfitting, data leakage, and distribution shift?
Overfitting means the learning or selection process adapts too specifically to finite development data. Leakage lets unavailable information influence training or selection and usually makes evaluation too optimistic. Distribution shift means deployment data differs from the evaluation distribution; a model can fail under shift without having overfit the original distribution.
Do more parameters or zero training error necessarily mean a model is overfit?
No. They show capacity and fit, not generalization failure. Modern neural networks can interpolate training data and still perform well on new samples because the learned solution also depends on architecture, optimizer, initialization, data, augmentation, and explicit or implicit regularization.
How should I prevent or reduce overfitting?
First validate the evaluation design. Then match the remedy to the cause: improve data coverage, group or time splits, regularize, augment data, reduce effective capacity, stop earlier, or use nested cross-validation for model selection. Confirm the final choice once on an untouched test set.