What is Curriculum Learning?
Curriculum Learning is a training strategy that changes the distribution, weight, or availability of examples, tasks, or model capacity over time according to a declared notion of difficulty or learning progress.
Quick Facts
| Specification | Official Specification |
|---|
How It Works
Define difficulty independently of the desired conclusion
Difficulty may come from domain rules, sequence length, noise, baseline loss, confidence, disagreement, success rate, or task parameters. Static curricula freeze this ordering; self-paced methods update sample weights from the current learner; teacher or reinforcement-learning policies adapt the schedule. Avoid using final test labels, and check whether the measure merely encodes class frequency, source, language, or another protected or operationally important attribute.
Specify pacing and restoration of the target distribution
The original Curriculum Learning paper connects gradually changing training distributions with continuation methods. Engineering details determine the intervention: start threshold, weight function, stage duration, promotion rule, replay of earlier examples, and whether the final phase samples the complete target distribution. Curriculum Learning is not synonymous with staged Fine-tuning or sorting one Prompt's demonstrations.
Compare schedules under equal budgets
A broad survey documents predefined, self-paced, teacher-driven, and other variants, as well as the difficulty of choosing ranks and pace. Compare random, curriculum, reversed, and hard-mining baselines with the same data, updates, tokens, optimizer, and stopping budget. Report learning curves, final task and slice metrics, seed variance, compute, forgotten easy cases, and performance after restoring the target distribution.
Key Characteristics
- Changes example, task, loss, environment, or capacity exposure over training time
- Requires an explicit Difficulty Measurer and Training Scheduler
- Can use fixed expert ordering, model-driven self-pacing, or learned teachers
- Should eventually represent the declared target distribution unless the protocol says otherwise
- Can introduce class, subgroup, source, and recency bias through its difficulty proxy
- Must beat random, reversed, or hard-example baselines under equal budgets
Common Use Cases
- Increasing sequence length or reasoning complexity during model training
- Scheduling simulation tasks as a reinforcement-learning policy improves
- Introducing reliable Pseudo-Labels before uncertain examples
- Balancing clean and noisy data over Fine-tuning stages
- Testing whether sample order changes convergence or final generalization
Example
Loading code...Frequently Asked Questions
Does Curriculum Learning always train from easy to hard?
Classic curricula do, but the broader design can schedule examples, tasks, environments, losses, or model capacity using a fixed or adaptive policy. Hard-example mining and anti-curricula deliberately emphasize difficult cases. The exact intervention and final target distribution should be stated instead of relying on the name.
How is Curriculum Learning different from Self-Paced Learning?
A conventional curriculum often uses difficulty supplied by an external teacher or fixed rule. Self-Paced Learning estimates inclusion or weight from the learner's current state and updates it during training. In practice hybrids are common, so report who scores difficulty, when it changes, and how pace is controlled.
How should difficulty be measured for Curriculum Learning?
Use a proxy tied to the intended skill, such as validated task complexity, sequence length, noise, baseline loss, disagreement, or environment success. Audit leakage and correlations with class, language, subgroup, and source. Compare multiple measures because low model loss can indicate redundancy or memorization rather than educational value.
Can Curriculum Learning hurt model quality?
Yes. A poor ranking can delay essential cases, amplify shortcuts, distort class prevalence, or cause forgetting. The schedule also changes the number and timing of optimization updates. Compare random and reversed schedules under equal examples, Tokens, updates, and stopping rules, then inspect final target-distribution and subgroup performance.
How is Curriculum Learning evaluated?
Freeze data, difficulty computation, optimizer, total examples or Tokens, update count, and final test set. Compare learning curves and final results for random, curriculum, reversed, and relevant hard-mining policies across seeds. Report task and slice metrics, compute, time to threshold, stability, and performance after full-distribution training.