What is Reinforcement Learning?
Reinforcement Learning (RL) is a machine-learning paradigm in which an agent learns a policy for sequential decisions from interaction or logged experience, with the objective of maximizing expected cumulative reward under an environment and horizon.
Quick Facts
| Created | 1980s, formalized by Richard Sutton and Andrew Barto |
|---|---|
| Specification | Official Specification |
How It Works
The agent-environment interface defines the problem
At step t, an agent receives an observation, chooses an action from a valid action space, and the environment returns a reward and next observation. A trajectory records this sequence. Return aggregates future rewards, often with a discount factor, while a policy maps observations to actions or action probabilities. State and observation differ when the agent cannot see all information needed to make the process Markov.
Value, policy, and model are separate choices
Value-based methods estimate expected return and derive actions from values; policy-gradient methods optimize a parameterized policy directly; actor-critic methods learn both. Model-based RL uses known or learned transition dynamics for planning, while model-free RL learns values or policies without an explicit environment model. On-policy and off-policy describe which behavior distribution supplies updates, not whether an environment model exists.
Experience can be online or logged
Online RL changes the data distribution while learning, so exploration affects both evidence and outcomes. Offline RL trains from a fixed behavior dataset and cannot safely assume support for unseen actions. In either case, coverage, delayed feedback, partial observability, non-stationarity, simulator mismatch, and reward misspecification can make a high estimated return fail in deployment.
Evaluation must test behavior, not only reward
Compare against scripted, supervised, contextual-bandit, or control baselines when they fit the decision. Report return distributions, success and violation rates, sample efficiency, stability across seeds, sensitivity to reward and environment changes, latency, and recovery from unsafe or terminal states. Use off-policy estimators cautiously and require constrained tests or human approval before real-world exploration.
Key Characteristics
- Optimizes a policy for expected cumulative reward over sequential decisions
- Represents experience as ordered observations, actions, rewards, and transitions
- Must address delayed consequences and credit assignment
- Balances information gathering with use of current policy knowledge
- Can learn online from interaction or offline from logged behavior
- Is sensitive to reward design, coverage, environment shift, and unsafe exploration
Common Use Cases
- Learning game or simulator policies with reproducible environment dynamics
- Optimizing robot control after simulation and safety-constrained validation
- Scheduling or resource allocation when actions affect future capacity
- Learning from logged decisions with explicit behavior-policy coverage checks
- Optimizing language-model policies from modeled or verifiable feedback
Example
Loading code...Frequently Asked Questions
What is the difference between reinforcement learning and supervised learning?
Supervised learning usually fits a target for each example from a fixed dataset. Reinforcement learning evaluates actions through rewards and later consequences, and the policy may influence which data is observed next. RL still has supervision in the broad sense: reward functions, demonstrations, preferences, simulators, and logged behavior all encode assumptions.
What is the difference between reward and return?
Reward is the scalar feedback from one transition. Return aggregates rewards over a future horizon, often with discounting. Optimizing immediate reward can conflict with long-term return, and maximizing either signal can still produce undesirable behavior when the reward omits safety, fairness, or operational constraints.
How do model-based and model-free reinforcement learning differ?
Model-based methods use known or learned environment dynamics to predict outcomes or plan. Model-free methods learn a value function, policy, or both without an explicit transition model. Model-free does not mean data-free, and model-based does not mean the model is correct; both require evaluation under model error and distribution shift.
What is offline reinforcement learning?
Offline RL learns only from a fixed dataset collected by another behavior policy. It avoids live exploration during training, but actions outside the dataset's support can receive unreliable value estimates. A safe comparison records behavior-policy coverage, conservative baselines, off-policy evaluation assumptions, and deployment constraints.
How is RLHF related to reinforcement learning?
Some RLHF pipelines fit a reward model from preference data and optimize a language-model policy with an RL algorithm such as PPO. Other alignment methods optimize preferences without an explicit RL loop. Human feedback, a learned reward, and policy optimization are separate components, and none alone guarantees helpful or safe behavior.