World Models vs LLMs: State, Actions, Planning, and AGI

Direct Answer

A world model learns how a task-relevant environment may evolve, often as a transition such as p(z_next | z_current, action). A large language model usually learns a distribution over tokens. These are modeling roles, not mutually exclusive neural architectures or proven routes to AGI. A Transformer can implement either role, and one system can combine both. The practical distinction appears in state, action conditioning, planning, uncertainty, and closed-loop evaluation.

Key Takeaways

• World model is a role, not one architecture. Recurrent state-space models, Transformers, diffusion models, and joint-embedding predictors can all model dynamics.
• Prediction is not planning. A model can forecast futures without selecting actions; a planner, actor, critic, or policy supplies the decision rule.
• Action conditioning matters for control. Passive video prediction may learn useful regularities, but it does not by itself establish counterfactual control.
• LLMs are not merely lookup tables. They can encode spatial and temporal structure, yet token fluency does not guarantee a sufficient control state.
• Closed-loop outcomes are the release test. Pixel realism and one-step loss are diagnostic metrics, not substitutes for task success, safety, calibration, and latency.
• AGI claims remain hypotheses. Current evidence supports specific capabilities under specific evaluations, not a settled universal architecture.

What a World Model Actually Is

A world model is a learned approximation of the dynamics that matter for an agent's task. It need not reproduce every detail of reality, and it need not be limited to the physical world. A game, software workflow, market simulator, or robot workspace can all be an environment if the model predicts how relevant state changes.

In a partially observed environment, the agent does not receive the true state s_t. It receives an observation o_t, such as an image or sensor vector, and maintains a belief or latent state z_t that summarizes relevant history. A minimal action-conditioned contract is:

state update:       z_t = encode(history, o_t)
dynamics:           z_(t+1) ~ p(z_(t+1) | z_t, a_t)
optional outputs:   reward, cost, termination, next observation
decision:           a_t = planner_or_policy(z_t, predicted futures)

This separation prevents four common category errors:

1. Observation is not state. A camera frame may omit velocity, hidden objects, intent, contact force, or prior events.
2. Dynamics is not policy. Predicting what follows an action does not decide which action is desirable.
3. Reward is not physics. A transition can be accurate while the objective is wrong, incomplete, or unsafe.
4. A simulator is not the real environment. It is an approximation whose errors may be exploited by the planner.

The 2018 World Models system made this modularity concrete with a visual encoder, a recurrent latent dynamics model, and a small controller. DreamerV3 uses a recurrent state-space model to predict representations, rewards, and continuation, then trains an actor and critic on imagined latent trajectories. Importantly, the deployed Dreamer actor selects actions without online lookahead; the world model helped train the policy, but it is not synonymous with the policy.

How World Models Work

A useful world-model system turns observations into a predictive state, rolls that state forward under actions, and connects predictions to a decision mechanism. Each component has a separate contract and failure surface.

State Estimation

The state estimator compresses current and historical observations into variables sufficient for the downstream task. Under partial observability, a single image is rarely enough. The representation may need memory for object permanence, velocity, prior actions, hidden goals, or uncertainty.

State sufficiency is task-dependent. A representation that predicts pixels well may omit a small obstacle critical to navigation. A compact control state may discard texture that matters for video generation. There is no universally correct latent state.

A Fixed-Lag Smoother is useful when the system can delay output and use a bounded window of future observations to revise recent states. The reported state must retain its original timestamp and declared lag; evaluating it as if it were a zero-delay online estimate leaks future information.

When dynamics switch among a known set of regimes, an Interacting Multiple Model estimator can maintain mode-conditioned filters and update their probabilities from predictive evidence. If only part of the latent state is conditionally tractable, a Rao-Blackwellized Particle Filter can sample the difficult part and integrate the rest analytically. Neither method creates an omitted regime or repairs an invalid state factorization.

Transition Prediction

The dynamics model predicts possible next states under an action. Deterministic models return one future; probabilistic models represent multiple futures or uncertainty. Action-free prediction can support representation learning, but decision-oriented control normally needs intervention data or another mechanism that distinguishes "what happened next" from "what would happen if the agent acted differently."

Reward, Cost, and Continuation

Many model-based reinforcement learning systems predict task reward and whether an episode continues. Safety-critical systems should also model constraints or costs rather than assuming reward captures them. A robot can reach a target while violating force, collision, or workspace constraints.

Planning or Policy Learning

Predictions become useful only when a decision procedure consumes them. Model-predictive control can score candidate action sequences and execute the first action before replanning. An actor can instead learn from imagined trajectories and run cheaply at inference time. Search, sampling, gradient-based optimization, and learned policies make different latency and robustness tradeoffs.

Three Major World Model Families

Current systems fall into overlapping families rather than one canonical design. The family determines what is predicted, how actions enter the model, and what evidence can establish usefulness.

Latent Dynamics

Ha and Schmidhuber's model and the Dreamer family compress observations and learn dynamics in latent space. This can make rollouts cheaper than predicting every pixel. It also creates an evaluation obligation: the latent state must preserve the variables needed for control, not merely reconstruction.

JEPA Predictive Representations

JEPA is a proposed predictive architecture, not the definition of every world model. Yann LeCun's position paper describes a broader autonomous-machine architecture with predictive models at multiple abstractions. V-JEPA 2 first learns action-free video representations, then post-trains a separate action-conditioned predictor for robot planning. That staging is important: observation-only pretraining and action-conditioned control are related but distinct capabilities.

Generative Simulators

Generative models predict observations directly or in a compressed visual space. OpenAI's Sora technical report presents qualitative simulation capabilities and also reports incorrect physics, state changes, and long-duration coherence. Google DeepMind describes Genie 3 as an interactive world model, while its published page also lists limits in action space, interaction duration, multi-agent interaction, and factual geographic accuracy. These are promising simulator results, not proof of a complete physical theory.

World Models vs LLMs

The clean comparison is between objectives and interfaces, not between two species of intelligence. "LLM" names a language-model family and training regime; "world model" names a predictive function inside a larger agent or simulator.

This table does not imply that LLMs have no internal model of the world. Research on spatial and temporal representations in language models found decodable structure in selected Llama-2 representations. That is evidence against the claim that language models contain only shallow phrase tables. It is not evidence that those representations form a calibrated, action-conditioned simulator suitable for arbitrary control.

The reverse caution also matters. A world model does not automatically understand causality, obey physics, plan well, or generalize outside its data. It can learn correlations in trajectories, omit hidden variables, and generate plausible but impossible futures. Calling a model a world model does not remove the need for explicit causal inference, interventions, or external validation.

Transformers illustrate why the categories overlap. IRIS uses an autoregressive Transformer inside an Atari world model. V-JEPA 2 uses Transformer components for predictive video representations and action-conditioned dynamics. Architecture alone therefore cannot decide whether a system is an LLM, a world model, both, or neither.

Video Realism Is Not Closed-Loop Competence

A convincing video proves that a model can generate a plausible sample under a prompt; it does not prove that the model predicts the consequences of controlled actions accurately enough for decisions. The missing evidence often appears under intervention.

Ask five questions before treating a video model as a decision model:

1. State: Does it preserve task-critical variables through occlusion and long horizons?
2. Action: Does changing only the action produce the correct change in the future?
3. Counterfactual: Can it distinguish futures for actions not taken in the logged trajectory?
4. Uncertainty: Does confidence fall when inputs or actions leave the training distribution?
5. Control: Does using the model improve closed-loop success without increasing safety violations?

The answer can differ by task. A model may be useful for game environment generation but unreliable for robot contact planning. The article on 3D generation and world-model engineering covers asset and spatial representation choices; this page focuses on predictive and decision contracts.

Failure Modes That Matter in Production

World models fail as systems, not only as predictors. A low average loss can coexist with dangerous decision errors.

Partial Observability and State Aliasing

Different real states can produce the same observation. If the latent state collapses them, the model predicts an average future that fits neither case. History, active sensing, and uncertainty-aware belief states can help, but their sufficiency must be tested.

Compounding Rollout Error

One-step errors feed back as inputs during imagined rollouts. Small drift can move the model into latent regions absent from training, making long-horizon predictions unreliable. Report error against horizon rather than one aggregate score.

Planner Exploitation

A planner searches specifically for high-scoring trajectories. It may discover unrealistic model states that were rare in ordinary validation. The original World Models work observed controllers exploiting imperfections of the learned environment. Adversarial planning tests and real-environment verification are therefore essential.

Confounding and Weak Action Coverage

Logged actions reflect an existing policy. If action choices correlate with hidden conditions, the model may confuse correlation with intervention. Diverse exploration, randomized actions where safe, and held-out counterfactual tests reduce this risk but do not establish universal causality.

Distribution Shift

New cameras, lighting, objects, operators, dynamics, or policies can invalidate the learned state and transition model. Monitor both input shift and outcome degradation, and define a fallback before deployment.

Objective and Safety Misspecification

Accurate dynamics cannot rescue an incorrect reward or missing constraint. Treat reward, safety cost, permission, and stop conditions as separate reviewed interfaces. For physical agents, the broader Embodied AI architecture and safety guide explains why the model is only one layer of the control stack.

How to Evaluate a World Model

Evaluation should connect prediction quality to the decisions the model is expected to support. A strong protocol uses held-out environments and trajectories, multiple seeds, baselines, uncertainty intervals, and explicit failure slices.

Pixel metrics can still be useful for generative simulators, but they should not dominate a control release. Conversely, a task-specific latent model may support excellent control while producing poor-looking reconstructions. Evaluation must follow the product contract.

A Go Release Gate

The following dependency-free Go program demonstrates a release gate over already-computed holdout metrics. It does not train or evaluate a model; it prevents a team from approving a candidate based only on an attractive demo or one-step score.

package main

import (
	"fmt"
	"math"
	"os"
)

type Metrics struct {
	HoldoutTrajectories    int
	OneStepStateError      float64
	LongHorizonStateError  float64
	ActionSensitivity      float64
	CounterfactualAccuracy float64
	ClosedLoopSuccess      float64
	BaselineSuccess        float64
	SafetyViolationRate    float64
	CalibrationError       float64
	P95LatencyMS           float64
}

type Limits struct {
	MinTrajectories        int
	MaxOneStepError        float64
	MaxLongHorizonError    float64
	MinActionSensitivity   float64
	MinCounterfactual      float64
	MinClosedLoopGain      float64
	MaxSafetyViolationRate float64
	MaxCalibrationError    float64
	MaxP95LatencyMS        float64
}

func validProbability(value float64) bool {
	return !math.IsNaN(value) && !math.IsInf(value, 0) &&
		value >= 0 && value <= 1
}

func evaluate(metrics Metrics, limits Limits) ([]string, error) {
	probabilities := []float64{
		metrics.OneStepStateError,
		metrics.LongHorizonStateError,
		metrics.ActionSensitivity,
		metrics.CounterfactualAccuracy,
		metrics.ClosedLoopSuccess,
		metrics.BaselineSuccess,
		metrics.SafetyViolationRate,
		metrics.CalibrationError,
	}
	for _, value := range probabilities {
		if !validProbability(value) {
			return nil, fmt.Errorf("invalid normalized metric: %v", value)
		}
	}
	if metrics.P95LatencyMS < 0 ||
		math.IsNaN(metrics.P95LatencyMS) ||
		math.IsInf(metrics.P95LatencyMS, 0) {
		return nil, fmt.Errorf("invalid latency: %v", metrics.P95LatencyMS)
	}

	var failures []string
	checks := []struct {
		ok      bool
		message string
	}{
		{metrics.HoldoutTrajectories >= limits.MinTrajectories, "insufficient holdout trajectories"},
		{metrics.OneStepStateError <= limits.MaxOneStepError, "one-step state error exceeds limit"},
		{metrics.LongHorizonStateError <= limits.MaxLongHorizonError, "long-horizon state error exceeds limit"},
		{metrics.ActionSensitivity >= limits.MinActionSensitivity, "action sensitivity is too low"},
		{metrics.CounterfactualAccuracy >= limits.MinCounterfactual, "counterfactual accuracy is too low"},
		{metrics.ClosedLoopSuccess-metrics.BaselineSuccess >= limits.MinClosedLoopGain, "closed-loop gain is too small"},
		{metrics.SafetyViolationRate <= limits.MaxSafetyViolationRate, "safety violation rate exceeds limit"},
		{metrics.CalibrationError <= limits.MaxCalibrationError, "calibration error exceeds limit"},
		{metrics.P95LatencyMS <= limits.MaxP95LatencyMS, "P95 latency exceeds control budget"},
	}
	for _, check := range checks {
		if !check.ok {
			failures = append(failures, check.message)
		}
	}
	return failures, nil
}

func main() {
	metrics := Metrics{
		HoldoutTrajectories:    1800,
		OneStepStateError:      0.07,
		LongHorizonStateError:  0.18,
		ActionSensitivity:      0.82,
		CounterfactualAccuracy: 0.79,
		ClosedLoopSuccess:      0.81,
		BaselineSuccess:        0.74,
		SafetyViolationRate:    0.006,
		CalibrationError:       0.03,
		P95LatencyMS:           38,
	}
	limits := Limits{
		MinTrajectories:        1000,
		MaxOneStepError:        0.10,
		MaxLongHorizonError:    0.20,
		MinActionSensitivity:   0.75,
		MinCounterfactual:      0.75,
		MinClosedLoopGain:      0.03,
		MaxSafetyViolationRate: 0.01,
		MaxCalibrationError:    0.05,
		MaxP95LatencyMS:        50,
	}

	failures, err := evaluate(metrics, limits)
	if err != nil {
		fmt.Fprintln(os.Stderr, err)
		os.Exit(2)
	}
	if len(failures) > 0 {
		for _, failure := range failures {
			fmt.Println("BLOCK:", failure)
		}
		os.Exit(1)
	}
	fmt.Println("release=approved")
}

Expected output:

release=approved

The thresholds are illustrative, not universal. A warehouse robot, a game simulator, and a recommendation policy have different consequences and latency budgets. Define thresholds before looking at the candidate result, compare against a no-world-model baseline, and preserve failed slices instead of averaging them away.

Choosing an Architecture

Use a world model when the product needs to compare actions before taking them, learn from expensive interaction, simulate task-relevant futures, or train a policy from imagined experience. Do not add one merely because the term is fashionable.

1. Write the environment contract. Name observations, hidden state assumptions, actions, prediction horizon, objectives, constraints, and control deadline.
2. Build a baseline. Compare with a reactive policy, model-free method, fixed simulator, or LLM plus tools.
3. Choose the prediction target. Use latent state for efficient control, explicit geometry where spatial constraints matter, and generated observations when humans or downstream perception must inspect the rollout.
4. Separate model and decision errors. Replay the same states with the ground-truth simulator where possible to identify whether dynamics, reward, planner, or controller caused failure.
5. Design fallback behavior. Out-of-distribution detection, uncertainty thresholds, conservative actions, and human intervention belong in the runtime architecture.
6. Release on closed-loop evidence. Require improvement over the baseline under the same environment distribution and safety constraints.

This approach also clarifies hybrid systems. An LLM may parse a user goal, retrieve procedural knowledge, or propose high-level subgoals. A world model may predict action consequences. A planner may choose among them. A deterministic safety layer may reject unsafe commands. Those modules can share a Transformer backbone or remain separate services; responsibility should still remain explicit.

Common Misconceptions

World Models Learn Causality Automatically

They learn predictive regularities from the available data and objective. Causal claims require interventions, assumptions, or experimental design beyond naming the model.

LLMs Have No World Knowledge

LLMs can encode substantial factual, spatial, temporal, and procedural structure. Their weakness for a control task must be demonstrated on that task, not inferred from the next-token objective alone.

JEPA Is the Canonical World Model

JEPA is one influential predictive-representation approach. Dreamer-style latent dynamics, Transformer simulators, explicit state-space models, and generative environment models are also legitimate designs.

Better Video Means Better Planning

Perceptual quality may help, but a planner needs correct action effects, stable task state, calibrated uncertainty, and usable latency. These properties require separate tests.

A World Model Is an AGI Roadmap

A world model is a component or capability. Whether it is necessary or sufficient for AGI remains unsettled, and "AGI" itself lacks a single accepted operational test.

Frequently Asked Questions

What is a world model in AI

A world model is a learned predictive model of environment dynamics. It maps current observations or latent state, history, and sometimes an action to possible future states or observations. It may also predict reward, safety cost, or termination. A separate planner or policy uses those predictions to choose behavior.

How do world models differ from LLMs

LLMs usually optimize token prediction, while decision-oriented world models optimize predictions of task-relevant transitions under actions. The boundary is functional rather than architectural: both may use Transformers, both may be multimodal, and one model may serve both roles if its interfaces and evaluations support them.

Can an LLM contain a world model

It can contain representations that behave like parts of a world model, and it can be trained on trajectories whose tokens include states and actions. That is not enough for deployment. Test whether the representation retains hidden state, responds correctly to interventions, remains calibrated over long rollouts, and improves closed-loop decisions.

Is Sora or another video generator a world model

It can be described as a general-purpose simulator when it predicts coherent environment evolution, but the label does not prove decision competence. OpenAI's Sora report explicitly documents physical and long-horizon failures. For control, require action-conditioned, counterfactual, closed-loop, safety, and latency evidence.

Will world models replace LLMs

Replacement is the wrong default assumption. Language understanding, environment prediction, planning, policy execution, and safety are separable responsibilities. A system may compose specialized modules or train a shared model, but each responsibility still needs its own evidence and failure controls.

Summary

World Models vs LLMs is not a contest between one model that "understands" and another that merely predicts. LLMs usually model token distributions; world models model task-relevant environment evolution, often under actions. Both can use the same neural architectures, overlap in learned representations, and contribute to one agent.

The engineering distinction is measurable. Define state, action, horizon, reward, safety, uncertainty, planner, and runtime contracts. Then test one-step prediction, long-horizon drift, interventions, counterfactuals, closed-loop task outcomes, safety, and latency against a baseline. Those results justify a world model's role. Visual realism or an AGI narrative does not.

Related Resources

• What Is Embodied AI
• 3D Generation and World Model Engineering
• Reasoning Models and Test-Time Compute
• Large Language Model Glossary
• Artificial General Intelligence Glossary
• Reinforcement Learning Glossary

Sources and Further Reading

• David Ha and Jürgen Schmidhuber, World Models
• Danijar Hafner et al., Mastering Diverse Control Tasks through World Models
• Mahmoud Assran et al., V-JEPA 2: Self-Supervised Video Models Enable Understanding, Prediction and Planning
• Yann LeCun, A Path Towards Autonomous Machine Intelligence
• OpenAI, Video Generation Models as World Simulators
• Google DeepMind, Genie 3: A New Frontier for World Models
• Wes Gurnee and Max Tegmark, Language Models Represent Space and Time
• Xinqing Li et al., A Comprehensive Survey on World Models for Embodied AI