TL;DR
OpenAI o1 and DeepSeek R1 popularized a different allocation of inference compute for difficult tasks. Public materials support discussing reinforcement learning, verifiers, distillation, and test-time compute; they do not establish a universal internal Chain of Thought or guarantee that extra deliberation solves every hard problem.
📋 Table of Contents
- The Paradigm Shift: From System 1 to System 2 Thinking
- How Reasoning Models Work: The Core Mechanics
- OpenAI o1 Architecture Insights
- DeepSeek R1: Open-Source Reasoning Breakthrough
- The Power of Test-Time Compute
- FAQ
- Summary
✨ Key Takeaways
- Extra Inference Budget: Reasoning systems can allocate more computation to selected tasks, with measurable cost and latency trade-offs.
- Limited Observability: A model's visible explanation is not necessarily its internal computation; evaluate outputs and verifiers instead of treating text as a trace.
- Training Recipes Matter: R1's public report describes multiple stages; avoid reducing it to “pure RL only.”
- Conditional Scaling: More inference compute can improve some tasks, but gains depend on task, evaluator, sampling, and stopping policy.
The Paradigm Shift: From System 1 to System 2 Thinking
In human psychology, Daniel Kahneman famously described two modes of thought:
- System 1: Fast, instinctive, and automatic (e.g., answering "2+2").
- System 2: Slow, deliberate, and logical (e.g., solving "17×24").
Standard autoregressive models and reasoning systems can both use iterative prompting, tools, sampling, or verification. The System 1/System 2 analogy is a teaching model, not a description of an exact neural mechanism; a standard model can sometimes solve hard tasks and a reasoning model can still fail.
Reasoning models are better described as systems trained or configured to allocate additional computation to selected problems. Whether they backtrack, verify, or search internally depends on the implementation and cannot be inferred from a fluent answer alone.
📝 Glossary: Chain of Thought (CoT) — A prompting technique where a model is asked to explain its reasoning step-by-step before providing an answer.
How Reasoning Models Work: The Core Mechanics
Reasoning systems may use post-training recipes beyond a conventional Supervised Fine-Tuning -> RLHF pipeline, but the exact recipe is model-specific and should be sourced from the model report.
1. The Hidden Chain of Thought (CoT)
When you ask a reasoning model a question, the service may spend additional computation before returning final text. The API may hide, summarize, or expose parts of that process, and the returned explanation should not be treated as a faithful internal trace.
- It breaks the problem down.
- It writes a hypothesis.
- It tests the hypothesis internally.
- A displayed phrase such as
Wait, this approach is flawedis an output pattern, not proof that the runtime performed a verified backtrack.
Hidden computation may contribute to a response, but its size, representation, and reliability are provider-specific. Applications should rely on structured outputs, external verifiers, and outcome checks rather than hidden traces.
The following diagram is a conceptual workflow, not a claim about the private implementation of o1 or R1:
2. Large-Scale Reinforcement Learning (RL)
Training a reasoning policy can combine supervised data, reinforcement learning, verifiers, distillation, and other techniques. Human-written solutions can still be useful even when they are not raw cognitive traces.
Instead, researchers use Reinforcement Learning. They give the model a hard math problem and an automated verifier (a Python script that checks if the final answer is right).
- If the model gets it right, it gets a reward.
- If it gets it wrong, it gets penalized. The reward design shapes which behaviors are reinforced. A phrase such as “let me check” is not itself a reliable indicator of correctness; the verifier and held-out evaluation determine whether the behavior helped.
OpenAI o1 Architecture Insights
While OpenAI has kept the exact architecture of the o1 series closed, technical papers and API behaviors reveal several key innovations:
- Inference-time trade-offs: Public evaluations show that additional inference budget can improve selected tasks under a stated protocol; this is not a universal monotonic law.
- Unpublished internals: Process reward models, hidden traces, and search strategies should be labeled hypotheses unless the provider documents them.
- Safety is separate from reasoning: A reasoning model still needs policy enforcement, adversarial testing, and tool-boundary controls; do not infer jailbreak resistance from model class.
DeepSeek R1: Open-Source Reasoning Breakthrough
DeepSeek R1 made a public training report and model artifacts available for study. Comparisons with OpenAI o1 depend on checkpoint, prompt, sampling, tools, date, and benchmark protocol; “matches or beats” is not a universal claim.
The DeepSeek R1 Pipeline:
- DeepSeek-V3 Base: It starts with a highly efficient Mixture of Experts (MoE) base model.
- R1-Zero exploration: The report describes a reinforcement-learning stage without the same conventional reasoning warm start, alongside observed behaviors such as longer traces. “Aha moment” is an informal description, not a mechanistic finding.
- Cold start and distillation: The reported R1 recipe includes supervised data, reinforcement learning, and distillation. Reproduce claims against the report and exact released artifacts rather than assuming every deployment uses the same path.
# Pseudo-code representing R1's RL Reward Function
def calculate_reward(model_output, ground_truth):
reward = 0
# 1. Accuracy Reward (Outcome)
if extract_final_answer(model_output) == ground_truth:
reward += 10.0
# 2. Format reward (illustrative only; not the published reward)
if "<think>" in model_output and "</think>" in model_output:
reward += 1.0
return reward
The Power of Test-Time Compute
Training compute and inference compute are separate cost and capacity decisions; the exact hardware and duration of a project should not be inferred from a slogan.
Reasoning models expose or internally allocate an additional inference budget. A higher budget can mean longer generation, multiple candidates, verification, or search; the product may expose only a few effort levels rather than an arbitrary “think for 30 minutes” control.
⚠️ Common Mistakes:
- Using Reasoning Models for every task → Correction: compare them with a standard model on representative translation, summarization, coding, and reasoning sets; select on quality, latency, cost, and error recovery rather than a fixed task blacklist.
FAQ
Q1: Can I see the hidden Chain of Thought in OpenAI o1?
Provider interfaces and releases differ. A visible <think> block, when present, is still generated text and should not be treated as a verified or complete internal trace; check the current API and model documentation.
Q2: Why do Reasoning Models sometimes output "Wait, let me rethink" in the middle of a sentence?
It may reflect a learned discourse pattern, a product rendering choice, or genuine intermediate computation. The phrase alone cannot establish why the model produced it or whether the correction is correct.
Q3: Are Reasoning Models replacing Agents?
Not entirely. An agent runtime can use a reasoning model as its decision component, while tools and state remain outside the model. Whether a provider performs an internal loop is implementation-specific; real-world actions still require explicit tool permissions and orchestration.
Summary
Reasoning models show the value of allocating inference compute to difficult tasks, but they remain probabilistic systems. Production use requires fixed evaluation protocols, external verification, bounded budgets, and policy controls; a displayed chain of thought is not a substitute for evidence.