What is KTO?
KTO (Kahneman-Tversky Optimization) is an offline LLM alignment objective that learns from individual prompt-response examples labeled desirable or undesirable instead of requiring a chosen-rejected pair for each prompt.
Quick Facts
| Full Name | Kahneman-Tversky Optimization |
|---|---|
| Specification | Official Specification |
How It Works
Data contract
A KTO example represents one prompt-response pair plus a binary label. Desirable and undesirable records do not need to share a prompt, which distinguishes the format from DPO's chosen-rejected triples. Define what the label means for each task, retain source and policy version, remove duplicates and leakage, and split evaluation by prompt rather than by response so near-identical examples cannot cross train and test sets.
Objective and implicit reward
KTO belongs to the paper's family of human-aware losses. It forms an implicit reward from the log-ratio between the trainable policy and a reference policy, then evaluates a labeled output relative to a reference point estimated from KL divergence. Desirable outputs are rewarded for exceeding that point and undesirable outputs for falling below it. This is a differentiable training objective, not a literal model of every user's preferences.
Reference model and reference point
Standard KTO uses a frozen reference policy to measure how far the trainable policy moves. The KL-derived reference point makes utility relative rather than treating each log-probability in isolation. The original paper also studies variants without a reference model, but reports that standard KTO performs better. Treat the reference checkpoint, tokenizer, chat template, sequence truncation, and beta as versioned experiment inputs.
Class balance and label quality
Desirable and undesirable examples contribute through separate weights, so the effective class ratio changes the optimization. The paper reports robustness to substantial imbalance in its experiments, but this is not a guarantee for a new domain. Audit counts and sources, compare weighted and stratified runs, inspect per-class loss, and keep a held-out set with expert judgments. A click, regeneration, or ticket outcome is only a preference label after its confounders are understood.
Selection and evaluation
Choose KTO when reliable unpaired binary feedback is materially easier to obtain than matched comparisons. Compare it with SFT and a paired method such as DPO when both data forms are available. Evaluate blinded pairwise quality, task success, factuality, safety, style, refusal calibration, capability retention, subgroup behavior, and inference cost. The original 1B-to-30B results establish evidence for those setups, not universal superiority.
Key Characteristics
- Unpaired supervision: each prompt-response example carries its own desirable or undesirable label
- Prospect-theoretic value: gains and losses are measured relative to a KL-derived reference point
- Implicit reward: policy-to-reference log-probability ratios replace a separately trained reward model
- Class-specific weighting: desirable and undesirable contributions can be tuned independently
- Offline optimization: training uses a fixed dataset rather than an online PPO rollout loop
- Empirical choice: suitability depends on label semantics, data mix, model scale, and held-out task results
Common Use Cases
- Training from reviewed thumbs-up and thumbs-down feedback without reconstructing response pairs
- Using task-specific accepted or rejected outcomes whose causal meaning has been audited
- Aligning an assistant when only one candidate response was shown to each reviewer
- Comparing unpaired KTO data with paired DPO data under the same evaluation protocol
- Running an offline post-training stage without a separately trained reward model or online PPO loop
Example
Loading code...Frequently Asked Questions
How is KTO different from DPO?
DPO trains on a chosen and rejected response for the same prompt, so its loss compares a pair directly. KTO accepts individual prompt-response records with desirable or undesirable labels and compares each output's implicit reward with a reference point. The methods therefore require different data contracts and should be compared on the same held-out tasks.
Does KTO require a reward model or PPO?
No. Like direct preference methods, KTO optimizes the language model with a closed-form loss and does not require a separately trained scalar reward model or an online PPO rollout loop. Standard KTO does use a frozen reference policy and an estimated KL reference point.
Does KTO require a reference model?
The standard formulation uses one. The original paper evaluated a reference-free variant and found that it worked but underperformed standard KTO. If an implementation offers reference precomputation or adapter-based memory savings, verify that its objective and checkpoint handling still match the intended experiment.
Can KTO train on raw product thumbs-up and thumbs-down logs?
Only after auditing what those events mean. Position, latency, user segment, whether alternatives were visible, repeated sessions, policy changes, and accidental clicks can confound the label. Keep provenance, deduplicate, protect privacy, define exclusions, and create a human-reviewed holdout before training.
Is KTO better than DPO?
Not universally. The KTO paper reported matching or exceeding preference-based methods across its evaluated 1B-to-30B setups and strong imbalance results, while also stating that no human-aware loss is universally superior. Data availability, label quality, model, hyperparameters, and task-specific evaluation should determine the choice.