What is Preference Data?
Updated on
Preference Data is data that records judgments about candidate model behaviors as relative choices, rankings, ratings, critiques, or desirable and undesirable labels, together with the context needed to interpret those judgments.
How It Works
Formats answer different questions
A paired record stores a prompt plus chosen and rejected completions. A listwise record ranks several candidates. A pointwise record rates one candidate against a rubric, while an unpaired record marks one completion as desirable or undesirable. Critiques and edits preserve richer reasons but require an explicit conversion rule before a pairwise trainer can use them. These formats are not interchangeable: the loss function and trainer must match the meaning of the label.
Candidate generation is part of the label
A comparison only distinguishes the candidates that were shown. Easy negatives may teach superficial separation, while candidates from one stronger model can leak generator identity. Record candidate model and checkpoint, decoding configuration, tool context, truncation, and sampling time. Randomize or counterbalance display order, hide model identity where possible, retain ties and invalid cases, and audit correlations with length, formatting, refusal style, and generator.
Rubrics and reviewer disagreement define the target
A usable rubric states which dimensions matter, how conflicts are prioritized, and when reviewers may tie, skip, or mark both answers invalid. Store raw judgments, rubric version, reviewer cohort, confidence, and optional justification before deriving a majority label. Disagreement can reflect task underspecification, style preferences, or legitimate value differences rather than annotation noise, so collapsing it too early destroys information needed for analysis and pluralistic evaluation.
Split by information lineage, not by rows
All pairs, ratings, turns, and paraphrases derived from one prompt or conversation should stay in the same split. Also group near duplicates, templates, source documents, users, and repeated tasks when they can leak answer structure. A row-random split can place one comparison from a ranking in training and another from the same prompt in validation, producing an optimistic result. Keep separate slices for new policy checkpoints, domains, languages, safety cases, and collection periods.
Synthetic feedback needs an independent control
AI feedback can expand coverage and attach critiques or per-axis scores, but it also inherits the judge model's position, verbosity, style, self-preference, and knowledge errors. Version the judge, prompt template, sampling settings, parser, and filtering rules. Audit a stratified sample with qualified humans and keep a human-only holdout when the target is human preference. Confirm output licenses and provider terms before using generated data for training.
Release a dataset contract, not just JSONL
A release gate should validate schema, stable identifiers, candidate distinctness, allowed labels, prompt-group split integrity, duplicates, missing context, PII handling, consent, and licenses. Report label and slice distributions, reviewer agreement without treating it as truth, position and length correlations, generator balance, invalid-rate, and downstream held-out behavior. Publish a dataset card, lineage, code and rubric versions, immutable artifact hash, and known limitations so a model result can be traced to the exact feedback policy.
Key Characteristics
- Represents paired choices, listwise rankings, pointwise ratings, critiques, or unpaired labels
- Conditions every judgment on a prompt, candidate set, rubric, reviewer population, and collection policy
- Requires prompt- or conversation-grouped splits to prevent correlated examples from crossing partitions
- Preserves raw votes, ties, skips, invalid cases, confidence, and rationales before aggregation
- Exposes position, length, generator, language, domain, and policy-distribution biases to measurement
- Needs versioned provenance, licensing, privacy controls, and downstream held-out evaluation
Common Use Cases
- Training and auditing pairwise or regression-style reward models
- Optimizing policies with DPO, KTO, ORPO, or other preference objectives
- Collecting specialist judgments for code, medicine, law, safety, or multilingual behavior
- Building offline evaluation sets for product quality, refusals, tone, and factual support
- Refreshing feedback with outputs from newer policy checkpoints to measure distribution shift
Example
Loading code...Frequently Asked Questions
How is preference data different from SFT data?
SFT usually trains on a selected target response and increases the likelihood of its tokens. Preference learning represents a judgment about one or more candidates under a rubric. Pairwise DPO data compares chosen and rejected answers; reward modeling may instead use rankings or ratings. A preferred response can also contain flaws, so preference is not the same as a verified reference answer.
Do preference datasets always need chosen and rejected answers?
No. Pairwise trainers commonly use prompt, chosen, and rejected fields, but listwise rankings, scalar ratings, critiques, edits, and unpaired desirable or undesirable labels encode different feedback. KTO-style objectives can use unpaired binary feedback. Choose the collection format from the intended loss and preserve the original signal before converting it.
Should annotator disagreement be removed?
Not automatically. Disagreement may expose an unclear task, conflicting rubric priorities, subjective style, or different legitimate user preferences. Keep raw votes, ties, confidence, reviewer cohort, and rationales. Investigate the cause before excluding a sample or deriving a majority label, and evaluate important slices against the intended user population.
Can an LLM generate reliable synthetic preference data?
An LLM judge can scale coverage, but its labels inherit model knowledge limits and position, verbosity, style, and self-preference biases. Version the judge and prompt, randomize candidate order, test counterfactuals, audit representative samples with qualified humans, and retain a human-only holdout when human preference is the target.
How should preference data be split for training and evaluation?
Group every record derived from the same prompt, conversation, source document, or near-duplicate template into one split. Otherwise correlated pairs can appear in both training and validation. Add held-out policy checkpoints, domains, languages, safety cases, and newer collection periods to test whether the learned preference transfers beyond the original candidate distribution.