What is PEFT?
PEFT (Parameter-Efficient Fine-Tuning) is a family of methods that adapts a pretrained model by training selected existing parameters or a comparatively small set of added parameters while most base weights remain frozen. Parameter efficiency does not by itself guarantee a fixed memory reduction or the quality of full fine-tuning.
Quick Facts
| Full Name | Parameter-Efficient Fine-Tuning |
|---|---|
| Created | Various techniques from 2019-2023, unified by Hugging Face |
| Specification | Official Specification |
How It Works
Choose a Method by What It Changes
Classify the candidate as additive, selective, reparameterized or hybrid, then verify that the target architecture and training backend implement that method. Bottleneck modules add sequential layers, soft prompts consume learned context positions, selective methods update chosen existing state, and LoRA factorizes weight updates. The independent [PEFT survey](https://arxiv.org/html/2303.15647) found substantial method, implementation and hyperparameter differences; a small trainable fraction alone does not identify the best method.
Measure the Whole Resource Envelope
Record trainable and total parameters, but also measure frozen base weights, activations, gradients, optimizer state, temporary kernels, communication buffers, host memory, fragmentation, wall time and energy on the intended sequence and Batch distribution. Hugging Face's [PEFT documentation](https://huggingface.co/docs/peft/main/en/index) covers adaptation methods across model families, while actual support and efficiency depend on the selected library, backend and model. Compare candidates under one dataset and evaluation contract.
Release a Base-Dependent Artifact Safely
A PEFT artifact normally contains only method-specific state and configuration. Bind it to immutable base weights, Tokenizer or Processor, Template, framework lockfile, data revisions and evaluation report. The official [PEFT checkpoint format](https://huggingface.co/docs/peft/main/en/developer_guides/checkpoint) requires the original base and notes that merge support varies by method and configuration. Validate the exact base-plus-adapter or merged serving form, restrict dynamic loading to trusted artifacts, and retain a compatible rollback target.
Key Characteristics
- Covers additive, selective, reparameterized, and hybrid adaptation methods rather than one fixed architecture
- Keeps most base parameters frozen while training a method-specific subset or newly introduced state
- Reduces trainable and checkpoint state, but peak memory depends on weights, activations, sequence shape, precision, optimizer, and runtime
- Produces behavior that must be evaluated by task slice, regression slice, safety policy, and repeated seed
- Usually creates a base-dependent checkpoint whose compatibility includes model, tokenizer, template, framework, and method configuration
- Has method-specific serving behavior: some updates can be merged, while modules and soft prompts may require runtime composition
Common Use Cases
- Comparing efficient adaptation methods under a fixed workload, device envelope, and evaluation protocol
- Maintaining independently versioned domain or task variants on an immutable shared base
- Running controlled adaptation experiments without storing a full model copy for every candidate
- Adapting language, vision, speech, or diffusion models when the framework supports the selected method
- Serving approved model variants with explicit compatibility checks, authorization, observability, and rollback
Example
Loading code...Frequently Asked Questions
What is the difference between PEFT and full fine-tuning?
Full fine-tuning updates all or most base-model weights. PEFT trains selected existing parameters or added state while most base weights remain frozen. That usually reduces gradient, optimizer, and checkpoint state, but the actual quality, wall time, and peak-memory difference depends on the model, method, precision, sequence shape, optimizer, runtime, and device topology.
How should a PEFT method be selected?
Start with a workload contract: target behavior, base architecture, allowed training and serving backends, device envelope, latency, storage, and release constraints. Compare viable methods under the same data splits and evaluation protocol. LoRA is broadly supported, bottleneck modules add sequential computation, and soft-prompt methods change attention inputs; none is universally best.
Does training fewer parameters reduce peak memory by the same ratio?
No. Trainable parameters determine only part of training memory. Base weights, quantization metadata, activations, gradients, optimizer state, temporary kernels, communication buffers, runtime allocations, fragmentation, and headroom all contribute. Measure peak device and host memory on the intended sequence length, batch shape, precision, checkpointing policy, and hardware.
Does PEFT always match full fine-tuning quality?
No. Paper results are tied to their models, tasks, data, method configurations, and evaluation protocols. A PEFT candidate can outperform, match, or trail full fine-tuning, and it can improve one slice while regressing another. Compare candidates with repeated seeds, held-out task slices, general-capability regressions, safety checks, and serving-artifact tests.
Is a PEFT checkpoint a standalone model?
Usually not. A PEFT checkpoint commonly stores method-specific parameters and configuration and still requires the compatible base model. Treat the base revision and hash, tokenizer, chat template, framework versions, method configuration, adapter revision, evaluation report, and serving composition as one identity. Merging is method- and configuration-specific and creates a new artifact that needs validation.