What is LoRA?
LoRA (Low-Rank Adaptation) is a parameter-efficient adaptation method that keeps selected pretrained weight matrices frozen and learns additive updates through smaller low-rank matrices. It reduces the trainable update surface, but it does not guarantee a fixed memory saving, hardware fit, training speed, or quality relative to full fine-tuning.
Quick Facts
| Full Name | Low-Rank Adaptation |
|---|---|
| Created | 2021 by Microsoft Research |
| Specification | Official Specification |
How It Works
For a target weight W, LoRA computes an effective weight W + sBA, where A and B are trainable matrices with rank r and s is a configured scale. The trainable parameter count for that target is r times the sum of its input and output dimensions, so total adapter size depends on rank, target modules, layer count, dtype, bias settings, and any additional modules saved. Training memory also includes frozen base weights, activations, gradients, optimizer state, temporary buffers, and runtime overhead. An adapter is not a standalone model: it is bound to the exact base-model revision, tokenizer and chat template, module layout, LoRA configuration, and software stack used to create it. It may be served separately for task switching or merged into a compatible base, but both paths require quality, safety, latency, and numerical-parity checks.
Key Characteristics
- Represents a selected weight update as a scaled low-rank product while keeping the corresponding base weight frozen
- Adapter capacity and size depend on rank, target modules, layer shapes, dtype, bias, and modules_to_save
- Rank, alpha, initialization, learning rate, data, and target modules form a coupled experiment rather than independent defaults
- Peak training memory includes the frozen base, activations, trainable state, temporary buffers, and framework overhead
- Adapters can remain separate for controlled multi-adapter serving or be merged into a compatible base artifact
- Every adapter requires an immutable base, tokenizer, chat-template, configuration, dataset, and evaluation identity
Common Use Cases
- Adapting a language model's behavior or output format with a versioned supervised dataset
- Maintaining separately evaluated adapters for tenants, domains, languages, or tasks on one compatible base
- Running controlled rank and target-module experiments before considering a broader fine-tuning update
- Adapting diffusion, vision, speech, or multimodal linear layers when the implementation supports their module layout
- Shipping either a governed adapter package or a validated merged model with an explicit rollback artifact
Example
Loading code...Frequently Asked Questions
How does LoRA work?
LoRA leaves a selected base weight W frozen and learns two smaller matrices A and B. The layer uses W plus a scaled product BA during the forward pass. Only the adapter and any explicitly selected extra modules receive gradient updates. The method is a low-rank constraint on the update, not proof that every useful task update is low-rank.
How should I choose LoRA rank and alpha?
Treat rank and alpha as coupled experiment variables with target modules, initialization, learning rate, and data order. Run a bounded sweep on fixed train and validation revisions, then compare task quality, safety slices, peak memory, throughput, and adapter size. A larger rank increases capacity and cost but does not guarantee better quality.
Does LoRA always match full fine-tuning quality?
No. The original paper reported competitive results for specific models, tasks, baselines, and budgets. Transfer to another workload is not guaranteed. Compare the unchanged base, LoRA candidate, and an appropriate alternative on held-out task, safety, robustness, and regression slices before making a release decision.
What is the difference between LoRA and QLoRA?
LoRA describes the low-rank trainable update. QLoRA combines that update with a frozen base stored in 4-bit quantized form and higher-precision adapter computation. QLoRA can reduce base-weight memory, but total peak memory and quality still depend on the quantizer, sequence length, batch, activations, kernels, optimizer, and device.
Should I merge a LoRA adapter for deployment?
Merge when the serving runtime needs a standalone model and you can validate the resulting precision, tokenizer, generation behavior, latency, and rollback path. Keep the adapter separate when governed task switching or multi-adapter serving matters. In both cases, pin the exact base and adapter revisions; dynamic loading endpoints must be restricted to trusted operators.