What is LoRA?
LoRA (Low-Rank Adaptation) is a parameter-efficient adaptation method that keeps selected pretrained weight matrices frozen and learns additive updates through smaller low-rank matrices. It reduces the trainable update surface, but it does not guarantee a fixed memory saving, hardware fit, training speed, or quality relative to full fine-tuning.
Quick Facts
| Full Name | Low-Rank Adaptation |
|---|---|
| Created | 2021 by Microsoft Research |
| Specification | Official Specification |
How It Works
Compute the Low-Rank Update Explicitly
For a frozen matrix W with input width d_in and output width d_out, choose rank r and train A with shape r by d_in plus B with shape d_out by r. The layer computes W x + (alpha / r) B A x under the original scaling rule. The adapter contributes r(d_in + d_out) weights for that matrix before bias or extra modules. The [LoRA paper](https://arxiv.org/html/2106.09685) motivates this parameterization from low intrinsic rank, but its GPT-3 parameter and memory reductions belong to the paper's tested setup.
Treat Rank and Target Modules as Coupled Variables
Inspect the exact model module graph and verify that every configured target resolves to the intended linear layer. Sweep a bounded set of ranks, scaling policies and target-module sets on fixed data and Seeds; measure task quality, critical errors, training stability, peak memory, wall time and adapter size. Higher rank increases capacity and trainable state but need not improve held-out quality. Current [NeMo RL LoRA documentation](https://docs.nvidia.com/nemo/rl/nightly/guides/lora.html) also shows that names, initialization and backend constraints differ across implementations.
Validate Adapter and Merged Deployment Separately
An unmerged adapter supports controlled switching but adds Runtime compatibility, cache, authorization and concurrency concerns. A merged artifact removes the separate LoRA branch for supported configurations but loses adapter-specific operations and creates a new full checkpoint. Bind either form to base, Tokenizer, Template, method configuration and evaluation identities. Current [NVIDIA NIM PEFT guidance](https://docs.nvidia.com/nim/large-language-models/1.11.0/peft.html) requires adapters to match the served base; dynamic multi-adapter support is a product capability, not a universal LoRA property.
Key Characteristics
- Represents a selected weight update as a scaled low-rank product while keeping the corresponding base weight frozen
- Adapter capacity and size depend on rank, target modules, layer shapes, dtype, bias, and modules_to_save
- Rank, alpha, initialization, learning rate, data, and target modules form a coupled experiment rather than independent defaults
- Peak training memory includes the frozen base, activations, trainable state, temporary buffers, and framework overhead
- Adapters can remain separate for controlled multi-adapter serving or be merged into a compatible base artifact
- Every adapter requires an immutable base, tokenizer, chat-template, configuration, dataset, and evaluation identity
Common Use Cases
- Adapting a language model's behavior or output format with a versioned supervised dataset
- Maintaining separately evaluated adapters for tenants, domains, languages, or tasks on one compatible base
- Running controlled rank and target-module experiments before considering a broader fine-tuning update
- Adapting diffusion, vision, speech, or multimodal linear layers when the implementation supports their module layout
- Shipping either a governed adapter package or a validated merged model with an explicit rollback artifact
Example
Loading code...Frequently Asked Questions
How does LoRA work?
LoRA leaves a selected base weight W frozen and learns two smaller matrices A and B. The layer uses W plus a scaled product BA during the forward pass. Only the adapter and any explicitly selected extra modules receive gradient updates. The method is a low-rank constraint on the update, not proof that every useful task update is low-rank.
How should I choose LoRA rank and alpha?
Treat rank and alpha as coupled experiment variables with target modules, initialization, learning rate, and data order. Run a bounded sweep on fixed train and validation revisions, then compare task quality, safety slices, peak memory, throughput, and adapter size. A larger rank increases capacity and cost but does not guarantee better quality.
Does LoRA always match full fine-tuning quality?
No. The original paper reported competitive results for specific models, tasks, baselines, and budgets. Transfer to another workload is not guaranteed. Compare the unchanged base, LoRA candidate, and an appropriate alternative on held-out task, safety, robustness, and regression slices before making a release decision.
What is the difference between LoRA and QLoRA?
LoRA describes the low-rank trainable update. QLoRA combines that update with a frozen base stored in 4-bit quantized form and higher-precision adapter computation. QLoRA can reduce base-weight memory, but total peak memory and quality still depend on the quantizer, sequence length, batch, activations, kernels, optimizer, and device.
Should I merge a LoRA adapter for deployment?
Merge when the serving runtime needs a standalone model and you can validate the resulting precision, tokenizer, generation behavior, latency, and rollback path. Keep the adapter separate when governed task switching or multi-adapter serving matters. In both cases, pin the exact base and adapter revisions; dynamic loading endpoints must be restricted to trusted operators.