What is LoRA Rank?
Updated on
LoRA Rank is the inner dimension r of the two trainable matrices used to represent a LoRA weight update. It bounds the update's matrix rank and controls parameter capacity, but it does not directly predict task quality or provide a universal default.
Quick Facts
| Specification | Official Specification |
|---|
How It Works
Calculate Rank Cost From Every Target Shape
For A shaped r by d_in and B shaped d_out by r, one target contributes r(d_in + d_out) trainable weights before Bias or modules_to_save. Sum every matched layer rather than multiplying a model-wide parameter count by a percentage. The [LoRA paper](https://arxiv.org/html/2106.09685) found low-rank structure in its tested updates, but its effective ranks and resource reductions are experimental results, not universal defaults.
Treat Rank, Scaling, and Placement as One Design
Rank changes capacity while alpha and the scaling rule change update magnitude; Target Module coverage decides where that capacity exists. Current [PEFT LoRA documentation](https://huggingface.co/docs/peft/main/en/package_reference/lora) supports per-layer rank_pattern and alpha_pattern, multiple initializations, and direct expert-parameter targets. Those capabilities are implementation-specific, and unmatched module patterns can silently invalidate the intended budget.
Select the Smallest Candidate That Clears Gates
Run a bounded Rank Sweep with immutable Base, data splits, Tokenizer, Template, target set, initialization, optimizer, step budget and Seeds. Compare task and regression slices, variance, peak device and host memory, Wall Time, Adapter bytes and serving latency. Adaptive allocation such as [AdaLoRA](https://arxiv.org/html/2303.10512) shows that uniform rank need not be optimal, but its reported gains remain tied to its models and protocol.
Key Characteristics
- Sets the shared inner dimension of the A and B update matrices for each selected weight
- Adds r times the sum of input and output dimensions parameters per target weight
- Changes capacity and resource use but does not monotonically determine held-out quality
- Interacts with target modules, alpha scaling, initialization, optimizer, data, and training budget
- Affects Adapter artifact size and the rank limits or cache capacity of some serving runtimes
- Must be selected with a controlled sweep on immutable train, validation, and test revisions
Common Use Cases
- Comparing Adapter capacity for a versioned instruction-following or classification task
- Budgeting trainable parameters, optimizer state, artifact storage, and serving cache
- Testing whether broader target-module coverage is more useful than increasing rank
- Defining the maximum supported rank for governed multi-Adapter serving
- Diagnosing capacity limits only after checking labels, loss masks, leakage, and data quality
Example
Loading code...Frequently Asked Questions
What LoRA rank should I use?
There is no universal rank. Choose a bounded candidate set that fits the resource envelope, keep the base, targets, data, scaling policy, optimizer, and evaluation fixed, then compare repeated runs on held-out task and regression slices. Select the smallest rank that clears the declared release gates rather than copying a popular value.
Does higher rank always improve quality?
No. Higher rank expands the update parameterization, but optimization, data quality, target modules, and evaluation determine whether that capacity is useful. It can produce no measurable gain, increase variance or overfitting, and consume more memory, storage, and serving cache.
How many parameters does LoRA rank add?
For one target weight with input dimension d_in and output dimension d_out, rank r adds r multiplied by d_in plus d_out parameters for A and B. Sum this value across all selected weights and layers, then add any trained bias, embedding, head, or modules_to_save parameters.
How does LoRA rank affect deployment?
Rank changes Adapter bytes and may change runtime compute, cache allocation, maximum concurrent Adapters, and engine compatibility. Serving systems can impose a maximum rank or reserve cache for that maximum, so training configuration and serving capacity must be planned together.
Should I increase rank when LoRA quality is poor?
Not first. Check labels, completion or assistant loss masks, truncation, chat templates, duplicate leakage, target-module names, and the unchanged-base baseline. Increase rank only as a controlled experiment after these failures are excluded; the problem may instead require better data or a different objective.