What is LoRA Rank?
LoRA Rank is the inner dimension r of the two trainable matrices used to represent a LoRA weight update. It bounds the update's matrix rank and controls parameter capacity, but it does not directly predict task quality or provide a universal default.
Quick Facts
| Specification | Official Specification |
|---|
How It Works
For a target weight with input dimension d_in and output dimension d_out, LoRA uses A with shape r by d_in and B with shape d_out by r. That target adds r times the sum of d_in and d_out trainable parameters, before biases or extra saved modules. Total Adapter size is the sum across every selected target and layer. Raising rank increases this representational budget and may raise optimizer, gradient, storage, cache, and compute costs, but it may not improve held-out quality. Rank also interacts with target modules, initialization, learning rate, data, and the scaling policy: original LoRA uses alpha divided by r, while variants may use a different rule. A valid rank comparison fixes these factors or declares how they change, then measures quality and resource results by slice.
Key Characteristics
- Sets the shared inner dimension of the A and B update matrices for each selected weight
- Adds r times the sum of input and output dimensions parameters per target weight
- Changes capacity and resource use but does not monotonically determine held-out quality
- Interacts with target modules, alpha scaling, initialization, optimizer, data, and training budget
- Affects Adapter artifact size and the rank limits or cache capacity of some serving runtimes
- Must be selected with a controlled sweep on immutable train, validation, and test revisions
Common Use Cases
- Comparing Adapter capacity for a versioned instruction-following or classification task
- Budgeting trainable parameters, optimizer state, artifact storage, and serving cache
- Testing whether broader target-module coverage is more useful than increasing rank
- Defining the maximum supported rank for governed multi-Adapter serving
- Diagnosing capacity limits only after checking labels, loss masks, leakage, and data quality
Example
Loading code...Frequently Asked Questions
What LoRA rank should I use?
There is no universal rank. Choose a bounded candidate set that fits the resource envelope, keep the base, targets, data, scaling policy, optimizer, and evaluation fixed, then compare repeated runs on held-out task and regression slices. Select the smallest rank that clears the declared release gates rather than copying a popular value.
Does higher rank always improve quality?
No. Higher rank expands the update parameterization, but optimization, data quality, target modules, and evaluation determine whether that capacity is useful. It can produce no measurable gain, increase variance or overfitting, and consume more memory, storage, and serving cache.
How many parameters does LoRA rank add?
For one target weight with input dimension d_in and output dimension d_out, rank r adds r multiplied by d_in plus d_out parameters for A and B. Sum this value across all selected weights and layers, then add any trained bias, embedding, head, or modules_to_save parameters.
How does LoRA rank affect deployment?
Rank changes Adapter bytes and may change runtime compute, cache allocation, maximum concurrent Adapters, and engine compatibility. Serving systems can impose a maximum rank or reserve cache for that maximum, so training configuration and serving capacity must be planned together.
Should I increase rank when LoRA quality is poor?
Not first. Check labels, completion or assistant loss masks, truncation, chat templates, duplicate leakage, target-module names, and the unchanged-base baseline. Increase rank only as a controlled experiment after these failures are excluded; the problem may instead require better data or a different objective.