What is Adapter?

Adapter is base-dependent trainable state that changes a pretrained model's behavior while most base parameters remain frozen. The term can mean an inserted bottleneck module, a broader PEFT component such as LoRA, or the checkpoint artifact that stores the learned adaptation; these meanings should not be treated as identical.

Quick Facts

SpecificationOfficial Specification

How It Works

Disambiguate the Module, Method, and Artifact

In the original [bottleneck Adapter paper](https://arxiv.org/html/1902.00751), an Adapter is a small residual network inserted into a frozen Transformer. Current libraries also use the word for a configurable PEFT method or its saved state. Name the exact mechanism before comparing it with LoRA, soft prompts, or selective tuning: they modify different tensors, add different forward-path work, and have different composition and merge behavior.

Trace the Bottleneck Residual Path

A canonical block projects a hidden vector from width d down to bottleneck width m, applies a nonlinearity, projects back to d, and adds the result to the residual stream. Its weight count is 2dm before biases and any task head, but parameter count alone does not determine activation memory or latency. The paper's reported BERT and GLUE efficiency belongs to that architecture and evaluation protocol, not every model or workload.

Qualify Composition and Serving Separately

Bind every Adapter to immutable base weights, architecture, Tokenizer or Processor, Template, configuration, framework and evaluation revisions. The current [AdapterHub composition guide](https://docs.adapterhub.ml/adapter_composition.html) supports method-specific Stack, Fuse, Split, Parallel and Average operations, with model-dependent exceptions. Test the exact activated composition, authorize catalog names instead of arbitrary paths, measure added latency and memory, and retain a compatible rollback.

Key Characteristics

  • Uses learned state attached to or composed with a frozen or mostly frozen base model
  • May be a bottleneck module, low-rank update, learned prompt, scaling vector, or framework-specific PEFT component
  • Has method-specific inference cost and mergeability; a canonical bottleneck module is not equivalent to mergeable LoRA
  • Requires exact Base-Adapter compatibility across weights, architecture, tokenizer or processor, template, and implementation
  • Creates a separately versioned behavior change that needs task, regression, safety, latency, and parity evaluation
  • Supports shared-base variants only when serving limits, cache policy, authorization, observability, and rollback are explicit

Common Use Cases

  1. Adding a task or domain specialization while retaining an immutable shared base
  2. Testing multiple adaptation mechanisms under the same data and evaluation protocol
  3. Shipping compact, independently versioned model variants with explicit base dependencies
  4. Serving a controlled catalog of tenant, language, or workflow variants on compatible infrastructure
  5. Rolling back an adapted behavior without misidentifying the underlying base, tokenizer, or template

Example

loading...
Loading code...

Frequently Asked Questions

Is LoRA an adapter method?

Many libraries use Adapter as an umbrella term that includes LoRA because LoRA adds separately trainable state to a frozen base. Mechanically, however, LoRA's low-rank weight update differs from a canonical bottleneck Adapter inserted into the forward path. State which meaning is intended when discussing architecture, latency, or merging.

What is the difference between an Adapter and PEFT?

PEFT is the broader family of parameter-efficient adaptation methods. Adapter may refer specifically to an inserted bottleneck module, generically to a PEFT component, or to a saved adaptation checkpoint. LoRA, soft prompts, selective parameter updates, and bottleneck modules can all belong to PEFT, but they do not have the same computation or artifact format.

Can every Adapter be merged into the base model?

No. LoRA-style delta weights are often mergeable when the framework, dtype, quantization, and configuration support it. Bottleneck modules and soft prompts change the forward path and generally remain separate. Even a supported merge creates a new full-model artifact that needs finite-output, parity, held-out, safety, and serving validation.

Can an Adapter checkpoint run without its base model?

Usually not. The checkpoint stores learned adaptation state and configuration, not all base weights. Loading requires a compatible base revision and architecture and often the same tokenizer or processor, input template, framework behavior, and target-module names. Record immutable revisions and hashes instead of relying only on a repository name.

Is switching between Adapters automatically safe?

No. A serving system must validate compatibility, authorize catalog entries, bound loaded and active Adapter counts, manage CPU and accelerator caches, isolate tenants, observe latency and quality, and provide rollback. User-controlled local or remote paths can also create a supply-chain risk; expose approved names rather than arbitrary artifact locations.

Related Terms

PEFT

PEFT (Parameter-Efficient Fine-Tuning) is a family of methods that adapts a pretrained model by training selected existing parameters or a comparatively small set of added parameters while most base weights remain frozen. Parameter efficiency does not by itself guarantee a fixed memory reduction or the quality of full fine-tuning.

LoRA

LoRA (Low-Rank Adaptation) is a parameter-efficient adaptation method that keeps selected pretrained weight matrices frozen and learns additive updates through smaller low-rank matrices. It reduces the trainable update surface, but it does not guarantee a fixed memory saving, hardware fit, training speed, or quality relative to full fine-tuning.

QLoRA

QLoRA (Quantized Low-Rank Adaptation) is a parameter-efficient fine-tuning method that keeps a pretrained base model frozen in 4-bit quantized storage while backpropagating through its dequantized computations into trainable LoRA adapters. It reduces base-weight memory but does not make all training state 4-bit or guarantee that a model will fit a specific device.

Fine-tuning

Fine-tuning is the process of continuing optimization from a pretrained model checkpoint on a target dataset and objective, updating some or all model parameters to change task, domain, or behavioral performance.

Related Articles

LLM Fine-Tuning: SFT, LoRA, QLoRA, and Evaluation

A rigorous LLM fine-tuning guide covering SFT, full updates, LoRA, QLoRA, data lineage, chat templates, leakage-resistant splits, reproducible training manifests, held-out evaluation, artifact identity, safety gates, deployment, and rollback without fixed hardware or quality promises.

2026-02-21

LoRA Fine-Tuning Tutorial: QLoRA & PEFT Guide (2026)

Learn LoRA fine-tuning step by step with PEFT and QLoRA. Configure rank, alpha, target modules, memory use, adapter merging, and deployment for production LLMs.

2026-02-21

RAG vs Fine-Tuning: An Evidence-Based Decision Guide

Compare RAG, fine-tuning, and long context as measurable system choices. Learn how knowledge freshness, behavior adaptation, retrieval quality, latency, cost, deletion, and evaluation determine the right architecture.

2026-04-08