What is BM25?
BM25 is a probabilistic lexical ranking function that scores documents based on query term matches, term frequency saturation, inverse document frequency, and document length normalization.
How It Works
BM25 is a durable baseline for information retrieval because it is simple, interpretable, and surprisingly strong for exact text matching. It improves on raw term frequency by reducing the marginal value of repeated terms and normalizing for document length. In AI search and RAG, BM25 is frequently used as the sparse retrieval branch in hybrid search, especially when queries include exact names, error codes, legal phrases, configuration keys, or other tokens where embeddings alone may be unreliable.
Key Characteristics
- Ranks documents using lexical term matching rather than learned semantic vectors
- Uses inverse document frequency to reward informative terms
- Applies term-frequency saturation so repeated words do not dominate without limit
- Normalizes by document length to avoid unfairly favoring long documents
- Remains a strong baseline and hybrid-search component for RAG systems
Common Use Cases
- Searching developer docs for exact API names and parameters
- Retrieving support articles by error code
- Providing a lexical baseline for RAG evaluation
- Combining BM25 hits with dense-retrieval hits through rank fusion
- Serving compliance or legal search where exact wording matters
Example
Loading code...Frequently Asked Questions
Why is BM25 still used with LLMs?
LLMs often need reliable retrieval. BM25 is strong for literal evidence, exact terms, and identifiers, so it remains useful as part of RAG pipelines.
Is BM25 semantic search?
No. BM25 is lexical: it scores documents by term matches and weighting. Semantic search usually relies on embeddings.
When does BM25 outperform dense retrieval?
It often performs better for exact strings, error codes, product names, legal phrases, and queries where the exact wording is the signal.
How is BM25 used in hybrid search?
BM25 produces sparse lexical candidates, dense retrieval produces semantic candidates, and the rankings are fused or reranked.