Semantic Search: Hybrid Retrieval and Evaluation Guide

Semantic search retrieves by learned relationships between a query and content, not by token overlap alone. That capability helps with paraphrases and conceptual discovery, but it does not reveal a user's "true intent" or guarantee a correct result. A production search system also needs lexical evidence, authorization, document lifecycle controls, versioned indexes, ranking, and evaluation.

The Semantic Search glossary owns the concise definition. This guide owns the engineering question: how do you build semantic search that is measurable, authorized, and replaceable without corrupting the index?

TL;DR

• Treat semantic search as a retrieval objective, not as a synonym for vector search.
• Keep lexical retrieval for identifiers, exact phrases, code symbols, names, and emerging vocabulary.
• Compare lexical-only, semantic-only, and hybrid pipelines on the same labeled corpus snapshot.
• Fuse incompatible score spaces by rank, such as Reciprocal Rank Fusion (RRF), unless calibrated score fusion has evidence.
• Derive tenant and ACL filters from the authenticated principal; never accept authorization scope from free-form query input.
• Version the complete index contract and migrate with a new index, replay evaluation, shadow traffic, and controlled cutover.
• Evaluate retriever, ranker, ANN index, product outcome, and security separately.

What Is Semantic Search?

Semantic search is an information retrieval approach that uses learned representations to find content related to the meaning of a query. Dense embeddings are one implementation. Learned sparse retrieval, lexical expansion, hybrid retrieval, and reranking can also contribute to the same product goal.

This boundary matters:

A vector search engine performs nearest-neighbor lookup over vectors. Those vectors might represent text, images, products, users, or fraud signals. Semantic search is the user-facing retrieval objective. A vector index alone is therefore neither a complete semantic search system nor evidence that returned documents are relevant.

Build the Retrieval Pipeline Around Explicit Contracts

A reliable pipeline separates offline indexing from online query execution and makes policy enforcement part of both.

The index contract should include at least:

corpus version
document parser and preprocessing revision
chunking policy and parent-child mapping
embedding provider, model revision, and task mode
vector dimensions and normalization
distance metric and ANN configuration
lexical analyzer and field schema
metadata schema, lifecycle state, and policy version

Changing any field can change retrieval behavior. Changing the embedding model, dimensions, normalization, or distance metric usually makes old document vectors incompatible with new query vectors. Do not mix them and hope similarity remains meaningful.

Chunk Documents Without Losing Identity

Chunking is a retrieval decision, not just string slicing. Every chunk needs stable identity, a parent document, source provenance, version, and offsets so results can be reconciled, cited, deleted, and reindexed.

This standard-library example avoids the common trailing-chunk loop and validates that overlap is smaller than chunk size:

from dataclasses import dataclass
from typing import List


@dataclass(frozen=True)
class Chunk:
    chunk_id: str
    parent_id: str
    source_uri: str
    corpus_version: str
    start: int
    end: int
    text: str


def chunk_text(
    text: str,
    parent_id: str,
    source_uri: str,
    corpus_version: str,
    chunk_size: int = 120,
    overlap: int = 20,
) -> List[Chunk]:
    if chunk_size <= 0 or overlap < 0 or overlap >= chunk_size:
        raise ValueError("require chunk_size > overlap >= 0")

    chunks: List[Chunk] = []
    start = 0
    while start < len(text):
        hard_end = min(start + chunk_size, len(text))
        end = hard_end
        if hard_end < len(text):
            boundary = max(
                text.rfind(". ", start, hard_end),
                text.rfind(" ", start, hard_end),
            )
            if boundary > start:
                end = boundary + 1

        body = text[start:end].strip()
        if body:
            chunks.append(
                Chunk(
                    chunk_id=f"{parent_id}:{start}:{end}",
                    parent_id=parent_id,
                    source_uri=source_uri,
                    corpus_version=corpus_version,
                    start=start,
                    end=end,
                    text=body,
                )
            )
        if end == len(text):
            break
        start = end - overlap

    return chunks


sample = "Semantic retrieval finds paraphrases. Lexical retrieval preserves exact identifiers."
result = chunk_text(sample, "doc-7", "kb://search", "corpus-2026-08", 48, 8)
assert result[-1].end == len(sample)
assert len({item.chunk_id for item in result}) == len(result)
print([(item.start, item.end) for item in result])

Chunk size must be evaluated with the retriever and task. Small chunks may improve passage precision but lose context; large chunks may preserve context but dilute the matching signal and consume more reranking or LLM budget. Parent-document retrieval can return a precise child hit while presenting a larger authorized context window.

Choose Retrieval Paths by Query Slice

The right retriever depends on the query distribution, corpus, and constraints.

Do not select an embedding model from a static leaderboard or dimension count. Define a model contract first: languages, asymmetric versus symmetric retrieval, domain, maximum input, privacy, hardware or API constraints, latency, and cost. Then benchmark candidates on the same versioned labels. The Sentence Transformers documentation explicitly distinguishes symmetric similarity tasks from asymmetric query-to-passage retrieval; the encoder mode must match the task.

The similarity metric is also part of the model contract. Cosine similarity, dot product, and Euclidean distance are not interchangeable labels. Normalized vectors make dot product numerically equal to cosine similarity, but unnormalized models can intentionally encode magnitude. Use the metric and preprocessing required by the selected model, then validate the full pipeline.

Fuse Rankings Without Pretending Scores Are Comparable

BM25 scores, cosine similarity, inner products, and model logits have different semantics and ranges. Query-local min-max normalization does not calibrate them: one outlier or a changed candidate set can alter every normalized score. A fixed weighted sum such as 0.5 * dense + 0.5 * BM25 therefore needs labeled calibration evidence.

Reciprocal Rank Fusion avoids raw-score comparability by using each document's rank:

RRF(d) = sum(1 / (k + rank_i(d)))

The following implementation keeps each path's contribution for debugging:

from collections import defaultdict
from dataclasses import dataclass
from typing import Dict, List, Sequence


@dataclass(frozen=True)
class FusedHit:
    document_id: str
    score: float
    contributions: Dict[str, float]


def reciprocal_rank_fusion(
    rankings: Dict[str, Sequence[str]],
    rank_constant: int = 60,
    limit: int = 10,
) -> List[FusedHit]:
    if rank_constant < 1 or limit < 1:
        raise ValueError("rank_constant and limit must be positive")

    totals = defaultdict(float)
    parts: Dict[str, Dict[str, float]] = defaultdict(dict)
    for path, document_ids in rankings.items():
        for rank, document_id in enumerate(document_ids, start=1):
            contribution = 1.0 / (rank_constant + rank)
            totals[document_id] += contribution
            parts[document_id][path] = contribution

    ordered = sorted(totals, key=lambda item: (-totals[item], item))
    return [
        FusedHit(item, totals[item], parts[item])
        for item in ordered[:limit]
    ]


hits = reciprocal_rank_fusion(
    {
        "lexical": ["doc-a", "doc-c", "doc-b"],
        "semantic": ["doc-b", "doc-a", "doc-d"],
    }
)
print([(hit.document_id, round(hit.score, 5)) for hit in hits])
# [('doc-a', 0.03252), ('doc-b', 0.03227), ...]

The original RRF paper used k=60 in its experiments, and several products use that value as a default. It is a baseline, not a universal optimum. Tune the constant, candidate depth, path weights, and tie handling on labels. Also compare against each single retriever: fusion can degrade quality when a weak path injects noise.

Enforce Authorization Inside the Retrieval Runtime

Authorization is not an optional metadata filter supplied by a caller, user, or LLM. The runtime must derive policy scope from the authenticated principal and an authoritative policy service, then apply the same scope to every retrieval path.

from dataclasses import dataclass
from typing import Callable, Dict, List, Sequence


@dataclass(frozen=True)
class Principal:
    subject: str
    tenant: str
    roles: Sequence[str]
    policy_version: str


@dataclass(frozen=True)
class Candidate:
    document_id: str
    tenant: str
    allowed_roles: Sequence[str]
    lifecycle: str


Retriever = Callable[[str, str], List[Candidate]]


def fuse_rankings(rankings: Dict[str, List[str]]) -> List[str]:
    scores: Dict[str, float] = {}
    for document_ids in rankings.values():
        for rank, document_id in enumerate(document_ids, start=1):
            scores[document_id] = (
                scores.get(document_id, 0.0) + 1.0 / (60 + rank)
            )
    return sorted(scores, key=lambda item: (-scores[item], item))


def authorized_search(
    query: str,
    principal: Principal,
    lexical_retriever: Retriever,
    semantic_retriever: Retriever,
) -> List[str]:
    def allowed(candidate: Candidate) -> bool:
        return (
            candidate.tenant == principal.tenant
            and candidate.lifecycle == "active"
            and bool(set(candidate.allowed_roles) & set(principal.roles))
        )

    lexical = [
        hit.document_id
        for hit in lexical_retriever(query, principal.tenant)
        if allowed(hit)
    ]
    semantic = [
        hit.document_id
        for hit in semantic_retriever(query, principal.tenant)
        if allowed(hit)
    ]
    return fuse_rankings({"lexical": lexical, "semantic": semantic})

In a real index, push tenant, lifecycle, and ACL constraints into candidate generation whenever the engine supports it. Post-filtering a small ANN top-K can return no authorized results even when relevant authorized documents exist deeper in the index. It can also hide a security dependency in application code. Measure filtered ANN recall and isolate tenants or payload indexes according to scale and threat model.

Every result should retain document_id, chunk_id, source, corpus version, policy version, and retrieval-path evidence. That provenance supports citations, deletion, audit, and incident response.

Treat Reranking and Similarity Scores as Bounded Signals

A bi-encoder independently encodes queries and documents, which makes document representations reusable for retrieval. A cross-encoder or another reranker jointly evaluates a query-document pair over a smaller candidate set. Joint scoring may improve ranking for a workload, but it increases latency and cost and can still fail.

Test candidate depth and reranking depth together. If the relevant document never enters the retriever's top-K, the reranker cannot recover it. If the reranker sees too many candidates, latency may violate the service objective. If it sees too few, it may discard useful diversity.

Never expose 1 - distance as a generic "relevance probability." Distance direction and range depend on the metric and implementation. Keep raw scores internal, label them by model and index version, and use rank plus evaluated thresholds where a product decision needs a boundary.

Make Cache Keys Policy- and Index-Aware

A cache keyed only by query text can return stale or cross-tenant results. The key must cover every input that can change authorization or ranking.

from dataclasses import asdict, dataclass
import hashlib
import json
from typing import Tuple


@dataclass(frozen=True)
class SearchCacheKey:
    subject: str
    tenant: str
    policy_version: str
    corpus_version: str
    index_version: str
    normalized_query: str
    filter_items: Tuple[Tuple[str, str], ...]
    ranking_config: str

    def digest(self) -> str:
        payload = json.dumps(
            asdict(self),
            ensure_ascii=True,
            sort_keys=True,
            separators=(",", ":"),
        )
        return hashlib.sha256(payload.encode("utf-8")).hexdigest()


key = SearchCacheKey(
    subject="user-17",
    tenant="tenant-a",
    policy_version="policy-42",
    corpus_version="corpus-9",
    index_version="search-12",
    normalized_query="refund policy",
    filter_items=(("language", "en"),),
    ranking_config="lexical+dense:rrf-v3",
)
print(key.digest())

Invalidate or naturally bypass cached entries when policy, corpus, index, filters, or ranking changes. Whether the subject can be replaced with a coarser authorization cohort is a security decision, not just a cache optimization.

Evaluate Five Different Layers

One aggregate relevance score cannot diagnose a semantic search system. Keep these evaluation layers separate:

Latency should include p50 and p95 for query encoding, each retriever, fusion, reranking, policy lookup, and total request time. Cost should be tied to a successful query, not just requests, so a cheaper but ineffective pipeline does not appear efficient.

This minimal evaluator demonstrates the distinction between retrieval and ranking:

from math import log2
from typing import Dict, Iterable, List, Set


def recall_at_k(ranking: List[str], relevant: Set[str], k: int) -> float:
    if not relevant:
        raise ValueError("relevant set must not be empty")
    return len(set(ranking[:k]) & relevant) / len(relevant)


def reciprocal_rank(ranking: Iterable[str], relevant: Set[str]) -> float:
    for rank, document_id in enumerate(ranking, start=1):
        if document_id in relevant:
            return 1.0 / rank
    return 0.0


def ndcg_at_k(ranking: List[str], grades: Dict[str, int], k: int) -> float:
    def dcg(items: List[str]) -> float:
        return sum(
            (2 ** grades.get(item, 0) - 1) / log2(rank + 1)
            for rank, item in enumerate(items, start=1)
        )

    actual = dcg(ranking[:k])
    ideal = dcg(sorted(grades, key=grades.get, reverse=True)[:k])
    return actual / ideal if ideal else 0.0


ranking = ["doc-b", "doc-a", "doc-c"]
grades = {"doc-a": 3, "doc-c": 1}
print(recall_at_k(ranking, set(grades), 3))  # 1.0
print(reciprocal_rank(ranking, set(grades)))  # 0.5
print(round(ndcg_at_k(ranking, grades, 3), 4))

Build judgments from real query logs with privacy controls, expert labels, support tickets, and known hard cases. Include exact identifiers, no-answer queries, multilingual pairs, stale documents, deleted documents, adversarial metadata, and every permission boundary. Public benchmarks such as BEIR help compare research methods, but they do not replace workload-specific labels.

Migrate an Embedding Index Without Mixing Vector Spaces

An embedding change is a data migration:

1. Freeze an explicit old and new index contract.
2. Create a new physical index; do not overwrite the serving index in place.
3. Re-parse, re-chunk if required, and re-embed the complete authorized corpus.
4. Validate document counts, deletions, metadata, vector dimensions, and policy fields.
5. Run exact-search checks on a sample to quantify ANN loss.
6. Replay the versioned labeled query suite against both indexes.
7. Shadow production queries without exposing shadow results to users.
8. Compare relevance slices, authorization, stale-result rate, latency, memory, and cost.
9. Move a read alias or controlled traffic percentage only after gates pass.
10. Retain rollback capability until the new index and caches are stable.

Dual writing during migration reduces drift, but it does not prove equivalence. Document updates and deletions need idempotent version handling in both indexes. Record which corpus and policy versions produced every result so incidents can be reconstructed.

Production Decision Checklist

Before shipping a retrieval change, answer these questions with evidence:

• What stable query intent does this pipeline serve?
• Which query slices are lexical, semantic, or hybrid?
• What is the corpus, label, policy, and index version?
• Does every retrieval path apply identical tenant and lifecycle constraints?
• Does hybrid beat both single-path baselines?
• Does reranking add enough nDCG or MRR to justify p95 latency and cost?
• What is ANN recall relative to exact search?
• Can the system remove a document from every index and cache?
• Can operators attribute a result to source, chunk, retriever, ranker, and policy?
• What thresholds stop rollout and trigger rollback?

Semantic search becomes production-ready when these contracts and measurements are reviewable. More vectors, a larger model, or an extra reranker are implementation choices, not proof of better search.

FAQ

Are semantic search and vector search the same?

No. Vector search is an operation over vector representations. Semantic search is a retrieval objective and system that may use dense vectors, learned sparse representations, lexical retrieval, filters, fusion, and reranking. The system boundary also includes authorization, lifecycle, provenance, and evaluation.

Is hybrid retrieval always better than one retriever?

No. Hybrid retrieval is useful when paths contribute complementary relevant candidates. If one path is weak, fusion can add noise, latency, memory, and tuning cost. Always compare hybrid against lexical-only and semantic-only baselines on the same labels and failure slices.

What does RRF solve?

RRF combines ranked lists without assuming their raw scores are comparable. It is simple and inspectable, but it discards score magnitude and still requires evaluation of its rank constant, path weights, candidate depths, and tie behavior.

Can post-filtering secure vector search?

Post-filtering is not a sufficient default. It may remove every item from a small top-K and reduce recall, and an omitted application check can become a disclosure path. Prefer policy-aware candidate generation, defense-in-depth validation, and explicit cross-tenant tests.

Does better retrieval guarantee a correct RAG answer?

No. Retrieval quality determines whether useful, authorized evidence is available. Generation can still ignore, misread, or contradict that evidence. Evaluate retrieval, citation support, answer correctness, abstention, and safety as separate layers. The RAG production guide covers the answer-system boundary.

Sources and Further Reading

• Sentence-BERT: Sentence Embeddings using Siamese BERT-Networks - foundational bi-encoder retrieval work.
• BEIR: A Heterogeneous Benchmark for Zero-shot Evaluation of Information Retrieval Models - evidence for evaluating retrieval across diverse tasks.
• Efficient and Robust Approximate Nearest Neighbor Search Using Hierarchical Navigable Small World Graphs - the HNSW algorithm and its approximation trade-offs.
• Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods - the original RRF experiments.
• Sentence Transformers: Semantic Search - symmetric and asymmetric retrieval guidance.
• Vector Embeddings Engineering Guide - embedding contracts, evaluation, and migration.
• Vector Database Guide - storage and ANN infrastructure choices.