Multimodal RAG: Production Architecture and Evaluation

Multimodal RAG is not a single model or pipeline. It is a retrieval system that preserves and searches the representations needed to answer a question: text for exact language, layout for reading order, page images for charts and tables, and region-level crops for fine visual evidence.

The central design question is therefore not “Which multimodal embedding model should we use?” It is:

What is the smallest evidence unit that contains the answer, and which representation retrieves that unit reliably?

This guide develops that decision from ingestion through evaluation. It focuses on visually rich PDFs because they expose the hardest production trade-offs, but the same principles apply to product images, slide decks, scanned forms, diagrams, video frames, and other mixed media. Technical claims were reviewed against primary sources on July 16, 2026.

Key Takeaways

• Keep original assets and provenance; derived text or captions are indexes, not sources of truth.
• Text-first, image-first, and hybrid retrieval solve different failure modes. None dominates every corpus and query.
• Page-level retrieval is often too coarse for generation. Preserve coordinates so a page hit can be narrowed to an answer-bearing region.
• ColPali-style late interaction preserves visual layout but uses many vectors per page and changes storage and ranking economics.
• Route queries by required evidence, then fuse independently calibrated rankings rather than mixing incomparable raw scores.
• Evaluate retrieval before answer generation. A fluent VLM cannot recover evidence that was never retrieved.
• Treat document text, OCR, captions, and pixels as untrusted input; prompt injection can be embedded in any modality.

What Multimodal RAG Actually Means

Retrieval-Augmented Generation has two separable responsibilities:

1. Retrieval selects evidence from a corpus.
2. Generation interprets that evidence and produces an answer.

A system becomes multimodal when either stage uses more than one modality or representation. A text query can retrieve an image; OCR text can locate a PDF page that is then passed as pixels to a vision-language model; or text and visual retrievers can run in parallel and combine their rankings.

This broader definition prevents two common mistakes:

• calling any PDF chatbot “multimodal” even when it discards every visual element during parsing;
• assuming that a shared text-image embedding is the only valid architecture.

The right architecture depends on the evidence. A contract clause usually needs exact text. A revenue chart needs axes, legend, and marks. A scanned invoice needs both recognized values and their spatial relationship. A product search query may need semantic image similarity but no document layout at all.

Preserve an Evidence Graph

Do not reduce each source file to one lossy representation. Store an evidence graph that connects the original asset to every derived unit:

document
  -> page image
  -> text blocks + bounding boxes
  -> table / figure regions
  -> OCR text
  -> captions or summaries
  -> embeddings for one or more retrievers

Every indexed unit should carry at least:

{
  "document_id": "annual-report-2025",
  "page": 42,
  "region": [0.08, 0.31, 0.91, 0.78],
  "modality": "chart",
  "source_uri": "s3://reports/annual-report-2025.pdf",
  "content_hash": "sha256:...",
  "parser_version": "layout-parser-3.2",
  "access_scope": ["finance-team"]
}

Normalized coordinates allow the generation stage to crop the relevant area at the required resolution. Hashes and parser versions make re-indexing auditable. Access scope must travel with every derivative; otherwise an authorized source document can produce an unauthorized search result through a secondary index.

Five Retrieval Architectures

1. Text and Layout Retrieval

Extract native PDF text or OCR, preserve blocks and coordinates, and index text with lexical and dense retrievers.

Best for: exact clauses, names, identifiers, prose-heavy documents, and corpora that require quotations.

Failure mode: charts, visual grouping, merged cells, and reading order may be lost or reconstructed incorrectly.

This is not inherently “old” or inferior. For exact-match queries, BM25 over clean text can outperform a visual embedding while using much less storage.

2. Caption or Summary Proxies

Generate a description for each image, chart, table, or page and index that text. The hit points back to the original visual evidence.

Best for: using mature text search infrastructure, enriching otherwise opaque images, and low-volume corpora.

Failure mode: the caption model can omit a detail, misread a chart, or normalize away the distinction a future query needs. A caption is a retrieval proxy, never the authoritative evidence.

3. Shared Text-Image Embeddings

Dual encoders such as CLIP-family models map text and images into a compatible space. This supports text-to-image and image-to-image retrieval with one similarity function.

Best for: products, scenes, artwork, and broad visual semantics.

Failure mode: one global vector compresses an entire page or image. It may miss a small table cell, footnote, or region whose meaning depends on layout.

4. Visual Document Retrieval

ColPali encodes a page image into multiple vectors and compares query-token vectors with visual patch vectors using late interaction. Its paper introduced the ViDoRe benchmark and reported stronger page retrieval than the evaluated text-centric pipelines on visually rich documents.

Best for: visually rich pages where tables, layout, typography, and figures carry meaning.

Trade-offs: multi-vector indexes consume more storage and compute than one-vector-per-page indexes; a page hit still may contain large irrelevant regions; exact text extraction and citations may require a second path.

“OCR-free” means the visual retrieval path can work directly on page images. It does not mean OCR has no value elsewhere in the product.

5. Hybrid Multi-Index Retrieval

Run lexical text, dense text, caption, global image, and visual document retrievers as appropriate; normalize or rank-fuse their results; then rerank a smaller candidate set.

Best for: enterprise corpora with mixed query types and high recall requirements.

Trade-off: more moving parts, duplicate candidates, and higher indexing cost. The gain must be measured against a strong text-only baseline.

Query Routing and Rank Fusion

Not every query needs every retriever. A lightweight router can identify evidence requirements:

• identifiers, quotations, error codes -> lexical text;
• conceptual prose -> dense text;
• “in the chart,” “shown in figure,” visual comparison -> visual retrieval;
• ambiguous or high-stakes questions -> parallel hybrid retrieval.

Routing should reduce cost, not create a single point of failure. Keep a confidence threshold and fall back to hybrid retrieval when intent is uncertain.

Do not directly average cosine similarity from unrelated models. Their score distributions are not calibrated. Reciprocal Rank Fusion (RRF) combines rank positions instead:

from collections import defaultdict
from dataclasses import dataclass


@dataclass(frozen=True)
class Hit:
    evidence_id: str
    rank: int
    retriever: str


def reciprocal_rank_fusion(
    rankings: list[list[str]],
    rank_constant: int = 60,
) -> list[tuple[str, float]]:
    scores: dict[str, float] = defaultdict(float)
    for ranking in rankings:
        for rank, evidence_id in enumerate(ranking, start=1):
            scores[evidence_id] += 1.0 / (rank_constant + rank)
    return sorted(scores.items(), key=lambda item: item[1], reverse=True)


text_hits = ["page-12", "page-42", "page-09"]
visual_hits = ["page-42", "page-18", "page-12"]
print(reciprocal_rank_fusion([text_hits, visual_hits]))

This example is runnable with the Python standard library. In production, preserve each contributing retriever and score in the trace, deduplicate at the evidence-unit level, and rerank the fused candidates with a model suited to the target corpus.

From Page Retrieval to Grounded Generation

Retrieval and generation should exchange an evidence package rather than an unstructured list of images:

from __future__ import annotations

from dataclasses import dataclass


@dataclass(frozen=True)
class Evidence:
    evidence_id: str
    document_id: str
    page: int
    region: tuple[float, float, float, float] | None
    text: str | None
    image_uri: str | None
    retrieval_reasons: tuple[str, ...]


def validate_evidence(item: Evidence) -> None:
    if item.page < 1:
        raise ValueError("page must be one-based")
    if item.text is None and item.image_uri is None:
        raise ValueError("evidence must contain text or an image")
    if item.region and any(value < 0 or value > 1 for value in item.region):
        raise ValueError("region coordinates must be normalized")

The generation prompt should require the model to:

1. answer only from supplied evidence;
2. cite document and page for every material claim;
3. distinguish observed values from inferred conclusions;
4. say when evidence is insufficient or contradictory.

For charts and dense pages, send a page overview plus high-resolution crops of the relevant regions. A fixed rule such as “always include the surrounding 200 words” is unreliable: the correct context boundary may be a caption, a table header, a referenced footnote, or a section spanning pages.

Evaluation: Four Separate Gates

End-to-end answer quality alone cannot diagnose a multimodal RAG system. Evaluate four layers independently.

1. Ingestion Fidelity

• OCR character or word error rate on representative scans;
• reading-order accuracy;
• table structure accuracy;
• figure-caption association;
• percentage of assets with valid provenance and permissions.

2. Retrieval

Build judgments at the unit used by the system: document, page, region, or asset.

• Recall@k: did the candidate set contain the answer-bearing evidence?
• MRR or nDCG@k: how high was useful evidence ranked?
• modality slices: prose, table, chart, diagram, scan, multilingual page;
• query slices: exact, semantic, visual, comparison, multi-hop.

The ViDoRe benchmark is useful for visual document retrieval, but a benchmark result does not replace evaluation on the layouts, languages, and query distribution of your corpus.

3. Grounding and Answer Quality

• citation correctness: does the cited region support the claim?
• citation completeness: are material claims cited?
• evidence sufficiency: could the answer be derived from the package?
• task accuracy: exact match, numeric tolerance, rubric, or expert review;
• abstention quality when evidence is missing.

4. Operations

• indexing throughput and failure rate;
• index bytes and vectors per page;
• p50/p95 retrieval and generation latency;
• image and output token consumption;
• cache hit rate and cost per successful answer;
• permission-filter and deletion propagation latency.

Compare every complex design with a strong baseline. A hybrid visual pipeline is not an improvement if it gains one point of Recall@10 while doubling latency and tripling cost on queries users actually ask.

Security and Privacy

Multimodal input expands the attack surface:

• instructions can be hidden in document text, OCR layers, images, QR codes, or metadata;
• retrieved documents can attempt to override system instructions;
• images can contain faces, signatures, account numbers, or location data;
• derived captions and embeddings can retain sensitive information after the original is deleted.

Treat retrieved content as quoted evidence, never as instructions. Enforce authorization before retrieval and again before generation. Sandbox parsers, limit file size and page count, scan active PDF content, record lineage for every derivative, and make deletion propagate to text, crops, captions, caches, and vector indexes.

Production Decision Checklist

1. Define query classes and the smallest answer-bearing evidence unit.
2. Establish a text-plus-layout baseline before adding visual indexes.
3. Preserve originals, coordinates, provenance, versions, and permissions.
4. Select retrievers per query class; use hybrid fallback for uncertainty.
5. Fuse ranks or calibrate scores; do not average unrelated similarities blindly.
6. Rerank and crop before sending expensive visual context to the VLM.
7. Require page- or region-level citations and allow abstention.
8. Evaluate ingestion, retrieval, grounding, answers, and operations separately.
9. Test prompt injection and deletion across every derived representation.
10. Re-index reproducibly when models, parsers, or access policies change.

Frequently Asked Questions

Can text embeddings process images?

No. A text embedding model processes text representations. You can index a generated caption with it, but retrieval quality is then bounded by what the caption preserved. For direct text-to-image retrieval, use a compatible multimodal encoder or a separate visual retrieval model.

Is ColPali always better than OCR plus text retrieval?

No. ColPali was designed for visually rich page retrieval and performed strongly on ViDoRe. Exact quotations, identifiers, clean prose, low-storage deployments, and compliance workflows may still favor text or hybrid retrieval. Test both on your corpus.

Should the vector database store image bytes?

Usually no. Store immutable assets in object storage and keep URI, hash, coordinates, permissions, and model version in the index. This limits index bloat and makes lifecycle management clearer.

How do I control multimodal RAG cost?

Route text-only queries away from visual generation, retrieve before invoking a VLM, crop answer-bearing regions, cap candidates and image resolution, cache immutable derivatives, and measure cost per correct answer rather than cost per request.

Conclusion

Production multimodal RAG is an evidence architecture, not a demo that sends a PDF screenshot to a vision model. Its quality depends on preserving representations, choosing the right retrieval unit, combining complementary retrievers, and proving that the final answer is supported by identifiable evidence.

Start with the simplest baseline that preserves the information your users ask about. Add visual retrieval only where measured failures justify its storage, latency, and operational cost.

Primary Sources

• ColPali: Efficient Document Retrieval with Vision Language Models
• ViDoRe benchmark and models
• CLIP: Learning Transferable Visual Models From Natural Language Supervision
• Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks