Multimodal RAG Retrieval: Image, Text, Pages, and Evaluation

The Retrieval Problem Is Not One Problem

“Multimodal RAG” often hides several different retrieval tasks:

1. Text-to-image retrieval: “Find the product photo with a red emergency stop button.”
2. Image-to-text retrieval: “Find the manual section that explains this photographed warning symbol.”
3. Visual document retrieval: “Find the page containing the quarterly margin chart and its footnote.”
4. Exact text retrieval: “Find invoice ACME-2026-00417.”
5. Mixed evidence retrieval: “Explain the wiring procedure using the diagram, table, and revision note.”

No single index is guaranteed to solve all five. A global image-text embedding can recognize broad visual concepts while missing a tiny identifier. OCR plus BM25 can recover the identifier while losing layout. A page-image late-interaction model can preserve visual structure but consume more index memory and still needs an evidence extraction plan.

The production goal is not “put images in a vector database.” It is:

retrieve authorized, versioned evidence with enough visual and textual context to support a verifiable answer.

For a broader introduction, read the Multimodal RAG guide. This article focuses on retrieval architecture and evaluation.

Three Complementary Retrieval Paths

Path A: Shared-Space Image and Text Embeddings

CLIP demonstrated contrastive training that maps images and natural-language descriptions into a shared representation space. Similar dual encoders support:

• text query to image;
• image query to text;
• image-to-image or caption-to-caption similarity through the aligned space;
• fast approximate nearest-neighbor retrieval with one global vector per asset.

Strengths:

• efficient indexing and candidate generation;
• good broad semantic matching;
• practical for photos, products, scenes, and illustrations;
• simple text-to-image and image-to-text symmetry.

Limitations:

• small text, exact identifiers, tables, and detailed spatial relations may collapse into one global vector;
• score meaning changes by model, preprocessing, language, and corpus;
• a visually similar result is not necessarily evidence for the requested fact.

CLIP is a model family and training approach, not a guarantee that every derived checkpoint has the same license, language support, input size, or calibration.

Path B: Visual Document Late Interaction

ColPali, published at ICLR 2025, encodes a document page image into multiple vectors and compares them with query-token vectors using late interaction. A simplified relevance function is:

score(query, page) =
  sum over query vectors i (
    max over page vectors j similarity(query_i, page_j)
  )

Unlike one-vector-per-page retrieval, different query tokens can match different visual regions. This helps retain signals from tables, figures, typography, and layout.

ColPali also introduced the ViDoRe benchmark for visually rich page-level retrieval. Its paper reports strong results under that benchmark's datasets and protocol. It does not prove that every ColPali checkpoint beats every OCR pipeline, indexes three times faster in every environment, or is the best model in 2026.

Operational trade-offs:

• many vectors per page increase index memory;
• candidate scoring is more expensive than one-vector cosine search;
• page-level retrieval still needs region localization or evidence extraction;
• exact text and policy filters may still require OCR or metadata;
• model and preprocessing versions must match between indexing and query time.

Path C: Text, Layout, and Metadata Retrieval

OCR is not obsolete. It provides capabilities visual embeddings do not guarantee:

• exact IDs, amounts, dates, clauses, and error codes;
• phrase and Boolean search;
• deterministic language or document-type filters;
• text highlighting and accessible output;
• redaction and compliance review;
• lexical evidence for citations.

A useful text path may combine:

• OCR tokens with bounding boxes and confidence;
• document-native text when available;
• BM25 or another lexical retriever;
• text embeddings;
• layout-aware chunks;
• document metadata and access-control fields.

Do not silently prefer OCR text over native PDF text or vice versa. Store source type and extraction version so evaluation can identify which path failed.

A Production Retrieval Architecture

ingestion
  -> immutable asset and version IDs
  -> page render + image hash
  -> native text / OCR + regions
  -> metadata and ACL normalization
  -> image-text global vectors
  -> page multi-vectors
  -> lexical/text vectors

query
  -> identity and authorization filter
  -> query classification
  -> parallel candidate generation
  -> per-path top-k with provenance
  -> rank fusion or calibrated fusion
  -> optional evidence-aware reranking
  -> page/region evidence package
  -> answer generation
  -> citation and support verification

Apply Authorization Before Retrieval

Tenant, user, legal-hold, geography, and document-state filters belong in candidate generation, not after ranking. Post-filtering an unauthorized top-k can:

• leak object existence through timing or counts;
• return fewer results than expected;
• let unauthorized candidates influence a reranker or model;
• contaminate caches and traces.

Every result should carry an authorization decision or query-scope identifier that can be audited without logging sensitive content.

Preserve One Evidence Identity

All representations of the same evidence need a shared identity:

{
  "asset_id": "manual-42",
  "version_id": "sha256:...",
  "page": 17,
  "region_id": "p17-r08",
  "source_uri": "s3://approved/manual-42.pdf",
  "render_hash": "sha256:...",
  "native_text_version": "pdf-parser-4.2",
  "ocr_version": "ocr-7.1",
  "image_encoder": "clip-checkpoint@digest",
  "page_encoder": "colpali-checkpoint@digest",
  "acl_scope": "tenant-7:engineering",
  "indexed_at": "2026-10-03T08:00:00Z"
}

Without shared IDs, the same page can appear as three unrelated candidates. Deduplicate by evidence identity before allocating the final context budget.

Do Not Average Raw Scores

BM25, cosine similarity, and late-interaction MaxSim have different scales and distributions. Even two cosine retrievers may use different normalization, dimensionality, and training objectives. This is invalid without evidence:

final = 0.4 * bm25 + 0.3 * clip_cosine + 0.3 * colpali_maxsim

The weights look precise but have no stable meaning.

Reciprocal Rank Fusion

Reciprocal Rank Fusion combines result positions instead of raw scores:

RRF(document) = sum over rankers r (weight_r / (k + rank_r(document)))

It is a strong baseline when retriever scores are incomparable. It does not need score calibration, and one extreme raw score cannot dominate the result. The constant k, path weights, candidate depth, and deduplication rule still need evaluation.

A Go Implementation

package main

import (
	"fmt"
	"sort"
)

type RankedItem struct {
	ID string
}

type FusedItem struct {
	ID    string
	Score float64
}

func reciprocalRankFusion(
	lists [][]RankedItem,
	weights []float64,
	k float64,
) []FusedItem {
	scores := make(map[string]float64)
	for listIndex, items := range lists {
		weight := 1.0
		if listIndex < len(weights) {
			weight = weights[listIndex]
		}
		seen := make(map[string]bool)
		for itemIndex, item := range items {
			if item.ID == "" || seen[item.ID] {
				continue
			}
			seen[item.ID] = true
			rank := float64(itemIndex + 1)
			scores[item.ID] += weight / (k + rank)
		}
	}

	fused := make([]FusedItem, 0, len(scores))
	for id, score := range scores {
		fused = append(fused, FusedItem{ID: id, Score: score})
	}
	sort.Slice(fused, func(i, j int) bool {
		if fused[i].Score == fused[j].Score {
			return fused[i].ID < fused[j].ID
		}
		return fused[i].Score > fused[j].Score
	})
	return fused
}

func main() {
	lists := [][]RankedItem{
		{{ID: "page-17"}, {ID: "page-8"}, {ID: "page-31"}},
		{{ID: "page-8"}, {ID: "page-17"}, {ID: "page-22"}},
	}
	for _, item := range reciprocalRankFusion(lists, []float64{1, 1}, 60) {
		fmt.Printf("%s %.6f\n", item.ID, item.Score)
	}
}

The duplicate guard prevents one retriever from increasing a document's score by returning it more than once. Production code should also carry path, rank, raw score, evidence version, and ACL scope for audit.

When to Calibrate or Learn Fusion

Use calibrated scores or a learned ranker only when representative labels exist. Features can include:

• per-path rank and score;
• OCR confidence;
• query type;
• language and document class;
• image quality;
• agreement across paths;
• freshness and version match.

Split evaluation by document or time to avoid near-duplicate leakage. A learned fuser that sees the same template in train and test may appear accurate while failing on a new supplier, language, or scan type.

Query Routing Without Brittle Rules

A router can save cost, but it creates a new failure mode. Start with an explainable policy:

Log the selected paths and evaluate router recall: how often did routing exclude the only path that contained the relevant item? A cheap router is not useful if it systematically removes evidence for minority languages or scanned documents.

Evidence Packaging for Generation

Retrieval output should not be a bag of vectors. Build an evidence package:

{
  "evidence_id": "manual-42@sha256:...#page=17&region=p17-r08",
  "retrieval_paths": [
    {"name": "bm25", "rank": 4},
    {"name": "colpali", "rank": 1}
  ],
  "page_image": "signed://...",
  "region": {"x": 0.14, "y": 0.28, "width": 0.62, "height": 0.31},
  "text": "Approved OCR or native-text excerpt",
  "source_title": "Pump Service Manual",
  "revision": "2026-08-14",
  "acl_scope": "tenant-7:engineering"
}

The generator should receive only authorized evidence and enough neighboring context to interpret the region. The final answer should cite the evidence ID, not a model-generated page label.

For diagrams and tables, preserve the page crop along with text. OCR alone may flatten row/column relationships; an image alone may make exact quotation difficult. The pair is often more useful than either representation.

Retrieval Evaluation

Build Labels Around Real Decisions

The label unit must match the product:

• asset level for media search;
• page level for visual document retrieval;
• region level when the UI highlights evidence;
• set level when several pages are jointly required.

Annotate more than one relevant item when the corpus contains legitimate alternatives. Include hard negatives from the same template, product family, month, or visual layout.

Required Query Slices

• modality: text-to-image, image-to-text, text-to-page, mixed;
• content: photo, diagram, chart, table, form, dense prose;
• text quality: native text, clean OCR, degraded scan, handwriting;
• language and script;
• exact identifier versus semantic description;
• single-hop versus multi-evidence;
• current versus superseded document;
• frequent versus rare template;
• authorized versus denied evidence;
• benign versus prompt-injected content.

An overall average can hide zero recall for one high-risk slice.

Retrieval Metrics

Report confidence intervals or bootstrap ranges where the sample size permits. Freeze the corpus snapshot and exact encoder/index configuration with every report.

End-to-End RAG Metrics

Retrieval quality and answer quality are separate:

• evidence sufficiency;
• citation precision and recall;
• claim-to-evidence entailment;
• unsupported-claim rate;
• refusal when evidence is insufficient;
• exact-field accuracy;
• visual-region attribution;
• latency and cost per supported answer.

A fluent answer can be unsupported even when Recall@k is high. Conversely, a retrieval miss cannot be repaired reliably by a stronger generator.

The RAG evaluation guide provides a broader evaluation framework, while the hybrid search and reranking guide covers text-heavy retrieval tuning.

Operational Evaluation

Quality alone is insufficient. Measure:

Do not publish a fixed VRAM or latency requirement without model checkpoint, precision, batch size, page resolution, vector count, hardware, and software versions.

Drift and Change Management

“Alignment drift” should mean a measured change, not a vague explanation for poor search. Monitor:

• query distribution by slice;
• corpus mix by language, template, and image quality;
• OCR and renderer versions;
• embedding checkpoint and preprocessing;
• index parameters and quantization;
• per-path recall on a stable canary set;
• score and rank distributions;
• disagreement between retrieval paths;
• unsupported-answer rate.

Re-indexing with a new encoder while serving queries from the old encoder is a contract violation. Use versioned indexes, dual-read evaluation, and atomic traffic switching.

Every model, OCR, rendering, chunking, fusion, or ACL change should create a new evaluation run. Roll back the complete compatible index and query stack, not one component.

Security and Privacy

Multimodal evidence can contain:

• visible personal or confidential information;
• hidden PDF layers;
• EXIF and document metadata;
• QR codes and embedded URLs;
• prompt injection in text or pixels;
• malicious files targeting parsers;
• cross-tenant near duplicates.

Controls:

1. scan and normalize files in an isolated ingestion environment;
2. remove unneeded metadata;
3. preserve originals under strict access while indexing approved derivatives;
4. enforce ACLs before candidate generation;
5. treat OCR and image instructions as untrusted evidence;
6. never grant tools based on retrieved content;
7. redact logs and evaluation exports;
8. propagate deletion to derivatives, indexes, caches, and backups;
9. verify signed evidence URLs expire and remain tenant-scoped.

Model Selection Checklist

Do not begin with “Which model is best?” Begin with:

• What is the retrieval unit: asset, page, region, or evidence set?
• Which query directions are required?
• Which languages, layouts, and scan qualities matter?
• Are exact identifiers business-critical?
• What is the maximum index memory and p95 latency?
• Can data leave the deployment boundary?
• How quickly must updates become searchable?
• Which labeled slices and failure costs define acceptance?

Then evaluate at least:

1. lexical/native-text baseline;
2. text embedding baseline;
3. global image-text embedding baseline;
4. page-image late-interaction path where relevant;
5. rank-fusion baseline;
6. any learned reranker only after labels are sufficient.

The embedding guide explains vector behavior, and the multimodal pipeline guide covers upstream image and text processing.

Sources

• CLIP: Learning Transferable Visual Models From Natural Language Supervision
• Sigmoid Loss for Language Image Pre-Training
• ColPali: Efficient Document Retrieval with Vision Language Models
• ViDoRe benchmark and models
• Reciprocal Rank Fusion Outperforms Condorcet and Individual Rank Learning Methods

Conclusion

Production Multimodal RAG needs several explicit contracts: which representation retrieves which evidence, how incompatible rankings are fused, how every page or region remains traceable to a versioned source, which authorization filter applies, and which labeled evaluation proves the change. CLIP-style shared embeddings, ColPali-style late interaction, and OCR or lexical retrieval are complementary tools. The winning system is the one that meets the measured workload, not the one attached to the largest unsupported benchmark claim.