What is Indexer?

Indexer is a pipeline component that converts processed documents into durable retrieval records and writes their text, embeddings, sparse features, metadata, or relationships to one or more searchable stores.

Quick Facts

SpecificationOfficial Specification

How It Works

Reconcile desired state with indexed state

Treat each ingestion run as a desired manifest keyed by stable chunk or document ID. An absent record may require deletion; a matching ID with a changed revision requires an upsert; an unchanged revision can be skipped. This model is safer than append-only writes because it handles renamed, filtered, and deleted source content. Operations should be idempotent so retries after timeouts do not create duplicate records.

Version every retrieval compatibility boundary

Dense retrieval requires the index and query path to use compatible embedding models, preprocessing, vector dimensions, and similarity conventions. Sparse analyzers, metadata schemas, and chunking strategies also affect query behavior. Record these values in an index manifest. For breaking changes, build a separate version, validate it, switch retrieval traffic explicitly, and retain a rollback window instead of mixing incompatible records in place.

Design for partial failure and verifiable cutover

Hybrid indexing may write vectors, lexical fields, and metadata to different systems that cannot share a transaction. Stage a run under a unique ID, record per-record outcomes, retry only failed idempotent operations, and expose reconciliation counts. Before cutover, compare expected and stored IDs, sample permissions and content hashes, run retrieval tests, and verify delete propagation. A successful batch call alone does not prove the index is complete.

Key Characteristics

  • Owns preparation and durable writes while the database owns storage and search execution
  • Uses stable record IDs, source revisions, and idempotent upsert or delete semantics
  • Carries text, vectors, sparse fields, metadata, permissions, and lineage as required
  • Pins embedding, analyzer, chunking, and schema compatibility in an index manifest
  • Supports staged rebuilds, explicit cutover, rollback, and full reconciliation
  • Reports per-record failures and completeness instead of hiding partial batch success

Common Use Cases

  1. Writing permission-aware chunks and embeddings for a RAG retriever
  2. Maintaining synchronized lexical and vector records for hybrid search
  3. Building a new index version after changing an embedding model or chunking policy
  4. Deleting all records derived from a removed or access-revoked source
  5. Comparing index manifests and retrieval quality before a blue-green cutover

Example

loading...
Loading code...

Frequently Asked Questions

Is an Indexer the same as a vector database?

No. A vector database is one possible storage and search backend. An Indexer is the write-side pipeline component that prepares identities, vectors, text fields, metadata, and permissions, then applies changes to one or more backends. The same indexer may coordinate a vector store and a lexical search engine.

What makes an indexing operation idempotent?

Repeating the same operation with the same stable record ID and revision must converge on the same stored state without duplicates. Use deterministic IDs, upsert semantics, explicit delete operations, and durable run manifests. Random IDs generated on every retry defeat idempotency and make stale records difficult to find.

When does a RAG index need a new version?

Create a new version when a change can make existing and new records incompatible or difficult to compare, such as an embedding model or dimension change, new tokenizer, different chunking policy, analyzer change, or breaking metadata schema. Minor content updates can usually use idempotent upserts within the same version.

How should an Indexer propagate source deletions?

Maintain the relationship from every derived record to its source ID. When a source is deleted or loses authorization, enumerate and delete all derived vector, lexical, graph, and metadata records, then verify absence. Periodic desired-versus-actual reconciliation catches tombstones lost during outages.

How do you verify an index before switching traffic?

Compare expected and stored record counts and IDs, inspect failed batches, sample content hashes and permissions, validate vector dimensions and model identifiers, and run a fixed retrieval evaluation. Cut over only after these checks pass, while preserving the previous index and an explicit rollback path.

Related Terms

Document Transformer

Document Transformer is a pipeline component that converts one or more loaded document objects into zero or more derived documents by cleaning, restructuring, enriching, filtering, redacting, or chunking content before indexing.

Embedding

Embedding is a learned numerical representation that maps text, images, entities, or other inputs into a vector space where distance encodes relationships useful to a specific training objective. Vectors are comparable only under a compatible model, revision, task mode, preprocessing, dimension, normalization, and metric.

Vector Database

A vector database is a datastore that persists vector embeddings and retrieves nearby records under a defined distance metric. Production implementations also manage stable identifiers, metadata filters, updates, deletion, durability, observability, and access control around exact or approximate nearest-neighbor search.

Retriever

Retriever is a query-to-evidence component that applies a declared search contract and returns ranked, attributable candidates or an explicit failure for downstream RAG and agent stages.

Related Articles

Eino RAG Pipeline: Production Indexing, Retrieval, and Evaluation

Build an Eino v0.9.21 RAG pipeline with explicit Loader, Transformer, Indexer, Retriever, and generation contracts. Learn chunk provenance, embedding compatibility, hybrid retrieval, authorization filtering, retrieval evaluation, safe rollout, and failure diagnosis through a runnable Go reference implementation.

2026-06-03

Vector Embeddings: Models, Search, RAG, and Migration

Learn how vector embeddings power search and RAG, then choose models, metrics, dimensions, indexes, evaluation datasets, hybrid retrieval, and migration plans.

2026-02-21

Vector Database Guide: RAG, pgvector, Search, and Cost

Learn when RAG needs a vector database. Compare pgvector, search engines, dedicated systems, HNSW, filtering, benchmarks, migration, and total cost in practice.

2026-02-21