What is Document Loader?

Document Loader is a source-facing ingestion component that reads content and source metadata from files, web pages, object storage, databases, SaaS systems, or APIs and returns normalized document objects for downstream processing.

Quick Facts

SpecificationOfficial Specification

How It Works

Separate source access from content interpretation

The loader authenticates to a source, enumerates records, fetches bytes or fields, and emits a common document contract. A format-specific parser may turn PDF, HTML, or office-file bytes into text and structural elements, but that is a distinct concern: the same source connector can select different parsers, and the same parser can serve different sources. Keeping the boundary explicit prevents source retries and permissions from becoming entangled with extraction heuristics.

Make synchronization restartable and deletion-aware

A production run needs a declared snapshot or cursor, bounded concurrency, retry classification, and a durable checkpoint written only after downstream work succeeds. Stable source IDs identify logical records, while revision hashes or source versions identify content changes. The loader must also emit or reconcile tombstones for records removed from the source; upserting changed documents without deleting disappeared ones leaves stale evidence searchable.

Evaluate coverage, provenance, freshness, and isolation

A successful HTTP status or file read does not prove ingestion quality. Track discovered versus emitted records, skipped and quarantined items, duplicate source IDs, revision lag, parse handoff failures, missing provenance fields, and permission mismatches. Test tenant isolation and least-privilege credentials with adversarial fixtures. Content quality should be evaluated after parsing, while the loader is accountable for complete, timely, attributable, and authorized delivery.

Key Characteristics

  • Connects to a source and emits a stable, pipeline-neutral document contract
  • Preserves source identity, revision, provenance, media type, timestamps, and permissions
  • Separates source access and synchronization from parsing, chunking, and embedding
  • Supports cursors, pagination, bounded reads, retries, checkpoints, and partial-failure recovery
  • Propagates deletion or tombstone information instead of only appending new content
  • Exposes coverage, freshness, duplication, and authorization evidence for audits

Common Use Cases

  1. Synchronizing an internal wiki into a permission-aware RAG knowledge base
  2. Loading versioned objects and preserving URI, checksum, owner, and access metadata
  3. Reading paginated SaaS records with resumable cursors and rate-limit handling
  4. Detecting changed or deleted repository documents before reindexing
  5. Producing citation-ready source records for regulated support or policy answers

Example

loading...
Loading code...

Frequently Asked Questions

Is a Document Loader the same as a parser?

No. A loader reaches the source, handles authentication, listing, pagination, revisions, and metadata, then returns document objects. A parser interprets a particular byte or markup format. A framework may package them together, but keeping their contracts separate makes failures and replacements easier to reason about.

Which metadata should a Document Loader preserve?

At minimum, preserve a stable source ID, source URI, revision or checksum, media type, observed and source modification times, and provenance. Enterprise systems commonly also require tenant, owner, classification, and permission attributes. Record the metadata schema and reject ambiguous identities rather than silently overwriting records.

How should a loader support incremental ingestion?

Use a source cursor or snapshot to enumerate changes, compare stable IDs and revisions, and checkpoint only after dependent parsing and indexing have committed. Retries should be idempotent. Periodic full reconciliation is still needed because change feeds can expire, omit deletes, or be reset by the provider.

How should deleted source documents be handled?

Treat deletion as an explicit state transition. Emit a tombstone or compare the current source manifest with the previously indexed manifest, then remove every derived chunk and search record. Define retention and legal-hold exceptions separately; simply stopping future loads does not remove stale indexed evidence.

How do you test a Document Loader before production?

Use fixtures for pagination boundaries, duplicate IDs, oversized objects, malformed content, rate limits, permission changes, interrupted checkpoints, and deletions. Compare discovered and emitted manifests, verify deterministic revisions, and run tenant-isolation tests. Parser quality and retrieval quality require separate downstream evaluations.

Related Terms

Document Transformer

Document Transformer is a pipeline component that converts one or more loaded document objects into zero or more derived documents by cleaning, restructuring, enriching, filtering, redacting, or chunking content before indexing.

Indexer

Indexer is a pipeline component that converts processed documents into durable retrieval records and writes their text, embeddings, sparse features, metadata, or relationships to one or more searchable stores.

RAG

RAG (Retrieval-Augmented Generation) is an architecture that retrieves external evidence at inference time and supplies selected context to a generative model so its answer can be grounded, updated, and attributed.

Chunking

Chunking is the process of splitting long documents or data sources into smaller retrievable units that preserve enough semantic context for embedding, indexing, retrieval, and grounded generation.

Related Articles

Multimodal Document Pipelines: From Pixels to Evidence

Build a production multimodal document pipeline with immutable inputs, layout-aware parsing, OCR or VLM routing, evidence-linked fields, reproducible evaluation, bounded retries, secure processing, and measurable cost per accepted page.

2026-05-16

Eino RAG Pipeline: Production Indexing, Retrieval, and Evaluation

Build an Eino v0.9.21 RAG pipeline with explicit Loader, Transformer, Indexer, Retriever, and generation contracts. Learn chunk provenance, embedding compatibility, hybrid retrieval, authorization filtering, retrieval evaluation, safe rollout, and failure diagnosis through a runnable Go reference implementation.

2026-06-03

What Is RAG? From Retrieval to Verifiable LLM Answers

Understand RAG through document ingestion, retrieval, permissions, citations, and evaluation, with a runnable Go example showing how evidence reaches an answer.

2026-02-21