What is Document Transformer?
Updated on
Document Transformer is a pipeline component that converts one or more loaded document objects into zero or more derived documents by cleaning, restructuring, enriching, filtering, redacting, or chunking content before indexing.
Quick Facts
| Specification | Official Specification |
|---|
How It Works
Treat transformation as an ordered document graph
Normalization, boilerplate removal, structural extraction, redaction, enrichment, deduplication, and chunking are not interchangeable steps. Removing layout before table extraction can destroy meaning; chunking before redaction can replicate sensitive text; enriching after ID generation can make identity ignore retrieval-relevant fields. Define the order, make each stage return new document objects, and preserve the parent source ID, source revision, page or section locator, and applied-transform versions.
Use deterministic identity and cache boundaries
A derived document needs a stable ID based on declared inputs such as parent ID, source revision, transform version, chunk ordinal, and normalized content. Deterministic IDs make retries idempotent and allow a pipeline cache to reuse unchanged outputs. When a parser, splitter, embedding model, or metadata schema changes, invalidate only the affected stage and build a new index version rather than mutating an untraceable corpus in place.
Evaluate the transformation before blaming retrieval
Inspect retained text coverage, heading and table preservation, chunk-length distribution, empty or duplicate outputs, lineage completeness, sensitive-data leakage, and deterministic rebuilds. Then run retrieval evaluation on a fixed query set. A better retrieval score does not excuse lost permissions or citations, and a deterministic pipeline does not prove useful chunk boundaries; quality, governance, and reproducibility are separate release gates.
Key Characteristics
- Consumes document objects after loading and emits zero, one, or many derived documents
- Applies an explicit order to cleaning, extraction, redaction, enrichment, and chunking
- Preserves parent identity, source revision, locators, permissions, and transform versions
- Uses deterministic output IDs so retries, caching, comparison, and rollback are reliable
- Separates transformation failures from source-access and durable-index failures
- Requires both corpus-level quality checks and downstream retrieval evaluation
Common Use Cases
- Splitting a policy manual by semantic headings while retaining section and page locators
- Removing navigation and repeated boilerplate from crawled pages before indexing
- Redacting sensitive fields before text is copied into chunks or embeddings
- Extracting tables into retrieval units without discarding row and source context
- Rebuilding only outputs affected by a parser, chunker, or metadata-schema change
Example
Loading code...Frequently Asked Questions
How is a Document Transformer different from a Document Loader?
A loader reaches an external source and creates source-faithful document objects. A transformer consumes those objects and creates cleaned, enriched, filtered, or split derivatives. Keeping the contracts separate lets teams retry source access without repeating every transformation and replace parsing or chunking without rewriting the connector.
In what order should document transformations run?
There is no universal order, but dependencies must be explicit. Structural extraction generally needs original layout; redaction must occur before sensitive text is replicated into chunks or embeddings; chunking needs the structure it uses for boundaries. Version the ordered pipeline and test the actual corpus instead of assuming every transformation commutes.
Should a transformer mutate the input document?
Prefer immutable outputs or defensive copies. In-place mutation can leak metadata changes across parallel stages, caches, and retries, making lineage difficult to audit. If a framework permits mutation for performance, document ownership rules and test that concurrent execution cannot observe partially transformed state.
How should transformed chunks be identified?
Derive stable IDs from declared identity inputs such as parent document ID, source revision, transform version, chunk ordinal, and normalized content. Do not use array position alone across changing source revisions. Store the parent locator separately so citations and deletions can resolve every chunk back to its source.
How do you evaluate a Document Transformer?
Measure retained-content coverage, structure and table preservation, duplicate and empty outputs, chunk distributions, lineage completeness, redaction leakage, and deterministic rebuilds. Use a fixed query and relevance set to assess retrieval impact, then inspect representative failures rather than accepting one aggregate score.