What is Metadata Filtering?
Updated on
Metadata Filtering is the enforcement of structured predicates over document or chunk attributes to define which candidates are eligible for retrieval.
Quick Facts
| Specification | Official Specification |
|---|
How It Works
Design Metadata as a Retrieval Contract
Define field type, allowed values, ownership, null semantics, inheritance and update behavior before indexing. Separate hard constraints such as tenant, ACL and validity from soft features such as authority or freshness boosts. Generate security predicates from an authenticated principal on the server, not from editable user input. Missing or stale authorization metadata should fail closed, while missing optional descriptive fields can follow a documented fallback.
Filter Placement Changes Search Behavior
Prefiltering searches within eligible items and favors filtered recall, but highly selective predicates may require more graph traversal or exhaustive search. Postfiltering searches first and intersects later, producing predictable ANN work but possibly fewer than k eligible results or false negatives. Azure AI Search documents these tradeoffs explicitly. Research on Filtered-DiskANN further shows that metadata-aware graph construction can improve filtered ANN efficiency, but its results depend on filter types, index design and datasets.
Test Selectivity, Security, and Freshness
Build exact filtered ground truth for representative queries, then measure Recall@K and latency across selectivity buckets, conjunction depth, shard count and update state. Include users with zero, one and many groups; revoked access; deleted documents; missing labels; version transitions; and filters that match fewer than k items. Track false negatives, unauthorized candidates before generation, stale-metadata age, empty-result rate, p95 latency and the gap between approximate and exhaustive filtered search.
Key Characteristics
- Applies typed predicates to nonvector document or chunk attributes
- Separates hard eligibility constraints from soft relevance and ranking signals
- Can run before, during, or after approximate search with different recall and latency behavior
- Depends on complete, current metadata propagated through ingestion and updates
- Supports tenant, authorization, version, region, language and lifecycle boundaries
- Requires selectivity-aware relevance, performance and negative-permission testing
Common Use Cases
- Restricting enterprise RAG evidence to documents the current principal may read
- Selecting documentation for the requested product, release and deployment region
- Excluding expired, superseded, quarantined or deleted knowledge
- Routing multilingual retrieval to eligible language and jurisdiction slices
- Combining structured catalog constraints with semantic product similarity
Example
Loading code...Frequently Asked Questions
Is metadata filtering the same as a relevance boost?
No. A filter decides eligibility: an item either belongs to the searchable set or it does not. A boost changes ordering among eligible items. Tenant, permission and legal-scope rules should normally be hard filters, not scores that another signal can override.
Should vector filters run before or after ANN search?
There is no backend-independent answer. Prefiltering usually protects filtered recall but can add traversal cost; postfiltering can be faster or more predictable but may return fewer than k results and miss eligible neighbors. Test the actual engine across filter selectivity buckets.
Can a user-supplied metadata filter enforce authorization?
No. Authorization scope must be derived from an authenticated identity and enforced server-side with deny-by-default behavior. User filters may narrow that scope but must never expand it, and authorization-service failure should not expose unfiltered candidates.
Why can a filtered query return fewer than topK results?
The eligible corpus may contain fewer than k items, or postfiltering may remove items from a bounded unfiltered window. Distinguish these cases with exhaustive ground truth, candidate counts and filter diagnostics instead of weakening a hard predicate.
How should metadata changes propagate to chunks?
Use an owned schema and versioned ingestion path that atomically updates all derived chunks. Preserve source identity, audit revocations and deletions, measure propagation lag, and test that stale replicas or caches cannot continue returning withdrawn content.