What is Embedding?
Embedding is a learned numerical representation that maps text, images, entities, or other inputs into a vector space where distance encodes relationships useful to a specific training objective. Vectors are comparable only under a compatible model, revision, task mode, preprocessing, dimension, normalization, and metric.
Quick Facts
| Created | 2013 by Tomas Mikolov et al. (Word2Vec) |
|---|---|
| Specification | Official Specification |
How It Works
Embedding models compress selected properties of an input into a fixed-length vector. Word2Vec and GloVe learn static word representations; transformer-based encoders can produce contextual token, sentence, document, image, or multimodal representations. The geometry is task-specific: a model trained for semantic textual similarity is not automatically optimal for asymmetric retrieval, and some retrieval models require different instructions for queries and documents. Similarity scores are meaningful only inside a compatible vector-space contract. Production systems therefore version the model revision, task mode, preprocessing, dimension, dtype, normalization, distance metric, source content, access policy, and index generation together.
Key Characteristics
- Produces a fixed-length vector under a specific model and output contract
- Encodes relationships selected by the model's training objective rather than every property of the source
- May use symmetric inputs or asymmetric query and document task modes
- Requires a matching normalization and distance metric for valid similarity ranking
- Trades representation size, retrieval quality, storage, memory, and latency in a model-specific way
- Requires a separate index generation when the model, revision, preprocessing, or dimension changes
Common Use Cases
- Semantic search engines that find conceptually related content
- Retrieval-Augmented Generation (RAG) for grounding LLM responses
- Recommendation systems based on content similarity
- Document clustering and topic modeling
- Anomaly detection through distance-based outlier identification
Example
Loading code...Frequently Asked Questions
What is embedding in machine learning?
Embedding is a learned numerical representation of text, images, entities, or other inputs. A model maps each input to a fixed-length vector so relationships useful to its training objective can be measured. Nearby vectors often indicate related items, but the meaning of distance is specific to the model and task contract.
What is the difference between Word2Vec and BERT embeddings?
Word2Vec produces a static vocabulary vector for each word, while BERT produces contextual token states that change with surrounding text. A raw BERT state is not automatically a sentence-retrieval embedding; sentence models add pooling and task-specific training so independently encoded texts can be compared.
How do you use embeddings for semantic search?
Embed documents and queries with compatible task modes from the same model contract, then run exact or approximate nearest-neighbor search with the documented distance metric. Apply authorization filters before exposing candidates, evaluate Recall@k against labeled queries, and add lexical retrieval or reranking when dense retrieval misses exact terms or ordering.
What is embedding dimension and how to choose it?
Embedding dimension is the number of values in one model output. A larger dimension is not a universal quality guarantee. Use only dimensions or shortening methods supported by the selected model, then compare retrieval quality, storage, memory, and latency on a versioned evaluation set before rebuilding the index.
Can embeddings from different models be compared or mixed?
Not by default. Different models, revisions, task instructions, preprocessing rules, dimensions, and normalization settings can produce incompatible vector spaces. During migration, build a separate index, encode each query with the matching model, compare results in shadow traffic, and switch only after quality and rollback checks pass.