What is Embedding?
Embedding is a learned numerical representation that maps text, images, entities, or other inputs into a vector space where distance encodes relationships useful to a specific training objective. Vectors are comparable only under a compatible model, revision, task mode, preprocessing, dimension, normalization, and metric.
Quick Facts
| Created | 2013 by Tomas Mikolov et al. (Word2Vec) |
|---|---|
| Specification | Official Specification |
How It Works
Embedding models compress selected properties of an input into a fixed-length vector. Word2Vec and GloVe learn static word representations; transformer-based encoders can produce contextual token, sentence, document, image, or multimodal representations. The geometry is task-specific: a model trained for semantic textual similarity is not automatically optimal for asymmetric retrieval, and some retrieval models require different instructions for queries and documents.
Model selection is a workload decision, not a global leaderboard lookup. MTEB separates retrieval, reranking, classification, clustering, semantic similarity, pair classification, bitext mining, and summarization, with additional language, domain, code, image, audio, and video suites. A 2026 controlled comparison of generative LLMs and dedicated text embedders found similar aggregate quality under its 37-task protocol but materially different task strengths, throughput, and cost. That result does not prove interchangeability: compare candidates on the target task, language, corpus scale, latency path, and pricing model.
Similarity scores are meaningful only inside a compatible vector-space contract. Production systems should version the model revision, task mode, preprocessing, dimension, dtype, normalization, distance metric, source content, access policy, and index generation together. Evaluate retrieval with labeled queries and corpus-scale hard negatives; evaluate classification or clustering with task-appropriate labels and uncertainty. Report per-language and per-domain slices instead of treating a multilingual average or a small leaderboard gap as proof for local traffic.
Key Characteristics
- Produces a fixed-length vector under a specific model and output contract
- Encodes relationships selected by the model's training objective rather than every property of the source
- May use symmetric inputs or asymmetric query and document task modes
- Requires a matching normalization and distance metric for valid similarity ranking
- Trades representation size, task quality, storage, memory, throughput, and latency, so selection needs local language, domain, scale, and uncertainty slices
- Requires a separate index generation when the model, revision, preprocessing, instruction, or dimension changes
Common Use Cases
- Semantic search engines that find conceptually related content
- Retrieval-Augmented Generation (RAG) for grounding LLM responses
- Recommendation systems based on content similarity
- Document clustering and topic modeling
- Cost-aware model selection using local quality, throughput, latency, and index-size measurements
Example
Loading code...Frequently Asked Questions
What is embedding in machine learning?
Embedding is a learned numerical representation of text, images, entities, or other inputs. A model maps each input to a fixed-length vector so relationships useful to its training objective can be measured. Nearby vectors often indicate related items, but the meaning of distance is specific to the model and task contract.
What is the difference between Word2Vec and BERT embeddings?
Word2Vec produces a static vocabulary vector for each word, while BERT produces contextual token states that change with surrounding text. A raw BERT state is not automatically a sentence-retrieval embedding; sentence models add pooling and task-specific training so independently encoded texts can be compared.
How do you use embeddings for semantic search?
Embed documents and queries with compatible task modes from the same model contract, then run exact or approximate nearest-neighbor search with the documented distance metric. Apply authorization filters before exposing candidates, evaluate Recall@k against labeled queries, and add lexical retrieval or reranking when dense retrieval misses exact terms or ordering.
What is embedding dimension and how to choose it?
Embedding dimension is the number of values in one model output. A larger dimension is not a universal quality guarantee. Use only dimensions or shortening methods supported by the selected model, then compare task quality, storage, memory, latency, and uncertainty on a versioned local evaluation set before rebuilding the index.
Can embeddings from different models be compared or mixed?
Not by default. Different models, revisions, task instructions, preprocessing rules, dimensions, and normalization settings can produce incompatible vector spaces. During migration, build a separate index, encode each query with the matching model, compare results in shadow traffic, and switch only after quality and rollback checks pass.