Embedding-Based Content Analysis: Clustering and Dedup

Embedding-based content analysis uses learned vector embeddings to organize, compare, and route large content collections. It can support text clustering, supervised classification, semantic duplicate detection, topic discovery, recommendation, and corpus auditing, but an embedding is only a feature. It does not define the business meaning of a cluster, prove that two items are duplicates, or authorize content to be compared.

The vector embeddings guide owns model selection, vector-space contracts, indexing, and migration. This article owns the downstream analysis problem: how to turn a compatible embedding matrix into reviewed, measurable content decisions.

Direct Answer

A production content-analysis pipeline needs five explicit contracts:

1. Content identity: immutable item ID, source version, language, tenant, and lifecycle.
2. Representation: model revision, task mode, preprocessing, dimensions, normalization, and metric.
3. Analysis task: clustering, classification, deduplication, topic discovery, or similarity retrieval.
4. Decision policy: thresholds, abstention, review, merge authority, and rollback.
5. Evaluation: labels, metrics, slices, drift tests, and downstream success criteria.

Separate the Four Analysis Tasks

The same vectors can feed different algorithms, but the tasks have different outputs and evaluation rules.

Clustering and classification are not interchangeable. If the business already has an approved taxonomy, supervised classification is easier to test and operate. If the taxonomy is unknown or stale, clustering can reveal candidate structure, but reviewers must decide whether that structure is useful.

Semantic deduplication is also narrower than semantic search. Similar documents may discuss the same event from different viewpoints, while duplicates may differ in wording. A merge policy therefore needs more than nearest-neighbor distance.

Select Models by Task Language and Cost

An embedding model is not “best” independently of the operation it supports. The 2026 preprint The Embedder's Dilemma compared LLM representations with dedicated embedding models across classification, semantic similarity, clustering, pair classification, and retrieval. In that study, the best aggregate scores were close, but the category results diverged: dedicated embedders led classification, LLMs led the tested reasoning-heavy retrieval tasks, and clustering was statistically tied. The authors also reported a large cost and throughput penalty for LLM representations.

Those results are evidence for task-specific selection, not a new universal ranking. Build a decision table for the actual workload:

MTEB's current catalog exposes separate task and language benchmarks rather than one interchangeable score. Native-language suites reinforce that distinction. For example, MTEB-BR evaluated 93 models on 22 native Brazilian Portuguese tasks and found that the top six were statistically too close to order, while agreement with a global multilingual leaderboard was only moderate for shared models. The transferable lesson is to report uncertainty and local slices; the specific Portuguese ranking does not establish Chinese or English production quality.

For multilingual content, evaluate at least monolingual English, monolingual Chinese, code-switched text, translations, names, numbers, and locale-specific near-duplicates. A single threshold across all slices is acceptable only after the labeled error curves show that its false-merge cost is tolerable.

Build a Versioned Embedding Contract

Two vectors are comparable only when they belong to a compatible representation space. Store the complete contract beside every item:

{
  "contentId": "article-2048",
  "sourceRevision": "sha256:...",
  "language": "en",
  "tenantId": "tenant-a",
  "embedding": {
    "modelId": "approved/model",
    "modelRevision": "immutable-revision",
    "task": "clustering",
    "preprocessingRevision": "content-v3",
    "dimensions": 768,
    "normalization": "l2",
    "metric": "cosine"
  },
  "analysisRelease": "content-intelligence-7"
}

Do not mix vectors produced by different model revisions, instructions, task modes, truncation rules, or normalization policies. Build a separate generation and replay the analysis before switching. The embedding migration guide explains the dual-index lifecycle in detail.

The representation unit also matters. A short title, a paragraph, and a full article encode different information. For article-level clustering, a single long vector can hide minority topics. For paragraph-level analysis, one article can appear in many clusters. Define whether decisions apply to documents, sections, claims, products, tickets, or another stable unit.

Choose the Right Analysis Path

Clustering for corpus exploration

K-Means Clustering is a useful baseline when the number of groups is known and roughly spherical, similarly sized clusters are plausible. It minimizes within-cluster squared distance and therefore carries geometric assumptions. DBSCAN, HDBSCAN, or another density-based method can identify outliers and uneven groups, but high-dimensional behavior and parameter sensitivity still require validation.

Hierarchical Clustering is useful for smaller corpora and nested inspection. A threshold can produce coarse or fine groups without fixing the count in advance, but pairwise work becomes expensive as the collection grows.

A Gaussian Mixture Model adds soft responsibilities and explicit covariance assumptions, while Spectral Clustering uses a similarity graph to expose non-convex connectivity. Both add modeling choices rather than removing the need for validation.

No algorithm discovers the one true taxonomy. Compare candidates against the operational question:

• Are reviewers trying to discover themes?
• Must every item receive a group?
• Are outliers valuable incidents or noise?
• Should groups remain stable as new content arrives?
• Is the output exploratory or a production routing decision?

Classification for a stable taxonomy

Classification assigns one or more known labels. You can train a classifier on embeddings, compare content to labeled prototypes, or fine-tune a task model. The production contract should include:

• label definitions and mutually exclusive or multi-label rules;
• a frozen test set plus difficult and novel examples;
• per-class precision and recall rather than accuracy alone;
• an abstention or unknown path;
• taxonomy and model versions;
• review and relabeling procedures.

Embedding similarity to a class description is a baseline, not calibrated probability. If a decision triggers moderation, publishing, billing, or access changes, use an independently validated classifier and deterministic policy controls.

Semantic deduplication for candidate generation

Deduplication should be a staged decision:

1. Exact hashes catch byte-identical content.
2. Canonicalization catches safe formatting variants.
3. Lexical fingerprints catch near-copy edits.
4. Embeddings generate semantic candidate pairs.
5. Deterministic rules or reviewers decide whether a merge is allowed.

Candidate generation and merge authority must remain separate. Keep both source IDs, provenance, timestamps, rights, and inbound links until the decision is approved. Never delete an item solely because its cosine similarity exceeds a borrowed threshold.

Calibrate Similarity Instead of Guessing

Cosine similarity is a model-specific ranking signal. Its distribution changes with language, content length, preprocessing, corpus, and model revision. Calibrate a duplicate threshold with labeled pairs:

For each threshold, record:

• precision among proposed duplicates;
• recall over known duplicates;
• false-merge rate;
• missed-duplicate rate;
• review volume and time;
• results by language, source, length, and content type.

Optimize for the cost of the error. A false merge can destroy unique content, links, attribution, or legal records; a missed duplicate may only add storage or editorial work. Those costs rarely justify the same threshold.

Runnable Go Pipeline

The following standard-library program validates normalized vectors, computes cosine similarity, and returns deterministic thresholded duplicate candidates. It deliberately accepts vectors as input so the analysis layer stays independent from any embedding provider.

package main

import (
	"errors"
	"fmt"
	"math"
	"sort"
)

type Item struct {
	ID       string
	Contract Contract
	Vector   []float64
}

type Contract struct {
	ModelRevision string
	Task          string
	Language      string
	Dimensions    int
}

type Pair struct {
	Left       string
	Right      string
	Similarity float64
}

func normalize(vector []float64) ([]float64, error) {
	if len(vector) == 0 {
		return nil, errors.New("vector must not be empty")
	}

	var squared float64
	for _, value := range vector {
		if math.IsNaN(value) || math.IsInf(value, 0) {
			return nil, errors.New("vector contains a non-finite value")
		}
		squared += value * value
	}
	if squared == 0 {
		return nil, errors.New("zero vector cannot be normalized")
	}

	norm := math.Sqrt(squared)
	result := make([]float64, len(vector))
	for index, value := range vector {
		result[index] = value / norm
	}
	return result, nil
}

func cosine(left, right []float64) (float64, error) {
	if len(left) != len(right) {
		return 0, errors.New("vector dimensions do not match")
	}

	var score float64
	for index := range left {
		score += left[index] * right[index]
	}
	return score, nil
}

func duplicateCandidates(items []Item, threshold float64) ([]Pair, error) {
	if threshold < -1 || threshold > 1 {
		return nil, errors.New("threshold must be between -1 and 1")
	}
	if len(items) < 2 {
		return nil, errors.New("at least two items are required")
	}

	normalized := make([]Item, len(items))
	contract := items[0].Contract
	if contract.ModelRevision == "" || contract.Task == "" ||
		contract.Language == "" || contract.Dimensions <= 0 {
		return nil, errors.New("embedding contract is incomplete")
	}
	seen := make(map[string]struct{}, len(items))
	for index, item := range items {
		if item.ID == "" {
			return nil, errors.New("item ID must not be empty")
		}
		if _, exists := seen[item.ID]; exists {
			return nil, fmt.Errorf("duplicate item ID %q", item.ID)
		}
		seen[item.ID] = struct{}{}
		if item.Contract != contract {
			return nil, fmt.Errorf("%s: incompatible embedding contract", item.ID)
		}
		if len(item.Vector) != contract.Dimensions {
			return nil, fmt.Errorf("%s: vector dimensions do not match contract", item.ID)
		}
		vector, err := normalize(item.Vector)
		if err != nil {
			return nil, fmt.Errorf("%s: %w", item.ID, err)
		}
		normalized[index] = Item{
			ID:       item.ID,
			Contract: item.Contract,
			Vector:   vector,
		}
	}

	var pairs []Pair
	for left := 0; left < len(normalized); left++ {
		for right := left + 1; right < len(normalized); right++ {
			score, err := cosine(
				normalized[left].Vector,
				normalized[right].Vector,
			)
			if err != nil {
				return nil, err
			}
			if score >= threshold {
				pairs = append(pairs, Pair{
					Left:       normalized[left].ID,
					Right:      normalized[right].ID,
					Similarity: score,
				})
			}
		}
	}

	sort.Slice(pairs, func(i, j int) bool {
		if pairs[i].Similarity == pairs[j].Similarity {
			return pairs[i].Left < pairs[j].Left
		}
		return pairs[i].Similarity > pairs[j].Similarity
	})
	return pairs, nil
}

func main() {
	contract := Contract{
		ModelRevision: "embedder-sha256",
		Task:          "semantic-deduplication",
		Language:      "en",
		Dimensions:    3,
	}
	items := []Item{
		{ID: "article-a", Contract: contract, Vector: []float64{0.90, 0.10, 0.05}},
		{ID: "article-b", Contract: contract, Vector: []float64{0.86, 0.13, 0.07}},
		{ID: "article-c", Contract: contract, Vector: []float64{0.05, 0.92, 0.20}},
	}

	pairs, err := duplicateCandidates(items, 0.98)
	if err != nil {
		panic(err)
	}
	for _, pair := range pairs {
		fmt.Printf("%s %s %.4f\n", pair.Left, pair.Right, pair.Similarity)
	}
	// Candidate only. A policy or reviewer still decides whether to merge.
}

For large corpora, replace the quadratic pair scan with a retriever designed for dense retrieval, backed by an approximate-nearest-neighbor index or vector database. Keep the same validation, threshold calibration, and review contract. Evaluate ANN candidate recall against an exact subset so speed does not silently remove duplicates.

Name Topics Without Rewriting Evidence

A cluster ID has no inherent meaning. Topic naming is a separate annotation step:

1. Select representative items and boundary examples.
2. Extract high-support terms, entities, and source metadata.
3. Ask a reviewer or model to propose a short label and description.
4. Store the proposal, evidence IDs, model or reviewer identity, and confidence.
5. Allow split, merge, rename, and mixed outcomes.

Generated topic names must never overwrite source content or become ground truth without review. A label such as “billing failures” may hide two operationally distinct groups: payment-provider outages and customer card declines. Review the members, not just a word cloud or model summary.

Evaluate Clustering, Classification, and Dedup Separately

Clustering

When trusted categories exist, compare cluster assignments with Adjusted Rand Index, Normalized Mutual Information, or homogeneity/completeness measures. When they do not, use multiple signals:

• Silhouette Score for sample-level cohesion and separation, plus Davies-Bouldin Index and Calinski-Harabasz Index as complementary centroid-based diagnostics;
• stability across random seeds, samples, and model revisions;
• percentage and quality of outliers;
• reviewer agreement on sampled members;
• cluster usefulness for discovery or routing;
• downstream search, recommendation, or editorial outcomes.

Establish a compression baseline before interpreting a plot. Truncated SVD is suitable for uncentered sparse matrices, while Random Projection provides a data-independent distance-preserving baseline. Neither guarantees that the retained coordinates encode the taxonomy being evaluated.

For nonlinear geometry, Manifold Learning is a family of competing assumptions rather than one algorithm. Kernel PCA preserves feature-space variance, and Multidimensional Scaling fits supplied dissimilarities. Two-dimensional PCA, t-SNE, UMAP, Isomap, or LLE views preserve still other quantities. Projection can create or hide separation and must not be used as the release gate.

A Gaussian Process Latent Variable Model takes a different, generative route: it learns latent coordinates whose GP covariance explains the observed dimensions. Its axes are non-identifiable and a new item needs explicit latent inference, so compare held-out predictive density, reconstruction, neighborhood stability, and downstream utility rather than reading semantic meaning from plot orientation.

Graph methods answer narrower questions. Laplacian Eigenmaps minimizes variation across affinity edges, while Diffusion Maps compares multi-step random-walk distributions. Hessian Eigenmaps targets a second-order null space, and Local Tangent Space Alignment aligns overlapping local charts. Estimate Intrinsic Dimension across scales before fixing an output size, and treat disagreement across scales or estimators as evidence rather than tuning noise.

Kernel methods start with a Reproducing Kernel Hilbert Space, where point evaluation is an inner product. The Kernel Trick lets compatible algorithms use that geometry through pairwise evaluations; the RBF Kernel supplies one localized geometry, while Kernel Ridge Regression applies it to regularized supervised prediction.

Support Vector Machines pair maximum-margin classification with Hinge Loss, while Support Vector Regression replaces exact residual fitting with an epsilon-insensitive tube. One-Class SVM estimates an acceptance region from reference data rather than learning a second class. Multiple Kernel Learning can combine several valid geometries, but every base kernel, normalization, and learned weight must remain inside the model-selection protocol.

Kernel methods also support distribution and dependence diagnostics. A Kernel Mean Embedding represents a distribution through RKHS expectations; Maximum Mean Discrepancy compares two such distributions, while HSIC tests whether two paired variables remain dependent under selected kernels. These statistics require explicit estimator, bandwidth, sampling, and null-calibration choices.

For large datasets, the Nyström Method builds a data-dependent low-rank basis from landmark columns, while Random Fourier Features approximate eligible shift-invariant kernels with a frozen spectral map. Both must be checked on the actual decision. When comparing neural activations rather than data distributions, Centered Kernel Alignment summarizes paired representation geometry; it does not establish functional or causal equivalence.

Report uncertainty across items and random seeds. When two candidate models have overlapping confidence intervals or unstable cluster assignments, call the result inconclusive instead of creating a precise ranking from noise.

Classification

Report per-class precision, recall, F1, confusion pairs, abstention quality, calibration, and results for rare, multilingual, and new-topic slices. A high aggregate score can hide a class that never works.

Deduplication

Use pairwise precision and recall, false merges, missed duplicates, canonical-selection errors, and review effort. Test different publication dates, templates, syndicated sources, quotations, translations, and documents that share boilerplate but contain distinct claims.

Monitor Drift and Lifecycle

Content analysis changes even when application code does not. Monitor:

• new source and language distributions;
• embedding model or preprocessing revisions;
• similarity-score and cluster-size distributions;
• outlier and abstention rates;
• label prevalence and reviewer overrides;
• duplicate-review acceptance rate;
• deleted or restricted records remaining in derived artifacts.

Version cluster assignments, labels, duplicate decisions, and topic names as derived data. When a source is corrected, deleted, or reclassified, propagate the lifecycle change to embeddings, indexes, analysis tables, caches, exports, and evaluation fixtures.

Security and Governance

Embeddings are derived data, not anonymous data. Apply the same classification and access rules as the source:

• derive tenant and policy scope from authenticated identity;
• do not compare restricted corpora unless policy explicitly permits it;
• validate and authenticate sources before embedding;
• test poisoning, hidden instructions, and adversarial near-duplicates;
• restrict bulk vector export and nearest-neighbor enumeration;
• retain provenance for every cluster member and duplicate decision;
• use human approval for destructive merges or high-impact labels.

OWASP identifies unauthorized access, cross-context leakage, inversion, and poisoning as embedding risks. Similarity can organize evidence, but it cannot replace access control, factual verification, or editorial responsibility.

Production Checklist

• [ ] The content unit and source identity are stable.
• [ ] The embedding and preprocessing contract is versioned.
• [ ] Clustering, classification, and deduplication have separate objectives.
• [ ] Thresholds are calibrated on labeled workload pairs.
• [ ] Unknown, outlier, and review paths exist.
• [ ] Cluster names retain representative evidence.
• [ ] Evaluation includes slices and error costs.
• [ ] Authorization applies before candidate comparison.
• [ ] Updates and deletions propagate to all derived records.
• [ ] Model and policy changes use replay, canary, and rollback.

Sources and Further Reading

• MTEB: Massive Text Embedding Benchmark - embedding tasks differ, and no single evaluated method dominates all tasks.
• MTEB available benchmarks - current task, language, and domain-specific benchmark catalog.
• The Embedder's Dilemma - task-level comparison of dedicated embedders and LLM representations, including cost and uncertainty.
• MTEB-BR - native-language benchmark design, confidence intervals, and cross-leaderboard disagreement.
• Sentence Transformers: Clustering - official examples and boundaries for k-means, agglomerative, fast clustering, and topic modeling.
• scikit-learn Clustering Guide - algorithm assumptions, scalability, and evaluation methods.
• Text Clustering with Large Language Model Embeddings - comparative research on embeddings and clustering methods.
• OWASP Vector and Embedding Weaknesses - access, leakage, inversion, and poisoning risks.
• Vector Embeddings Engineering Guide - model contracts, retrieval evaluation, migration, and lifecycle.
• Semantic Search Engineering Guide - authorized retrieval, ranking, and search evaluation.

Summary

Embedding-based content analysis is not one algorithm. It is a governed family of clustering, classification, semantic deduplication, and topic-discovery decisions built on a versioned representation. Keep candidate generation separate from destructive actions, calibrate every threshold on the real workload, review the evidence behind cluster names, and measure each task with its own errors and outcomes.