Embedding-Based Content Analysis: Clustering and Dedup

Embedding-based content analysis uses learned vector embeddings to organize, compare, and route large content collections. It can support text clustering, supervised classification, semantic duplicate detection, topic discovery, recommendation, and corpus auditing, but an embedding is only a feature. It does not define the business meaning of a cluster, prove that two items are duplicates, or authorize content to be compared.

The vector embeddings guide owns model selection, vector-space contracts, indexing, and migration. This article owns the downstream analysis problem: how to turn a compatible embedding matrix into reviewed, measurable content decisions.

Direct Answer

A production content-analysis pipeline needs five explicit contracts:

1. Content identity: immutable item ID, source version, language, tenant, and lifecycle.
2. Representation: model revision, task mode, preprocessing, dimensions, normalization, and metric.
3. Analysis task: clustering, classification, deduplication, topic discovery, or similarity retrieval.
4. Decision policy: thresholds, abstention, review, merge authority, and rollback.
5. Evaluation: labels, metrics, slices, drift tests, and downstream success criteria.

Separate the Four Analysis Tasks

The same vectors can feed different algorithms, but the tasks have different outputs and evaluation rules.

Clustering and classification are not interchangeable. If the business already has an approved taxonomy, supervised classification is easier to test and operate. If the taxonomy is unknown or stale, clustering can reveal candidate structure, but reviewers must decide whether that structure is useful.

Semantic deduplication is also narrower than semantic search. Similar documents may discuss the same event from different viewpoints, while duplicates may differ in wording. A merge policy therefore needs more than nearest-neighbor distance.

Build a Versioned Embedding Contract

Two vectors are comparable only when they belong to a compatible representation space. Store the complete contract beside every item:

{
  "contentId": "article-2048",
  "sourceRevision": "sha256:...",
  "language": "en",
  "tenantId": "tenant-a",
  "embedding": {
    "modelId": "approved/model",
    "modelRevision": "immutable-revision",
    "task": "clustering",
    "preprocessingRevision": "content-v3",
    "dimensions": 768,
    "normalization": "l2",
    "metric": "cosine"
  },
  "analysisRelease": "content-intelligence-7"
}

Do not mix vectors produced by different model revisions, instructions, task modes, truncation rules, or normalization policies. Build a separate generation and replay the analysis before switching. The embedding migration guide explains the dual-index lifecycle in detail.

The representation unit also matters. A short title, a paragraph, and a full article encode different information. For article-level clustering, a single long vector can hide minority topics. For paragraph-level analysis, one article can appear in many clusters. Define whether decisions apply to documents, sections, claims, products, tickets, or another stable unit.

Choose the Right Analysis Path

Clustering for corpus exploration

K-means is a useful baseline when the number of groups is known and roughly spherical, similarly sized clusters are plausible. It minimizes within-cluster squared distance and therefore carries geometric assumptions. HDBSCAN or another density-based method can identify outliers and uneven groups, but high-dimensional behavior and parameter sensitivity still require validation.

Agglomerative clustering is useful for smaller corpora and hierarchical inspection. A threshold can produce coarse or fine groups without fixing the count in advance, but pairwise work becomes expensive as the collection grows.

No algorithm discovers the one true taxonomy. Compare candidates against the operational question:

• Are reviewers trying to discover themes?
• Must every item receive a group?
• Are outliers valuable incidents or noise?
• Should groups remain stable as new content arrives?
• Is the output exploratory or a production routing decision?

Classification for a stable taxonomy

Classification assigns one or more known labels. You can train a classifier on embeddings, compare content to labeled prototypes, or fine-tune a task model. The production contract should include:

• label definitions and mutually exclusive or multi-label rules;
• a frozen test set plus difficult and novel examples;
• per-class precision and recall rather than accuracy alone;
• an abstention or unknown path;
• taxonomy and model versions;
• review and relabeling procedures.

Embedding similarity to a class description is a baseline, not calibrated probability. If a decision triggers moderation, publishing, billing, or access changes, use an independently validated classifier and deterministic policy controls.

Semantic deduplication for candidate generation

Deduplication should be a staged decision:

1. Exact hashes catch byte-identical content.
2. Canonicalization catches safe formatting variants.
3. Lexical fingerprints catch near-copy edits.
4. Embeddings generate semantic candidate pairs.
5. Deterministic rules or reviewers decide whether a merge is allowed.

Candidate generation and merge authority must remain separate. Keep both source IDs, provenance, timestamps, rights, and inbound links until the decision is approved. Never delete an item solely because its cosine similarity exceeds a borrowed threshold.

Calibrate Similarity Instead of Guessing

Cosine similarity is a model-specific ranking signal. Its distribution changes with language, content length, preprocessing, corpus, and model revision. Calibrate a duplicate threshold with labeled pairs:

For each threshold, record:

• precision among proposed duplicates;
• recall over known duplicates;
• false-merge rate;
• missed-duplicate rate;
• review volume and time;
• results by language, source, length, and content type.

Optimize for the cost of the error. A false merge can destroy unique content, links, attribution, or legal records; a missed duplicate may only add storage or editorial work. Those costs rarely justify the same threshold.

Runnable Go Pipeline

The following standard-library program validates normalized vectors, computes cosine similarity, and returns deterministic thresholded duplicate candidates. It deliberately accepts vectors as input so the analysis layer stays independent from any embedding provider.

package main

import (
	"errors"
	"fmt"
	"math"
	"sort"
)

type Item struct {
	ID     string
	Vector []float64
}

type Pair struct {
	Left       string
	Right      string
	Similarity float64
}

func normalize(vector []float64) ([]float64, error) {
	if len(vector) == 0 {
		return nil, errors.New("vector must not be empty")
	}

	var squared float64
	for _, value := range vector {
		if math.IsNaN(value) || math.IsInf(value, 0) {
			return nil, errors.New("vector contains a non-finite value")
		}
		squared += value * value
	}
	if squared == 0 {
		return nil, errors.New("zero vector cannot be normalized")
	}

	norm := math.Sqrt(squared)
	result := make([]float64, len(vector))
	for index, value := range vector {
		result[index] = value / norm
	}
	return result, nil
}

func cosine(left, right []float64) (float64, error) {
	if len(left) != len(right) {
		return 0, errors.New("vector dimensions do not match")
	}

	var score float64
	for index := range left {
		score += left[index] * right[index]
	}
	return score, nil
}

func duplicateCandidates(items []Item, threshold float64) ([]Pair, error) {
	if threshold < -1 || threshold > 1 {
		return nil, errors.New("threshold must be between -1 and 1")
	}

	normalized := make([]Item, len(items))
	for index, item := range items {
		vector, err := normalize(item.Vector)
		if err != nil {
			return nil, fmt.Errorf("%s: %w", item.ID, err)
		}
		normalized[index] = Item{ID: item.ID, Vector: vector}
	}

	var pairs []Pair
	for left := 0; left < len(normalized); left++ {
		for right := left + 1; right < len(normalized); right++ {
			score, err := cosine(
				normalized[left].Vector,
				normalized[right].Vector,
			)
			if err != nil {
				return nil, err
			}
			if score >= threshold {
				pairs = append(pairs, Pair{
					Left:       normalized[left].ID,
					Right:      normalized[right].ID,
					Similarity: score,
				})
			}
		}
	}

	sort.Slice(pairs, func(i, j int) bool {
		if pairs[i].Similarity == pairs[j].Similarity {
			return pairs[i].Left < pairs[j].Left
		}
		return pairs[i].Similarity > pairs[j].Similarity
	})
	return pairs, nil
}

func main() {
	items := []Item{
		{ID: "article-a", Vector: []float64{0.90, 0.10, 0.05}},
		{ID: "article-b", Vector: []float64{0.86, 0.13, 0.07}},
		{ID: "article-c", Vector: []float64{0.05, 0.92, 0.20}},
	}

	pairs, err := duplicateCandidates(items, 0.98)
	if err != nil {
		panic(err)
	}
	for _, pair := range pairs {
		fmt.Printf("%s %s %.4f\n", pair.Left, pair.Right, pair.Similarity)
	}
	// Candidate only. A policy or reviewer still decides whether to merge.
}

For large corpora, replace the quadratic pair scan with a retriever designed for dense retrieval, backed by an approximate-nearest-neighbor index or vector database. Keep the same validation, threshold calibration, and review contract. Evaluate ANN candidate recall against an exact subset so speed does not silently remove duplicates.

Name Topics Without Rewriting Evidence

A cluster ID has no inherent meaning. Topic naming is a separate annotation step:

1. Select representative items and boundary examples.
2. Extract high-support terms, entities, and source metadata.
3. Ask a reviewer or model to propose a short label and description.
4. Store the proposal, evidence IDs, model or reviewer identity, and confidence.
5. Allow split, merge, rename, and mixed outcomes.

Generated topic names must never overwrite source content or become ground truth without review. A label such as “billing failures” may hide two operationally distinct groups: payment-provider outages and customer card declines. Review the members, not just a word cloud or model summary.

Evaluate Clustering, Classification, and Dedup Separately

Clustering

When trusted categories exist, compare cluster assignments with adjusted Rand index, normalized mutual information, or homogeneity/completeness measures. When they do not, use multiple signals:

• silhouette or related internal diagnostics;
• stability across random seeds, samples, and model revisions;
• percentage and quality of outliers;
• reviewer agreement on sampled members;
• cluster usefulness for discovery or routing;
• downstream search, recommendation, or editorial outcomes.

Two-dimensional PCA or UMAP plots are exploratory views. Projection can create or hide separation and must not be used as the release gate.

Classification

Report per-class precision, recall, F1, confusion pairs, abstention quality, calibration, and results for rare, multilingual, and new-topic slices. A high aggregate score can hide a class that never works.

Deduplication

Use pairwise precision and recall, false merges, missed duplicates, canonical-selection errors, and review effort. Test different publication dates, templates, syndicated sources, quotations, translations, and documents that share boilerplate but contain distinct claims.

Monitor Drift and Lifecycle

Content analysis changes even when application code does not. Monitor:

• new source and language distributions;
• embedding model or preprocessing revisions;
• similarity-score and cluster-size distributions;
• outlier and abstention rates;
• label prevalence and reviewer overrides;
• duplicate-review acceptance rate;
• deleted or restricted records remaining in derived artifacts.

Version cluster assignments, labels, duplicate decisions, and topic names as derived data. When a source is corrected, deleted, or reclassified, propagate the lifecycle change to embeddings, indexes, analysis tables, caches, exports, and evaluation fixtures.

Security and Governance

Embeddings are derived data, not anonymous data. Apply the same classification and access rules as the source:

• derive tenant and policy scope from authenticated identity;
• do not compare restricted corpora unless policy explicitly permits it;
• validate and authenticate sources before embedding;
• test poisoning, hidden instructions, and adversarial near-duplicates;
• restrict bulk vector export and nearest-neighbor enumeration;
• retain provenance for every cluster member and duplicate decision;
• use human approval for destructive merges or high-impact labels.

OWASP identifies unauthorized access, cross-context leakage, inversion, and poisoning as embedding risks. Similarity can organize evidence, but it cannot replace access control, factual verification, or editorial responsibility.

Production Checklist

• [ ] The content unit and source identity are stable.
• [ ] The embedding and preprocessing contract is versioned.
• [ ] Clustering, classification, and deduplication have separate objectives.
• [ ] Thresholds are calibrated on labeled workload pairs.
• [ ] Unknown, outlier, and review paths exist.
• [ ] Cluster names retain representative evidence.
• [ ] Evaluation includes slices and error costs.
• [ ] Authorization applies before candidate comparison.
• [ ] Updates and deletions propagate to all derived records.
• [ ] Model and policy changes use replay, canary, and rollback.

Sources and Further Reading

• MTEB: Massive Text Embedding Benchmark - embedding tasks differ, and no single evaluated method dominates all tasks.
• Sentence Transformers: Clustering - official examples and boundaries for k-means, agglomerative, fast clustering, and topic modeling.
• scikit-learn Clustering Guide - algorithm assumptions, scalability, and evaluation methods.
• Text Clustering with Large Language Model Embeddings - comparative research on embeddings and clustering methods.
• OWASP Vector and Embedding Weaknesses - access, leakage, inversion, and poisoning risks.
• Vector Embeddings Engineering Guide - model contracts, retrieval evaluation, migration, and lifecycle.
• Semantic Search Engineering Guide - authorized retrieval, ranking, and search evaluation.

Summary

Embedding-based content analysis is not one algorithm. It is a governed family of clustering, classification, semantic deduplication, and topic-discovery decisions built on a versioned representation. Keep candidate generation separate from destructive actions, calibrate every threshold on the real workload, review the evidence behind cluster names, and measure each task with its own errors and outcomes.