Vector Database Guide: RAG, pgvector, Search, and Cost

A vector database stores embeddings and retrieves nearby items for semantic search, recommendations, and RAG. You need one when similarity retrieval is a real requirement, but not necessarily a separate product: PostgreSQL with pgvector, a search engine, or an existing data platform may be sufficient. This guide compares those options and covers index trade-offs, filtering, tenant isolation, evaluation, and cost.

📋 Table of Contents

• TL;DR Key Takeaways
• What is a Vector Database
• Vector Indexing Algorithms Explained
• Mainstream Vector Database Comparison
• Selection Guide
• Practical Code Examples
• Vector Databases in RAG Applications
• Performance Optimization Best Practices
• FAQ
• Summary

TL;DR Key Takeaways

• Vector databases specialize in storing and retrieving high-dimensional vectors, enabling fast semantic similarity search
• Core algorithms: HNSW for high accuracy, IVF for large-scale data, PQ for compressed storage
• Architecture first: Compare pgvector, search engines, dedicated databases, and managed services
• Production controls: Treat metadata filters, tenant isolation, evaluation, and cost as design inputs
• RAG applications: Vector retrieval is one component; ranking and answer quality require separate evaluation

What is a Vector Database

A vector database is a database system specifically designed to store, index, and query high-dimensional vector data. Unlike traditional relational databases based on exact matching, vector databases achieve semantic-level retrieval by calculating similarity between vectors.

Vector Database vs Traditional Database

Source of Vectors: Embeddings

Vectors in vector databases typically come from embedding models that convert unstructured data like text, images, and audio into dense numerical vectors:

from openai import OpenAI

client = OpenAI()

def get_embedding(text: str, model: str = "text-embedding-3-small") -> list[float]:
    response = client.embeddings.create(input=text, model=model)
    return response.data[0].embedding

text = "Vector retrieval supports semantic search in AI applications"
embedding = get_embedding(text)
print(f"Vector dimension: {len(embedding)}")  # Output: Vector dimension: 1536

Similarity Metrics

Vector databases support multiple similarity calculation methods:

Vector Indexing Algorithms Explained

Efficient indexing algorithms are the core competitive advantage of vector databases. Here's an in-depth analysis of three mainstream algorithms:

HNSW (Hierarchical Navigable Small World)

HNSW is a widely implemented vector indexing algorithm that supports approximate nearest neighbor search with a hierarchical small-world graph.

Core Principles:

• Builds multi-layer graph structure with sparse upper layers and dense lower layers
• Search starts from the highest layer and refines downward
• Time complexity: O(log N)

Pros and Cons:

• ✅ Fast query speed, high recall rate
• ✅ Supports dynamic insertion and deletion
• ❌ Higher memory consumption
• ❌ Longer index build time

# HNSW parameter configuration example
hnsw_params = {
    "M": 16,                # Maximum connections per node
    "ef_construction": 200, # Search width during construction
    "ef_search": 100        # Search width during query
}

IVF (Inverted File Index)

IVF partitions the vector space into multiple regions through clustering, searching only relevant regions during queries.

Core Principles:

• Uses K-Means to cluster vectors into nlist clusters
• Each vector is assigned to the nearest cluster center
• Queries search only nprobe nearest clusters

Pros and Cons:

• ✅ Suitable for ultra-large-scale datasets
• ✅ High memory efficiency
• ❌ Requires pre-training cluster centers
• ❌ Recall rate affected by nprobe parameter

PQ (Product Quantization)

PQ is a vector compression technique that significantly reduces storage space through quantization.

Core Principles:

• Splits high-dimensional vectors into multiple sub-vectors
• Independently clusters and quantizes each subspace
• Replaces original vectors with cluster center IDs

Pros and Cons:

• ✅ Dramatically compresses storage (10-100x)
• ✅ Suitable for memory-constrained scenarios
• ❌ Some precision loss
• ❌ Requires codebook training

Algorithm Selection Guide

Mainstream Vector Database Comparison

Product Feature Comparison Table

Detailed Product Analysis

Pinecone - Managed Cloud Option

Pinecone offers a fully managed serverless service. Evaluate current regions, filter behavior, consistency, limits, and workload pricing rather than selecting it from brand familiarity.

Use Cases:

• Rapid prototyping
• Teams not wanting to manage infrastructure
• Latency-sensitive production applications

Pricing Model: Pay per storage and query volume

Milvus - Enterprise-Grade Open Source Solution

Milvus is a CNCF graduated project providing enterprise-grade distributed vector database capabilities.

Use Cases:

• Large-scale production deployments
• GPU acceleration needs
• High data compliance requirements (self-hosted)

Cloud-hosted Version: Zilliz Cloud

Weaviate - Semantic Search Expert

Weaviate has built-in vectorization modules that automatically convert text to vectors.

Use Cases:

• Semantic search applications
• GraphQL API requirements
• Multi-modal data processing

Chroma - Lightweight Development Option

Chroma is designed specifically for AI applications with deep integration into frameworks like LangChain.

Use Cases:

• Local development and testing
• Small-scale applications
• Quick LLM application integration

Qdrant - High-Performance Rust Implementation

Qdrant is written in Rust, providing excellent performance and rich filtering capabilities.

Use Cases:

• High performance requirements
• Complex filtering conditions
• Payload storage needs

Selection Guide

Cloud-Hosted vs Self-Hosted

Architecture Decision Matrix

Do not rank products with a universal performance or cost ordering. Benchmark the same embeddings, filters, top-k, concurrency, update rate, and recall target against each candidate.

Production Checklist

• Index and quality: establish an exact-search baseline; measure recall@k, nDCG/MRR, p95/p99 latency, and index build/update time.
• Filtering: test filter selectivity and whether pre-filtering or post-filtering changes recall.
• Hybrid retrieval: measure lexical, vector, and reranked variants on the same labeled queries.
• Tenancy and authorization: derive tenant and ACL filters from trusted identity; verify isolation under retries, caches, and index rebuilds.
• Lifecycle: version embeddings, support backfills and dual-read migration, and define deletion propagation.
• Cost: include replicas, memory, storage, writes, queries, backups, network egress, and operator time.

If repeated public or tenant-scoped questions dominate cost, a semantic cache can sit above retrieval, but it requires separate thresholds, versioning, and authorization boundaries. See the production semantic caching guide.

Practical Code Examples

Building Local Vector Storage with Chroma

import chromadb
from chromadb.utils import embedding_functions

openai_ef = embedding_functions.OpenAIEmbeddingFunction(
    api_key="your-api-key",
    model_name="text-embedding-3-small"
)

client = chromadb.PersistentClient(path="./chroma_db")

collection = client.get_or_create_collection(
    name="documents",
    embedding_function=openai_ef,
    metadata={"hnsw:space": "cosine"}
)

documents = [
    "Vector databases are specialized databases for storing and retrieving high-dimensional vectors",
    "HNSW algorithm achieves efficient approximate nearest neighbor search through hierarchical graph structures",
    "RAG technology combines retrieval and generation to improve LLM response quality",
    "Embedding models convert text into dense numerical vector representations"
]

collection.add(
    documents=documents,
    ids=[f"doc_{i}" for i in range(len(documents))]
)

results = collection.query(
    query_texts=["What is vector search"],
    n_results=2
)

print("Search results:")
for doc, distance in zip(results['documents'][0], results['distances'][0]):
    print(f"  Similarity: {1-distance:.4f} | {doc[:50]}...")

Building Cloud Vector Index with Pinecone

from pinecone import Pinecone, ServerlessSpec
from openai import OpenAI

pc = Pinecone(api_key="your-pinecone-api-key")
openai_client = OpenAI()

index_name = "document-search"

if index_name not in pc.list_indexes().names():
    pc.create_index(
        name=index_name,
        dimension=1536,
        metric="cosine",
        spec=ServerlessSpec(cloud="aws", region="us-east-1")
    )

index = pc.Index(index_name)

def get_embedding(text: str) -> list[float]:
    response = openai_client.embeddings.create(
        input=text,
        model="text-embedding-3-small"
    )
    return response.data[0].embedding

documents = [
    {"id": "1", "text": "Vector databases support semantic search", "category": "database"},
    {"id": "2", "text": "HNSW is an efficient indexing algorithm", "category": "algorithm"},
    {"id": "3", "text": "RAG improves LLM response quality", "category": "llm"}
]

vectors = []
for doc in documents:
    embedding = get_embedding(doc["text"])
    vectors.append({
        "id": doc["id"],
        "values": embedding,
        "metadata": {"text": doc["text"], "category": doc["category"]}
    })

index.upsert(vectors=vectors)

query = "How to improve AI response accuracy"
query_embedding = get_embedding(query)

results = index.query(
    vector=query_embedding,
    top_k=2,
    include_metadata=True
)

print("Search results:")
for match in results.matches:
    print(f"  Score: {match.score:.4f} | {match.metadata['text']}")

Implementing Filtered Search with Qdrant

from qdrant_client import QdrantClient
from qdrant_client.models import Distance, VectorParams, PointStruct, Filter, FieldCondition, MatchValue

client = QdrantClient(path="./qdrant_db")

client.recreate_collection(
    collection_name="articles",
    vectors_config=VectorParams(size=1536, distance=Distance.COSINE)
)

points = [
    PointStruct(
        id=1,
        vector=get_embedding("Vector database beginner's guide"),
        payload={"title": "Vector DB Introduction", "category": "tutorial", "views": 1000}
    ),
    PointStruct(
        id=2,
        vector=get_embedding("HNSW algorithm deep dive"),
        payload={"title": "HNSW Algorithm Analysis", "category": "algorithm", "views": 500}
    ),
    PointStruct(
        id=3,
        vector=get_embedding("RAG application case study"),
        payload={"title": "RAG in Practice", "category": "tutorial", "views": 2000}
    )
]

client.upsert(collection_name="articles", points=points)

results = client.search(
    collection_name="articles",
    query_vector=get_embedding("Learn vector databases"),
    query_filter=Filter(
        must=[
            FieldCondition(key="category", match=MatchValue(value="tutorial"))
        ]
    ),
    limit=2
)

print("Filtered search results:")
for result in results:
    print(f"  Score: {result.score:.4f} | {result.payload['title']}")

Vector Databases in RAG Applications

RAG (Retrieval-Augmented Generation) is one of the most important use cases for vector databases. It enhances LLM response quality by retrieving relevant documents.

RAG Architecture Flow

User Question → Embedding → Vector Search → Relevant Docs → LLM Generation → Answer

Complete RAG Implementation Example

from openai import OpenAI
import chromadb
from chromadb.utils import embedding_functions

class SimpleRAG:
    def __init__(self):
        self.openai = OpenAI()
        self.chroma = chromadb.PersistentClient(path="./rag_db")
        self.embedding_fn = embedding_functions.OpenAIEmbeddingFunction(
            api_key="your-api-key",
            model_name="text-embedding-3-small"
        )
        self.collection = self.chroma.get_or_create_collection(
            name="knowledge_base",
            embedding_function=self.embedding_fn
        )
    
    def add_documents(self, documents: list[str], ids: list[str] = None):
        if ids is None:
            ids = [f"doc_{i}" for i in range(len(documents))]
        self.collection.add(documents=documents, ids=ids)
    
    def query(self, question: str, top_k: int = 3) -> str:
        results = self.collection.query(
            query_texts=[question],
            n_results=top_k
        )
        
        context = "\n".join(results['documents'][0])
        
        response = self.openai.chat.completions.create(
            model="gpt-4-turbo",
            messages=[
                {"role": "system", "content": f"Answer the question based on the following context:\n\n{context}"},
                {"role": "user", "content": question}
            ]
        )
        
        return response.choices[0].message.content

rag = SimpleRAG()

knowledge = [
    "Vector databases are database systems specifically designed for storing and retrieving high-dimensional vectors.",
    "The HNSW algorithm achieves efficient approximate nearest neighbor search by building hierarchical small-world graphs.",
    "Pinecone is a fully managed cloud vector database service.",
    "Chroma is a lightweight open-source vector database suitable for local development."
]
rag.add_documents(knowledge)

answer = rag.query("What is a vector database? What are the common products?")
print(answer)

Performance Optimization Best Practices

1. Index Parameter Tuning

# HNSW parameter optimization recommendations
hnsw_config = {
    "M": 16,                    # Can reduce to 8 for small datasets
    "ef_construction": 200,     # Build quality, higher is better but slower
    "ef_search": 100            # Query precision, adjust based on recall requirements
}

# IVF parameter optimization recommendations
ivf_config = {
    "nlist": 1024,              # Number of clusters, recommend sqrt(N) to 4*sqrt(N)
    "nprobe": 16                # Clusters to search, higher means better recall
}

2. Batch Operation Optimization

# Batch insert instead of one-by-one
batch_size = 100
for i in range(0, len(documents), batch_size):
    batch = documents[i:i+batch_size]
    collection.add(
        documents=[d["text"] for d in batch],
        ids=[d["id"] for d in batch],
        metadatas=[d["metadata"] for d in batch]
    )

3. Vector Dimension Selection

4. Caching Strategy

from functools import lru_cache

@lru_cache(maxsize=1000)
def get_cached_embedding(text: str) -> tuple:
    embedding = get_embedding(text)
    return tuple(embedding)

FAQ

Can vector databases be used together with traditional databases?

Yes, this is a common hybrid architecture pattern. Traditional databases store structured data and metadata, while vector databases store embedding vectors. During queries, first retrieve similar document IDs from the vector database, then fetch complete information from the traditional database.

How to evaluate vector database retrieval quality?

Use the production RAG evaluation guide to separate retrieval metrics from answer quality. Key retrieval metrics include:

• Recall@K: Proportion of correct answers in Top-K results
• Precision@K: Proportion of relevant documents in Top-K results
• MRR (Mean Reciprocal Rank): Average reciprocal of correct answer rankings
• Latency: Query response time

Do vector databases require GPUs?

For most use cases, CPU is sufficient. GPUs are mainly helpful in:

• Ultra-large-scale datasets (100M+ records)
• Real-time index training requirements
• High-concurrency query scenarios

Milvus supports GPU acceleration, significantly improving large-scale data processing capabilities.

How to handle data updates in vector databases?

• HNSW: Supports dynamic insertion and deletion, but frequent updates may affect performance
• IVF: Updates require retraining cluster centers, batch updates recommended
• Best Practice: Use soft delete + periodic index rebuild strategy

How to backup vector database data?

• Pinecone: Automatic backup, supports Collection snapshots
• Milvus: Supports data export and S3 backup
• Chroma: Persists to local files, can be directly copied
• Qdrant: Supports snapshots and incremental backups

Summary

Vector retrieval is useful infrastructure when semantic similarity is part of the product contract. The main decision is not a vendor ranking; it is whether an existing database, search engine, dedicated system, or managed service best satisfies measured quality, isolation, latency, and cost requirements.

Key Takeaways Review

✅ Vector databases enable semantic-level retrieval through similarity search
✅ HNSW for high accuracy, IVF+PQ for large-scale data
✅ Choose architecture from workload evidence, not a universal vendor ranking ✅ Validate filters, tenant boundaries, updates, backup, and total cost ✅ Evaluate retrieval and generated answers as separate quality layers

Related Resources

• RAG Retrieval-Augmented Generation Guide - Build an end-to-end retrieval and generation pipeline
• Vector Embeddings Guide - Understand the model outputs your index searches
• AI Agent Development Guide - Build intelligent agent applications

Further Reading

• Prompt Engineering Complete Guide - Optimize LLM prompts