Vector distance metrics compared: cosine, dot product, L2

Vector distance metrics comparison comes down to one question: what does “similar” mean for your embeddings? Cosine similarity measures angle, dot product measures projection magnitude, and L2 distance measures absolute displacement. Each metric induces a different geometry on your vector space, and picking the wrong one silently degrades retrieval quality. This post breaks down the mathematical differences, practical implications, and decision criteria so you can choose confidently.

The three metrics defined

Given two vectors a and b in ℝⁿ:

Cosine similarity computes the cosine of the angle between them:

cos(θ) = (a · b) / (||a|| ||b||)

Range: [-1, 1]. Higher = more similar. Ignores magnitude entirely.

Dot product (inner product) computes the sum of element-wise products:

a · b = Σ aᵢ bᵢ

Range: (-∞, ∞). Higher = more similar. Sensitive to both angle and magnitude.

L2 distance (Euclidean distance) computes the straight-line distance:

||a - b||₂ = √(Σ (aᵢ - bᵢ)²)

Range: [0, ∞). Lower = more similar. Sensitive to both angle and magnitude.

Normalization changes everything

The relationship between these metrics hinges on whether your vectors are normalized to unit length.

If all vectors are L2-normalized (||v|| = 1), the three metrics become monotonically related:

• cos(θ) = a · b (since ||a|| = ||b|| = 1)
• ||a - b||² = 2 - 2(a · b) = 2(1 - cos(θ))

In this regime, ranking by any metric produces identical orderings. The choice reduces to implementation convenience and index compatibility.

If vectors are not normalized, the metrics diverge sharply. Dot product and L2 distance both conflate direction and magnitude. A long vector pointing in a moderately relevant direction can outrank a short vector pointing in the exactly correct direction. Cosine similarity ignores magnitude entirely, which can be desirable or disastrous depending on your embedding model.

import numpy as np

def cosine_similarity(a, b):
    return np.dot(a, b) / (np.linalg.norm(a) * np.linalg.norm(b))

def dot_product(a, b):
    return np.dot(a, b)

def l2_distance(a, b):
    return np.linalg.norm(a - b)

# Unnormalized example
a = np.array([10.0, 0.0])      # long vector, direction X
b = np.array([0.0, 10.0])      # long vector, direction Y
c = np.array([0.1, 0.1])       # short vector, direction X+Y

print(f"cosine(a, c): {cosine_similarity(a, c):.3f}")  # 0.707
print(f"cosine(b, c): {cosine_similarity(b, c):.3f}")  # 0.707
print(f"dot(a, c):    {dot_product(a, c):.3f}")        # 1.0
print(f"dot(b, c):    {dot_product(b, c):.3f}")        # 1.0
print(f"l2(a, c):     {l2_distance(a, c):.3f}")        # 9.95
print(f"l2(b, c):     {l2_distance(b, c):.3f}")        # 9.95

# Normalized changes everything
a_n = a / np.linalg.norm(a)
b_n = b / np.linalg.norm(b)
c_n = c / np.linalg.norm(c)
print(f"\nAfter normalization:")
print(f"cosine(a_n, c_n): {cosine_similarity(a_n, c_n):.3f}")  # 0.707
print(f"dot(a_n, c_n):    {dot_product(a_n, c_n):.3f}")        # 0.707
print(f"l2(a_n, c_n):     {l2_distance(a_n, c_n):.3f}")        # 0.765

Embedding model behavior dictates the metric

Different embedding models produce vectors with different norm distributions. This is the single most important factor in metric selection.

Models that output normalized vectors (OpenAI text-embedding-3-*, Cohere embed-v3, Voyage AI): These models explicitly L2-normalize outputs. Cosine, dot product, and L2 all yield identical rankings. Use whichever your vector database optimizes for — typically dot product (via inner product index) or cosine (via normalized vectors + inner product).

Models with variable norms (older BERT-based models, some open-source models, custom fine-tunes): Vector magnitude often correlates with semantic “confidence” or information content. A longer vector may represent a more specific, detailed concept. In this case:

• Cosine similarity discards that signal
• Dot product amplifies it (magnitude × alignment)
• L2 distance penalizes magnitude differences

Models where magnitude is noise (some contrastive learning outputs, poorly calibrated models): Magnitude varies randomly. Cosine similarity is more robust.

# Quick diagnostic: check norm distribution of your embeddings
import numpy as np
norms = np.linalg.norm(embeddings, axis=1)
print(f"Norm stats: mean={norms.mean():.3f}, std={norms.std():.3f}, "
      f"min={norms.min():.3f}, max={norms.max():.3f}")
print(f"Coefficient of variation: {norms.std() / norms.mean():.3f}")
# CV < 0.05 → effectively normalized, metric choice doesn't matter for ranking
# CV > 0.2  → magnitude carries signal or noise, choose deliberately

Index compatibility and performance

Vector databases implement approximate nearest neighbor (ANN) indexes optimized for specific distance functions. Your metric choice constrains index selection.

HNSW (Hierarchical Navigable Small World) is the dominant index for high-recall, low-latency search. Most implementations (FAISS, hnswlib, Milvus, Weaviate, Qdrant, Pinecone) support all three distance functions. However, HNSW with L2 distance on unnormalized vectors can produce counterintuitive results because the index navigates by Euclidean proximity, not angular proximity.

ScaNN (Scalable Nearest Neighbors) from Google is specifically optimized for maximum inner product search (MIPS) — dot product. It uses anisotropic quantization to handle the magnitude component efficiently. If you need dot product on unnormalized vectors at scale, ScaNN is worth evaluating.

IVF (Inverted File) and PQ (Product Quantization) indexes in FAISS support all three metrics but require training. The training data distribution should match your query distribution for optimal quantization.

# FAISS index creation examples
import faiss

d = 1536  # dimension

# Cosine similarity: normalize + inner product index
index_cosine = faiss.IndexFlatIP(d)  # or IndexHNSWFlat(d, 32) + index.hnsw.efSearch = 64
# Must normalize vectors before add/search
faiss.normalize_L2(vectors)
index_cosine.add(vectors)

# Dot product: inner product index, no normalization required
index_dot = faiss.IndexFlatIP(d)
index_dot.add(vectors)  # raw vectors

# L2 distance: L2 index
index_l2 = faiss.IndexFlatL2(d)
index_l2.add(vectors)  # raw vectors

Quantization interactions

Product quantization (PQ) and scalar quantization (SQ) compress vectors for memory efficiency. The metric affects quantization error distribution.

Cosine/dot product on normalized vectors: Quantization error is bounded. Angular distortion from PQ is well-studied; 8-bit PQ typically preserves >95% recall@10 for 768-dim vectors.

Dot product on unnormalized vectors: Quantization must preserve magnitude information. Standard PQ allocates bits uniformly across dimensions, which can poorly represent magnitude if it concentrates in few dimensions. ScaNN’s anisotropic quantization addresses this.

L2 distance: Quantization error adds directly to distance. For unnormalized vectors, magnitude differences dominate, so quantization of high-magnitude dimensions matters more.

If you’re quantizing to 4-bit or 8-bit (common for cost reduction at scale), test recall at your target quantization level with your actual metric. Don’t assume cosine results transfer to dot product.

Numerical stability and implementation details

Cosine similarity requires division by norms. Zero vectors produce NaN. Guard against this:

def safe_cosine(a, b, eps=1e-8):
    norm_a = np.linalg.norm(a)
    norm_b = np.linalg.norm(b)
    if norm_a < eps or norm_b < eps:
        return 0.0
    return np.dot(a, b) / (norm_a * norm_b)

Dot product can overflow float32 for high-dimensional, high-magnitude vectors. Use float64 accumulation or normalize first.

L2 distance squared (||a-b||²) avoids the sqrt for ranking. Expand: ||a||² + ||b||² - 2(a·b). Precompute ||v||² for all vectors to accelerate:

# Precompute squared norms for fast L2 distance
norms_sq = np.sum(vectors ** 2, axis=1)  # shape (N,)
# Distance to query q: norms_sq + ||q||² - 2 * (vectors @ q)
distances = norms_sq + np.dot(q, q) - 2 * np.dot(vectors, q)

This is how FAISS’s IndexFlatL2 works internally — it’s a dot product index with precomputed norms.

Comparison table

Which to choose: verdict by use case

Use cosine similarity when:

• Your embedding model outputs normalized vectors (OpenAI, Cohere, Voyage, most modern APIs)
• You want semantic similarity independent of document length or embedding magnitude
• You’re building a general-purpose semantic search or RAG system
• You need maximum compatibility across vector databases and indexes

Default choice for 90% of LLM-era retrieval systems. Normalize once at ingest, use inner product index (HNSW-IP), done.

Use dot product when:

• Your embeddings are unnormalized and magnitude carries signal (e.g., older BERT models, custom contrastive models where norm correlates with specificity)
• You’re doing maximum inner product search (MIPS) for recommendation — user and item embeddings where magnitude represents preference strength
• You’re using ScaNN or an index specifically optimized for MIPS
• You need to preserve magnitude information for downstream scoring

Test cosine vs. dot product on a held-out evaluation set. If dot product improves nDCG or recall@k meaningfully, the magnitude signal is real. If not, normalize and use cosine.

Use L2 distance when:

• You’re clustering embeddings (k-means, HDBSCAN) — L2 is the natural objective
• You’re doing outlier/anomaly detection — absolute displacement matters
• Your vectors represent spatial or geometric data (not semantic embeddings)
• You’re implementing a custom index or algorithm that requires metric space properties (triangle inequality)

Avoid L2 for semantic search on unnormalized embeddings. It conflates “different direction” with “different magnitude” in ways that rarely match human similarity judgments.

Hybrid approach for uncertain cases:

If you don’t know whether magnitude matters, run this diagnostic:

def metric_agreement(embeddings, queries, k=10, sample=1000):
    """Measure ranking agreement between metrics on a sample."""
    from scipy.stats import spearmanr
    idx = np.random.choice(len(queries), min(sample, len(queries)), replace=False)
    agreements = []
    for q_idx in idx:
        q = queries[q_idx]
        cos_scores = cosine_similarity(embeddings, q)
        dot_scores = dot_product(embeddings, q)
        l2_scores = -l2_distance(embeddings, q)  # negate so higher=similar
        # Rank correlation
        rho_cos_dot, _ = spearmanr(cos_scores, dot_scores)
        rho_cos_l2, _ = spearmanr(cos_scores, l2_scores)
        agreements.append((rho_cos_dot, rho_cos_l2))
    return np.mean(agreements, axis=0)

# If both correlations > 0.95, metric choice doesn't matter for ranking
# If cos/dot correlation is low, magnitude matters → choose deliberately

One final consideration: reranking

If you’re using a two-stage retrieve-then-rerank pipeline (common in production RAG), the first-stage metric matters less. Retrieve 50-100 candidates with whatever metric your index supports efficiently, then rerank with a cross-encoder or LLM judge. The first stage only needs high recall; precision comes from reranking.

In this architecture, use the metric your vector database optimizes best. For most managed services (Pinecone, Weaviate, Qdrant, Milvus), that’s cosine via HNSW on normalized vectors or dot product via HNSW-IP. The marginal gain from a theoretically “better” first-stage metric rarely justifies the operational complexity of a custom index configuration.

Bottom line: Normalize your embeddings at ingest. Use cosine similarity (implemented as inner product on normalized vectors) with an HNSW index. This covers the vast majority of semantic search and RAG workloads. Only deviate when you have empirical evidence that magnitude carries signal your evaluation metrics reward.