Common embedding model mistakes that hurt search quality

Most embedding model mistakes don’t throw errors. They return plausible-looking vectors that produce subtly wrong rankings, and you only discover the problem when users stop trusting search. The fixes are usually small — normalize your vectors, match your model to your domain, evaluate with real queries — but they require knowing where to look.

1. Using a general-purpose model for domain-specific text

General models like text-embedding-3-small or bge-base-en are trained on web-scale corpora. They understand “apple” as fruit and company, but they don’t know your internal acronyms, product codes, or the fact that “RMA” in your support tickets means something completely different than in medical literature.

# Bad: one model for everything
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("sentence-transformers/all-MiniLM-L6-v2")
embeddings = model.encode(documents)  # legal contracts, support tickets, marketing copy

# Better: route by domain
from sentence_transformers import SentenceTransformer

MODELS = {
    "legal": "sentence-transformers/legal-bert-base-uncased",
    "support": "sentence-transformers/multi-qa-mpnet-base-dot-v1",
    "general": "sentence-transformers/all-MiniLM-L6-v2",
}

def embed_for_domain(texts, domain):
    model = SentenceTransformer(MODELS[domain])
    return model.encode(texts, normalize_embeddings=True)

If you’re indexing mixed corpora, either segment by domain at ingest time or use a model trained on in-domain data. Fine-tuning a base model on 1-2k labeled pairs from your own data often beats the best general model for your specific retrieval task.

2. Ignoring token limits and truncation strategy

Every embedding model has a maximum context length — 512 tokens for BERT-based models, 8192 for text-embedding-3-large. When you shove a 3,000-token document into a 512-token model, the library silently truncates. Usually it keeps the first 512 tokens and drops the rest. Your carefully crafted conclusion, the part that actually answers the query, never makes it into the vector.

# What actually happens with long documents
from transformers import AutoTokenizer

tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/all-MiniLM-L6-v2")
doc = "your 3000 token document..."
tokens = tokenizer.encode(doc, truncation=True, max_length=512)
print(len(tokens))  # 512 — but which 512?

The fix depends on your retrieval granularity. For document-level search, chunk first, embed chunks, then aggregate at query time (max-sim, mean-sim, or learned weighting). For passage-level search, use a model with longer context or a sliding-window approach with overlap.

# Chunk-then-embed pattern
def chunk_and_embed(text, model, chunk_size=256, overlap=50):
    tokens = tokenizer.encode(text)
    chunks = []
    for i in range(0, len(tokens), chunk_size - overlap):
        chunk_tokens = tokens[i:i + chunk_size]
        chunks.append(tokenizer.decode(chunk_tokens))
    embeddings = model.encode(chunks, normalize_embeddings=True)
    return embeddings  # shape: (n_chunks, dim)

At query time, compare the query vector against all chunk vectors and take the maximum similarity — this is “max-sim” retrieval and it works surprisingly well for long documents.

3. Not normalizing embeddings before cosine similarity

Cosine similarity assumes unit vectors. If you compute np.dot(a, b) on raw embeddings without normalizing, you’re actually computing dot product, not cosine. The results will be dominated by vector magnitude, which correlates with… nothing useful. Longer texts, certain token patterns, and model quirks all produce different magnitudes.

# Wrong: dot product on unnormalized vectors
similarities = np.dot(query_emb, doc_embeddings.T)  # magnitudes leak in

# Right: normalize first, then dot product = cosine
query_norm = query_emb / np.linalg.norm(query_emb)
doc_norms = doc_embeddings / np.linalg.norm(doc_embeddings, axis=1, keepdims=True)
similarities = np.dot(query_norm, doc_norms.T)

Most embedding libraries have a normalize_embeddings=True flag. Use it. If you’re storing vectors in a vector database, store them normalized — it saves a normalization step at query time and makes inner_product indexes equivalent to cosine.

# SentenceTransformers handles this
model.encode(texts, normalize_embeddings=True)  # returns unit vectors

# OpenAI returns normalized vectors by default for v3 models
# but verify: check the docs for your specific model version

4. Treating all dimensions as equally important

A 1536-dimensional embedding from text-embedding-3-large contains signal and noise. The first few principal components often capture syntax and generic semantic features (is this English? is it a question?) while discriminative signal for your specific task lives in later components. Blindly using all dimensions adds noise and compute cost.

# Diagnose: check variance explained by top components
from sklearn.decomposition import PCA
import numpy as np

pca = PCA().fit(embeddings)
cumulative_variance = np.cumsum(pca.explained_variance_ratio_)
print(f"Top 100 dims explain {cumulative_variance[99]:.1%} of variance")
print(f"Top 256 dims explain {cumulative_variance[255]:.1%} of variance")

For many tasks, truncating to 256-512 dimensions preserves >95% of retrieval quality while cutting storage and latency in half. Matryoshka embeddings (like text-embedding-3-large with dimensions=256) are trained specifically for this — the first N dimensions are the most informative.

# OpenAI Matryoshka: request truncated dimensions
response = client.embeddings.create(
    model="text-embedding-3-large",
    input=texts,
    dimensions=256,  # returns 256-dim vectors, optimized for this size
    encoding_format="float"
)

If you’re using open models, learn a projection matrix on a validation set: W = V_k.T @ V_full where V_k are the top-k right singular vectors from SVD on your corpus embeddings.

5. Skipping evaluation with real queries

You wouldn’t ship a classifier without a test set. Yet teams routinely deploy embedding models with zero retrieval evaluation. Offline metrics (MRR@10, nDCG@10, Recall@k) on a held-out query set catch the obvious failures: wrong model, bad chunking, normalization bugs.

# Minimal evaluation harness
from rank_bm25 import BM25Okapi
import numpy as np

def evaluate_retrieval(queries, qrels, embed_fn, k=10):
    """qrels: dict[query_id] -> set(relevant_doc_ids)"""
    query_embs = embed_fn([q["text"] for q in queries])
    doc_embs = embed_fn([d["text"] for d in corpus])
    
    # normalize
    query_embs = query_embs / np.linalg.norm(query_embs, axis=1, keepdims=True)
    doc_embs = doc_embs / np.linalg.norm(doc_embs, axis=1, keepdims=True)
    
    scores = query_embs @ doc_embs.T  # (n_queries, n_docs)
    rankings = np.argsort(-scores, axis=1)[:, :k]
    
    mrr = 0.0
    recall = 0.0
    for i, q in enumerate(queries):
        relevant = qrels[q["id"]]
        ranked = rankings[i]
        # MRR
        for rank, doc_idx in enumerate(ranked):
            if doc_idx in relevant:
                mrr += 1.0 / (rank + 1)
                break
        # Recall@k
        recall += len(set(ranked) & relevant) / len(relevant)
    
    return {"MRR@10": mrr / len(queries), "Recall@10": recall / len(queries)}

Build a small labeled set (50-100 queries with relevance judgments) from real user traffic or manual annotation. Run it on every model change. If you don’t have labels, use BM25 as a weak baseline — if your dense retriever doesn’t beat BM25 on your domain, something is wrong.

6. Forgetting that embeddings age

Model providers deprecate models. OpenAI retired text-embedding-ada-002 in favor of v3 models. Open-source models get updated weights. If you re-embed your corpus with a new model version, every vector changes — and your search results shift, sometimes dramatically.

# Version your embeddings at ingest time
import hashlib
from datetime import datetime

def embed_with_metadata(texts, model_name, model_version):
    embeddings = model.encode(texts, normalize_embeddings=True)
    return [{
        "vector": emb.tolist(),
        "model": model_name,
        "model_version": model_version,
        "embedded_at": datetime.utcnow().isoformat(),
        "content_hash": hashlib.sha256(text.encode()).hexdigest()[:16]
    } for emb, text in zip(embeddings, texts)]

Store the model identifier and version alongside every vector. When you migrate, re-embed a sample first, run your evaluation harness, and only flip the traffic if metrics hold. Keep the old vectors for rollback. Some vector databases (Pinecone, Weaviate) support namespaces or collections — use them to run A/B migrations safely.

# Migration pattern: dual-write during transition
def search(query, primary_namespace, shadow_namespace=None):
    primary_results = db.query(primary_namespace, query, top_k=20)
    if shadow_namespace:
        shadow_results = db.query(shadow_namespace, query, top_k=20)
        # compare, log discrepancies, don't serve shadow yet
    return primary_results

Summary

The thread connecting these: embedding quality is a data engineering problem, not a model selection problem. The model is a component; the pipeline — chunking, normalization, evaluation, versioning — determines whether search actually works.