How text embeddings turn words into vectors

Text embeddings convert variable-length text into fixed-size vectors that capture semantic meaning. The core idea is simple: words or phrases with similar meanings end up close together in vector space, while unrelated concepts drift apart. This guide walks through the mechanics, the practical choices you’ll face when shipping an embedding pipeline, and the failure modes that only show up under load.

Choose the right model for your task

Not all embedding models are interchangeable. General-purpose models like text-embedding-3-small or bge-base-en-v1.5 work well for semantic search and clustering. Domain-specific models (legal, biomedical, code) outperform general ones on their turf but degrade sharply outside it. Multilingual models like e5-multilingual or jina-embeddings-v3 handle cross-lingual retrieval but cost more compute per token.

Before committing, run a small eval on your actual data. A 50-query test set with labeled relevance judgments tells you more than any leaderboard. Measure recall@k, not just cosine similarity distributions.

from sentence_transformers import SentenceTransformer
import numpy as np

model = SentenceTransformer("BAAI/bge-base-en-v1.5")
queries = ["how to cancel subscription", "refund policy"]
docs = ["subscription cancellation steps", "billing and refunds"]

q_emb = model.encode(queries, normalize_embeddings=True)
d_emb = model.encode(docs, normalize_embeddings=True)

scores = q_emb @ d_emb.T  # cosine similarity since normalized
print(scores)

Understand dimensionality and quantization tradeoffs

Higher dimensions capture more nuance but increase storage, latency, and index build time. text-embedding-3-large produces 3072-dim vectors; text-embedding-3-small produces 1536. OpenAI’s Matryoshka embeddings let you truncate to 256, 512, or 1024 dims with minimal quality loss — useful when you need to fit a vector DB’s limits or reduce P99 latency.

Quantization compresses float32 vectors to int8 or binary. Product quantization (PQ) and scalar quantization (SQ) can shrink index size 4-8x with 1-3% recall drop. Most vector databases (pgvector, Qdrant, Weaviate, Pinecone) handle this transparently. Test your specific workload before enabling; some query patterns degrade more than others.

# Truncating Matryoshka embeddings — no retraining needed
full_emb = model.encode(["your text"], normalize_embeddings=True)[0]
truncated_256 = full_emb[:256]
truncated_256 = truncated_256 / np.linalg.norm(truncated_256)  # renormalize

Preprocess consistently or pay at query time

Embedding models are sensitive to input formatting. Trailing whitespace, inconsistent newlines, and unicode normalization differences all shift vectors. The safest pattern: normalize once at ingestion, store the normalized text alongside the vector, and apply the exact same pipeline at query time.

import unicodedata
import re

def normalize_text(text: str) -> str:
    text = unicodedata.normalize("NFC", text)
    text = text.strip()
    text = re.sub(r"\s+", " ", text)  # collapse whitespace
    return text

Chunking strategy matters as much as normalization. Fixed-size chunks (512 tokens with 50-token overlap) are easy to implement but split semantic units. Recursive character splitting or semantic chunking (using an embedding model to detect topic boundaries) preserves coherence at the cost of variable chunk sizes and more complex retrieval logic.

Index for your access pattern

Flat (brute-force) search is fine under 100k vectors. Beyond that, you need an approximate nearest neighbor (ANN) index. HNSW is the default choice for most workloads — fast builds, excellent recall, predictable latency. IVF (inverted file) scales better to billions of vectors but requires training and tuning nprobe at query time. DiskANN and ScaNN optimize for SSD-backed indexes and high throughput respectively.

Key HNSW parameters:

M (bi-directional links per node): 16-48. Higher = better recall, more memory.
ef_construction: 100-400. Build-time search width. Higher = better index quality, slower builds.
ef_search: 50-500. Query-time search width. Tune per-query for latency/recall tradeoff.

# Qdrant HNSW config example
from qdrant_client import QdrantClient
from qdrant_client.models import VectorParams, Distance, HnswConfigDiff

client = QdrantClient(":memory:")
client.create_collection(
    collection_name="docs",
    vectors_config=VectorParams(size=768, distance=Distance.COSINE),
    hnsw_config=HnswConfigDiff(m=32, ef_construct=200),
)

Handle out-of-vocabulary and domain drift

Static embeddings (Word2Vec, GloVe) fail on unseen tokens. Subword tokenizers (BPE, WordPiece, Unigram) mitigate this but produce vectors for subword pieces, not whole words. Modern transformer embeddings avoid OOV entirely — the model encodes any byte sequence — but domain drift still hurts. A model trained on web text degrades on legal contracts or clinical notes.

Monitor embedding quality in production. Track:

Average cosine similarity between query and top-k results (should stay stable)
Fraction of queries returning zero results above threshold
Embedding norm distribution shifts (indicates input distribution change)

Retrain or fine-tune when metrics drift. Contrastive fine-tuning on in-domain pairs (query, relevant doc) typically yields 5-15% recall improvement over base models.

# Simple contrastive fine-tuning setup with sentence-transformers
from sentence_transformers import SentenceTransformer, InputExample, losses
from torch.utils.data import DataLoader

model = SentenceTransformer("BAAI/bge-base-en-v1.5")
train_examples = [
    InputExample(texts=["cancel subscription", "how to end my subscription"], label=1.0),
    InputExample(texts=["cancel subscription", "upgrade plan"], label=0.0),
]
train_dataloader = DataLoader(train_examples, shuffle=True, batch_size=16)
train_loss = losses.CosineSimilarityLoss(model)
model.fit(train_objectives=[(train_dataloader, train_loss)], epochs=3, warmup_steps=100)

Cache aggressively but invalidate deliberately

Embedding the same text twice is wasteful. Cache at the application layer (Redis, in-memory LRU) keyed by normalized text hash. For high-cardinality workloads, a two-tier cache works: hot keys in local memory, warm keys in distributed cache.

Invalidation is the hard part. When a document updates, you must recompute its chunks’ embeddings and update the vector index. Most vector databases support point updates, but batch re-indexing is faster for bulk changes. Design your ingestion pipeline to emit embedding jobs asynchronously — don’t block the write path on vector computation.

import hashlib
import redis

r = redis.Redis(decode_responses=True)

def get_embedding(text: str, model) -> list[float]:
    key = f"emb:{hashlib.sha256(text.encode()).hexdigest()[:16]}"
    cached = r.get(key)
    if cached:
        return json.loads(cached)
    emb = model.encode([text], normalize_embeddings=True)[0].tolist()
    r.setex(key, 86400, json.dumps(emb))  # 24hr TTL
    return emb

Evaluate with real queries, not synthetic ones

Benchmark datasets (BEIR, MTEB) are useful for model selection but don’t reflect your users’ vocabulary, typos, or intent distribution. Build an eval set from production logs: sample 500-1000 real queries, have domain experts label relevant documents, then measure nDCG@10, recall@10, and MRR.

Run this eval on every model upgrade, index parameter change, and chunking strategy tweak. Automate it in CI. A 2% recall drop that passes unit tests will show up in user complaints within days.

Common pitfalls

Cosine similarity on non-normalized vectors. Most embedding models output normalized vectors, but truncation, quantization, or custom fine-tuning can break this. Always normalize before computing cosine similarity, or use dot product on pre-normalized vectors.

Ignoring token limits. Embedding models have hard context windows (512, 8192, 32768 tokens). Truncating silently loses information. Chunk with overlap, or use a model with sufficient context for your documents.

Treating all dimensions equally. In high-dimensional space, distance concentration makes all vectors roughly equidistant. Dimensionality reduction (PCA, UMAP) before indexing can improve ANN quality, but test first — sometimes it hurts.

Single-vector representations for long documents. A 10k-token document compressed to one 768-dim vector loses granularity. Use late interaction (ColBERT), multi-vector representations, or retrieve at chunk level then rerank.

Forgetting that embeddings are model-versioned. Vectors from text-embedding-3-small v1 and v2 are not comparable. Store the model identifier with every vector. When you upgrade, re-embed the entire corpus or maintain separate indexes per version.

Production checklist

Embeddings are a compression tool. They trade precision for scalability. The engineering work isn’t in calling the model — it’s in the pipeline that feeds it, the index that serves it, and the eval loop that keeps it honest.