Embeddings vs tokens: what's the difference?

Tokens are the atomic units that language models read and write. Embeddings are the dense vectors that represent what those tokens mean. Confusing the two leads to wasted compute, broken retrieval, and debugging sessions that last longer than they should. This post breaks down the embeddings vs tokens distinction across the dimensions that matter when you’re shipping production systems.

What tokens are

A token is an integer ID from a fixed vocabulary. The tokenizer — a deterministic, model-specific algorithm like BPE, WordPiece, or Unigram — maps raw text to a sequence of these IDs. GPT-4o uses a 200k-token vocabulary. Llama 3 uses 128k. The same word can split differently across tokenizers: “tokenization” might be one token in one vocabulary and three in another.

Tokens are discrete. They have no inherent semantic relationship to each other. Token 4521 (“cat”) and token 8932 (“dog”) are just integers. The model learns their relationship during training by seeing them in similar contexts, but the tokenizer itself knows nothing about meaning.

Key properties:

• Deterministic: Same input, same token sequence, every time.
• Model-specific: You cannot feed GPT-4o tokens into Llama 3 and expect coherent output.
• Countable: Context windows, pricing, and rate limits are all denominated in tokens.

# tiktoken example — OpenAI's tokenizer
import tiktoken

enc = tiktoken.encoding_for_model("gpt-4o")
tokens = enc.encode("Hello, world!")
# [9906, 11, 1917, 0]
print(len(tokens))  # 4 tokens

What embeddings are

An embedding is a fixed-length vector of floating-point numbers. An embedding model — a separate neural network, often a BERT-style encoder or the encoder portion of a transformer — maps a token sequence (or a whole document) to a point in high-dimensional space. Text-embedding-3-large outputs 3072 dimensions. BGE-large-en-v1.5 outputs 1024. The dimensionality is a model hyperparameter, not a function of input length.

Embeddings are continuous. Distance in this space correlates with semantic similarity. Cosine similarity between the vector for “cat” and “dog” will be high; between “cat” and “database” it will be low. This property makes embeddings useful for retrieval, clustering, classification, and anomaly detection.

Key properties:

• Model-specific: Vectors from different embedding models live in different spaces. You cannot compare them directly.
• Fixed width: Input length varies; output dimension does not.
• Lossy compression: A 3072-dim vector cannot losslessly represent a 10,000-token document. It captures “gist,” not detail.

# sentence-transformers example
from sentence_transformers import SentenceTransformer

model = SentenceTransformer("BAAI/bge-large-en-v1.5")
vec = model.encode("Hello, world!")
# array([-0.023, 0.041, ...], shape=(1024,), dtype=float32)
print(vec.shape)  # (1024,)

Head-to-head comparison

Capabilities

Tokens are the interface to generation. Every LLM API — OpenAI, Anthropic, local vLLM — accepts tokens and returns tokens. You cannot prompt a model with embeddings. You cannot get logits from an embedding model. If you need text out, you work with tokens.

Embeddings are the interface to similarity. They power RAG retrieval, duplicate detection, semantic deduplication, intent routing, and label-free classification. You do not generate with them. You do not “chat” with an embedding model. Some teams try to feed embeddings into an LLM as soft prompts; this works in research but adds fragility and is rarely worth the complexity in production.

The two interact in one critical path: tokenization precedes embedding. An embedding model tokenizes your input internally, runs those tokens through its encoder, and pools the final hidden states (CLS token, mean pooling, last token) into a single vector. You pay for the tokenization compute whether you see the tokens or not.

Price and cost model

Token pricing is straightforward: you pay per million tokens in and per million tokens out. OpenAI charges $2.50/1M input + $10/1M output for GPT-4o. Anthropic charges $3/1M + $15/1M for Claude 3.5 Sonnet. Local inference costs GPU-hours.

Embedding pricing is input-only. OpenAI’s text-embedding-3-small is $0.02/1M tokens. Text-embedding-3-large is $0.13/1M. Open-source models (BGE, E5, Nomic) run on your hardware — cost is GPU memory and latency, not API calls.

A common mistake: assuming embedding cost is negligible. At scale, embedding 100M documents with text-embedding-3-large costs $13,000 just for the API calls. Re-embedding a corpus after a model upgrade costs the same again. Budget for re-indexing.

# Rough cost estimate: 100M docs × ~500 tokens/doc × $0.13/1M
# = 50B tokens × $0.13/1M = $6,500 per embedding run

Latency and throughput

Token generation is sequential. Each output token requires a full forward pass through the model. Latency scales linearly with output length. Throughput is bounded by KV cache memory and compute. vLLM, TensorRT-LLM, and SGLang optimize this with continuous batching and paged attention, but the fundamental constraint remains.

Embedding inference is parallelizable across the input sequence (up to the model’s context window). A single forward pass produces the vector. Batch 1000 documents, get 1000 vectors. Latency per document drops with batch size until GPU memory saturates. Throughput is typically 10-100× higher than token generation for equivalent hardware.

# Embedding throughput example (BGE-large on A100)
# Batch size 64, seq len 512: ~2,500 docs/sec
# Same GPU generating GPT-4o tokens: ~50 tokens/sec per stream

If your pipeline embeds at ingest and generates at query time, the embedding stage is rarely the bottleneck. The generation stage almost always is.

Ergonomics

Tokens require a tokenizer library that matches your model exactly. Mismatched tokenizers produce garbage. You must handle truncation, special tokens, and chat templates. The OpenAI API hides this; local serving does not.

# Local Llama 3 chat template — you must apply this
from transformers import AutoTokenizer

tok = AutoTokenizer.from_pretrained("meta-llama/Meta-Llama-3-8B-Instruct")
messages = [{"role": "user", "content": "Hello"}]
prompt = tok.apply_chat_template(messages, tokenize=False, add_generation_prompt=True)
# "<|begin_of_text|><|start_header_id|>user<|end_header_id|>\n\nHello<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\n"

Embeddings are simpler: pass strings, get arrays. The embedding model handles tokenization internally. The main ergonomic decision is pooling strategy (CLS, mean, last token) and whether to normalize. Most libraries default to sensible choices.

# Normalized embeddings for cosine similarity
vec = model.encode(texts, normalize_embeddings=True)
# Now dot product == cosine similarity
scores = np.dot(query_vec, doc_vecs.T)

Ecosystem

Token ecosystem = every LLM framework. LangChain, LlamaIndex, Haystack, instructor, guidance, outlines — all operate on tokens. Token-level tooling includes logit bias, stop sequences, structured generation (JSON mode, regex constraints), and token-by-token streaming.

Embedding ecosystem = vector databases (Pinecone, Weaviate, Qdrant, Milvus, pgvector), rerankers (Cohere, BGE-reranker, Jina), and retrieval frameworks. The embedding model is a pluggable component. Swapping text-embedding-3-large for BGE-large-en-v1.5 changes your vectors, your index, and your retrieval quality — but not your generation stack.

This separation is healthy. It lets you upgrade retrieval without touching generation, and vice versa. But it also means you own the compatibility contract: if you change embedding models, you must re-index everything.

Limits

Token limits are hard constraints. Context window (128k, 200k, 1M, 2M). Output token cap (4k, 8k, 16k, 128k). Exceed them and the request fails or truncates silently. You must count tokens before sending. Tiktoken makes this easy for OpenAI models; for local models you need the exact tokenizer.

# Always count before sending
def count_tokens(text: str, model: str = "gpt-4o") -> int:
    enc = tiktoken.encoding_for_model(model)
    return len(enc.encode(text))

if count_tokens(prompt) > 120_000:
    raise ValueError("Prompt exceeds context window")

Embedding limits are softer but real. Maximum sequence length (512, 8192, 32768 tokens). Inputs longer than this truncate — usually silently, from the right. A 50-page PDF fed to a 512-token embedding model embeds only the first ~2 pages. You must chunk. Chunking strategy (fixed size, semantic, recursive) materially affects retrieval quality.

Embedding dimensions are fixed. You cannot “resize” a vector without retraining or projection layers. Matryoshka embeddings (like text-embedding-3) let you truncate dimensions at inference time with graceful degradation, but this is a model-specific feature.

Which to choose

Use tokens when:

• You need the model to generate text, code, or structured output.
• You are building chat, completion, extraction, or reasoning pipelines.
• You need logprobs, token-level streaming, or constrained decoding.
• You are accounting for context window usage or API costs.

Use embeddings when:

• You need semantic similarity: search, clustering, deduplication, classification.
• You are building the retrieval half of RAG.
• You need a fixed-size representation for downstream ML (classifier input, anomaly detection).
• You are routing requests by intent without an LLM call.

Use both when:

• Building RAG: embed chunks at ingest, tokenize prompts at query time.
• Building eval pipelines: embed expected vs actual outputs for semantic similarity metrics, then use token-level judges for finer-grained scoring.
• Doing speculative decoding: embed draft model outputs to verify semantic equivalence before accepting.

Don’t:

• Feed embeddings into an LLM expecting it to “understand” them without a projection layer and fine-tuning.
• Compare vectors from different embedding models.
• Assume token counts transfer across models.
• Skip chunking strategy design for long-document embeddings.

The embeddings vs tokens distinction isn’t academic — it determines your architecture, your costs, and your failure modes. Treat them as separate subsystems with separate contracts, and your pipelines will be easier to debug, upgrade, and scale.