Topic
Tokens & Tokenization
7 posts on tokens & tokenization — part of glossary on the n4n AI blog.
What is a token in AI? A plain-English definition
A precise technical definition of tokens in AI, covering tokenization mechanics, vocabulary design, context limits, and practical implications for LLM engineering.
Tokens vs words: the real difference, with examples
Understand the practical difference between tokens and words for LLM engineering — billing, context limits, latency, and multilingual behavior with concrete examples.
How tokenizer choice changes your inference bill
Tokenizer choice directly changes token counts — and your inference bill. A practical breakdown of how BPE vs Unigram, code vs prose, and multilingual text shift costs across models.
How tokenization turns text into numbers
A practical guide to how tokenization works in LLMs, covering BPE, WordPiece, special tokens, and engineering tradeoffs with code examples.
How to count tokens before sending an API request
Learn how to count tokens locally before making LLM API calls to control costs, avoid truncation, and optimize context windows.
How many tokens is one word, on average?
A precise answer to how many tokens a word averages, with tokenizer mechanics, model differences, and practical estimation techniques for LLM engineers.
Byte pair encoding explained with real examples
Learn byte pair encoding from scratch with runnable Python code — build a BPE tokenizer step by step and see exactly how subword tokenization works.
More topics in glossary
- Structured Outputs & JSON Mode19
- AI Agents Fundamentals12
- Hallucination in LLMs11
- Sampling Parameters: Top-p, Top-k & Penalties11
- Context Window & Context Length10
- Fine-Tuning Fundamentals9
- Foundation Models: Base vs Instruct vs Chat9
- Model Families & Naming Conventions: GPT-5, Claude, Gemini 3, Llama 4, Mistral, DeepSeek, Qwen, Grok9
- Grounding & Fact-Checking in AI8
- LLM Benchmarks: MMLU, HumanEval, SWE-bench & GPQA8
- Max Tokens, Stop Sequences & Output Truncation8
- Quantization Formats: GGUF, GPTQ, AWQ & INT4/INT88