GGUF vs GGML: what changed and why

GGML was the format that made local LLM inference practical. GGUF replaced it entirely. If you’re loading models in llama.cpp, ollama, or any GGML-derived runtime today, you’re using GGUF — even when the file extension says .gguf and the quantization label reads Q4_K_M. The transition wasn’t a minor revision; it was a complete redesign of the container format, quantization scheme, and metadata model. Understanding what changed tells you why certain quantizations behave differently, why your old model files stopped working, and how to pick the right artifact for your hardware.

The short history

Georgi Gerganov released GGML (Georgi Gerganov Machine Learning) in March 2023 as a C library for CPU inference with 4-bit quantization. It used a custom binary format: a flat header followed by tensor data, with quantization type encoded as a single enum per tensor. The format worked but had hard limits — no metadata extensibility, no support for new quantization schemes without breaking parsers, and no clean way to store tokenizer config or model architecture details alongside weights.

By August 2023, GGUF (GGML Universal Format) shipped in llama.cpp PR #2976. It deprecated GGML immediately. The last GGML-compatible release was llama.cpp b247; everything since expects GGUF. If you have .bin files from mid-2023, they’re GGML. They will not load in current tooling without conversion.

Container architecture: flat vs. key-value

GGML used a fixed-layout header:

struct GGMLHeader {
    uint32_t magic;        // 0x67676d6c ("ggml")
    uint32_t version;      // 1
    uint64_t n_tensors;
    // ... fixed fields for each tensor: name, n_dims, dims[], type, offset
};

Every tensor entry had identical structure. Adding a field meant version bump and parser updates everywhere.

GGUF introduces a key-value metadata section before tensor data:

struct GGUFHeader {
    uint32_t magic;        // 0x46554747 ("GGUF")
    uint32_t version;      // 2 (current), 3 (in development)
    uint64_t n_tensors;
    uint64_t n_kv;         // number of key-value pairs
    // KV pairs follow: key (string), type (enum), value (varies)
    // Tensor infos follow: name, n_dims, dims[], type, offset
    // Tensor data follows (aligned)
};

The KV section stores architecture (general.architecture = "llama"), tokenizer config (tokenizer.ggml.tokens, tokenizer.ggml.scores, tokenizer.ggml.merges), quantization parameters, and arbitrary user metadata. Parsers ignore unknown keys — forward compatibility by design.

Quantization scheme overhaul

GGML quantization types were a flat enum: Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, etc. Each encoded block size, quantization method, and bit width in the name. Adding Q3_K or Q6_K required new enum values and parser updates.

GGUF separates quantization type from block layout. The type enum now includes Q2_K, Q3_K_S, Q3_K_M, Q3_K_L, Q4_K_S, Q4_K_M, Q5_K_S, Q5_K_M, Q6_K, Q8_K — the “K-quants” — plus legacy types for compatibility. Each K-quant uses a super-block structure: 256 weights per block, with per-block scaling factors and optional importance weights (the _S, _M, _L suffixes denote small/medium/large importance matrix sizes).

This matters for quality. Q4_K_M typically matches or exceeds old Q5_0 perplexity at smaller size. Q3_K_L is usable for 7B models where old Q4_0 collapsed. The K-quants also expose their internal parameters (block size, scale bits, importance matrix dimensions) in the GGUF metadata, so a parser can decode them without hardcoding every variant.

Memory mapping and alignment

GGML tensors were packed sequentially with no alignment guarantees. Mmap worked but could cause unaligned loads on some architectures (especially ARM NEON), forcing fallback to memcpy.

GGUF aligns tensor data to 32-byte boundaries by default (configurable via general.alignment). The header specifies each tensor’s offset explicitly. This makes mmap-safe zero-copy loads reliable across x86-64, ARM64, and Apple Silicon. llama.cpp’s ggml_backend layer now assumes aligned offsets — a subtle but real performance win on M-series Macs and Raspberry Pi 5.

Tokenizer and vocabulary in-container

GGML stored tokenizer merges and vocabulary in separate files (tokenizer.model, tokenizer.json) or baked them into the model name convention. Mismatched tokenizer/model pairs were a common failure mode.

GGUF embeds the full tokenizer: tokenizer.ggml.tokens (byte-array tokens), tokenizer.ggml.scores (float scores), tokenizer.ggml.merges (BPE merge rules), tokenizer.ggml.token_type (control/normal/unknown), and tokenizer.ggml.bos_token_id / eos_token_id / unk_token_id / pad_token_id. A GGUF file is self-contained — llama-cli -m model.gguf works without external files. This also enables tooling like llama-gguf-split to shard models while preserving tokenizer integrity.

Metadata extensibility in practice

Because GGUF KV pairs are typed (uint8, int8, uint16, int16, uint32, int32, float32, bool, string, array), you can embed arbitrary data without breaking parsers. Common extensions:

Custom fine-tunes add finetune.dataset, finetune.epochs, finetune.learning_rate. Quantization scripts write quantization.method, quantization.version, quantization.datetime. None of this requires parser changes.

Conversion and tooling

The canonical converter is llama.cpp/convert-hf-to-gguf.py. It reads Hugging Face safetensors or pytorch_model.bin, applies quantization via llama-quantize, and writes GGUF. A minimal invocation:

python convert-hf-to-gguf.py /path/to/hf-model \
  --outfile model-Q4_K_M.gguf \
  --outtype q4_k_m

llama-quantize re-quantizes existing GGUF files:

./llama-quantize model-F16.gguf model-Q4_K_M.gguf Q4_K_M

GGML files convert via the deprecated ggml-convert tool (removed in recent llama.cpp). If you have legacy .bin files, use an older llama.cpp checkout (commit b247 or tag b247) to convert to FP16 GGUF, then re-quantize.

Ecosystem support matrix

Anything built on llama.cpp after August 2023 is GGUF-only. The ecosystem consolidated fast because the format solved real interop problems.

Performance implications

Quantization quality dominates latency/throughput more than container format. But GGUF’s alignment and K-quants yield measurable differences:

• Memory bandwidth: K-quants pack more effective bits per weight. Q4_K_M ≈ 4.5 bpw vs old Q4_0 at 4.0 bpw, with better perplexity. Smaller model → more layers fit in cache → higher tokens/sec on memory-bound CPU inference.
• Mmap cold start: Aligned offsets eliminate the first-token memcpy penalty on ARM. Typical improvement: 10-30ms on Raspberry Pi 4, negligible on desktop x86.
• Quantization speed: llama-quantize is single-threaded. Converting a 70B FP16 → Q4_K_M takes 8-12 minutes on a 16-core Ryzen. GGML quantization was similarly single-threaded; no regression.

GPU offload (Metal, CUDA, Vulkan, SYCL) depends on the backend’s kernel support for each quantization type. As of llama.cpp b4000+, all K-quants have Metal and CUDA kernels. Vulkan covers Q4_K_M, Q5_K_M, Q8_K. SYCL (Intel GPU) covers Q4_K_M and Q8_K. Legacy GGML types (Q4_0, Q5_0, Q8_0) are still supported but not optimized for new backends.

Common failure modes

Mismatched tokenizer: Loading a GGUF with embedded tokenizer that differs from the HF original (some quantizers strip or truncate). Symptom: garbage output, repeated tokens, early EOS. Fix: verify tokenizer.ggml.tokens length matches vocab size; re-convert from HF with --vocab-only to inspect.

Architecture misidentification: general.architecture set to llama for a Mistral or Qwen model. Symptom: wrong RoPE scaling, wrong attention layout, silent quality degradation. Fix: llama-gguf-dump model.gguf | grep architecture; correct with llama-gguf-set-architecture if needed.

Quantization type unsupported on backend: Running Q6_K on Vulkan backend without kernel. Symptom: fallback to CPU, 10x slowdown. Fix: use Q4_K_M or Q8_K for Vulkan; check llama.cpp/ggml-backend-*.c for kernel coverage.

Version skew: GGUF v3 (in development) adds tensor-level quantization overrides and sparse tensor support. Current llama.cpp reads v3 but writes v2. If you hand-edit metadata to v3, older parsers may reject the file.

Comparison table

Which to choose

You are downloading a model today for llama.cpp / ollama / llama-cpp-python
→ GGUF. There is no alternative. Pick Q4_K_M as default for 7B–70B models; Q5_K_M if you have VRAM/RAM headroom and want quality; Q3_K_L for 7B on 8GB systems; Q8_K for near-FP16 quality with 25% size reduction.

You have legacy .bin GGML files from 2023
→ Convert them. Check out llama.cpp at tag b247, run ./ggml-convert model.bin model-f16.gguf, then ./llama-quantize model-f16.gguf model-q4_k_m.gguf Q4_K_M. Delete the .bin.

You are building a new inference engine or model format
→ Study GGUF’s KV design. It solves the “container versioning” problem cleanly: typed KV pairs, explicit tensor offsets, alignment guarantees. The spec is in llama.cpp/gguf.py and ggml/include/gguf.h — readable, no external dependencies.

You are quantizing for GPU (CUDA/Metal) with maximum throughput
→ Q4_K_M or Q8_K. Both have highly optimized kernels. Q4_K_M gives ~3.5x memory reduction over FP16 with <1% perplexity delta on most benchmarks. Q8_K is effectively indistinguishable from FP16.

You are quantizing for CPU-only, memory-constrained (Apple Silicon unified memory, Raspberry Pi, laptop)
→ Q3_K_L for 7B models (fits 7B in ~4.5GB), Q4_K_M for 13B+ (fits 13B in ~8GB). Avoid Q2_K — quality collapse is severe below 3bpw for current architectures.

You need to embed custom metadata (dataset, training config, eval results)
→ GGUF. Add KV pairs via llama-gguf-set-kv or modify convert-hf-to-gguf.py to inject them. Downstream tools preserve unknown keys.

The format war ended in August 2023. GGUF won because it solved the right problems: extensibility without fragmentation, self-contained tokenizer, quantization schemes that actually improve quality-per-bit, and mmap-friendly layout. If you’re working with local LLMs, you’re already using it. The only decision left is which K-quant fits your hardware budget.