What is GGUF? The format behind llama.cpp models

GGUF (GPT-Generated Unified Format) is a single-file model format designed for efficient local inference on consumer hardware. It packages model weights, tokenizer metadata, and quantization parameters into a self-contained binary that loads without external dependencies. The format succeeds GGML and serves as the native serialization target for llama.cpp.

Why GGUF exists

Before GGUF, running LLMs locally meant juggling multiple files: PyTorch safetensors for weights, separate tokenizer.json and config.json files, and often custom quantization scripts that produced incompatible outputs. GGUF solves this by defining a strict binary layout with a versioned header, tensor metadata blocks, and key-value metadata for everything from architecture hyperparameters to tokenizer vocabulary.

The format was introduced in August 2023 by the llama.cpp team to replace GGML. Key improvements include explicit alignment requirements for memory-mapped loading, extensible metadata via a typed key-value store, and built-in support for per-tensor quantization schemes. Because the specification is stable and documented, tools like ollama, llamafile, and kobold.cpp all consume GGUF directly.

File structure at a glance

A GGUF file consists of three regions:

1. Header — magic bytes (GGUF), version (currently 3), tensor count, metadata KV count
2. Metadata key-value store — typed entries (string, uint32, float32, arrays) describing architecture, tokenizer, quantization, and user-defined fields
3. Tensor data — weight buffers laid out sequentially with alignment padding

+------------------+
| Header (24 bytes)|
+------------------+
| Metadata KV pairs|
| (variable)       |
+------------------+
| Alignment padding|
+------------------+
| Tensor 0 weights |
+------------------+
| Tensor 1 weights |
+------------------+
| ...              |
+------------------+

The header is fixed at 24 bytes:

struct GGUFHeader {
    uint32_t magic;      // 0x46554747 ("GGUF")
    uint32_t version;    // 3
    uint64_t tensor_count;
    uint64_t metadata_kv_count;
};

All integers are little-endian. Alignment is 32 bytes by default (configurable via general.alignment metadata), enabling direct mmap on most platforms without copy.

Quantization in GGUF

GGUF does not mandate a single quantization scheme. Instead, each tensor declares its own quantization_type in the metadata block. Common types include:

K-quants (introduced in GGUF v3) use a block-wise scheme with per-block scaling factors and optional importance weights. This yields better perplexity than legacy Q4_0/Q5_0 at similar size. The quantization type is stored per-tensor, so you can quantize attention matrices to Q4_K_M while keeping embeddings at F16 — a common pattern for quality preservation.

To inspect a file’s quantization:

llama-gguf-dump model.gguf | head -40

Output shows each tensor name, shape, type, and offset:

tensor   0: "blk.0.attn_q.weight"  shape: [4096, 4096]  type: Q4_K_M  offset: 0x00001000
tensor   1: "blk.0.attn_k.weight"  shape: [4096, 1024]  type: Q4_K_M  offset: 0x00801000
...

Metadata you actually use

The KV store contains standardized keys that inference engines rely on. Critical entries include:

{
  "general.architecture": "llama",
  "general.file_type": 15,                    // quantization enum
  "general.parameter_count": 7062732800,
  "general.quantization_version": 2,
  "llama.context_length": 8192,
  "llama.embedding_length": 4096,
  "llama.block_count": 32,
  "llama.feed_forward_length": 11008,
  "llama.attention.head_count": 32,
  "llama.attention.head_count_kv": 8,
  "llama.rope.dimension_count": 128,
  "llama.rope.freq_base": 1000000.0,
  "tokenizer.ggml.model": "llama",
  "tokenizer.ggml.tokens": ["<|endoftext|>", "▁The", ...],
  "tokenizer.ggml.scores": [0.0, -0.2, ...],
  "tokenizer.ggml.token_type": [3, 1, ...],
  "tokenizer.ggml.merges": ["a b", "c d", ...],
  "tokenizer.ggml.bos_token_id": 1,
  "tokenizer.ggml.eos_token_id": 2
}

Custom keys are allowed — tools prefix with com.example. to avoid collisions. This extensibility lets downstream applications store prompt templates, chat formats, or training hyperparameters without breaking loaders.

Converting to GGUF

Most users convert from Hugging Face safetensors using the official conversion script:

python convert-hf-to-gguf.py \
  --model-dir /path/to/hf-model \
  --outfile model.gguf \
  --outtype q4_k_m

The --outtype flag accepts f32, f16, q4_k_m, q5_k_m, q8_0, and others. For per-tensor control, pass a JSON quantization config:

{
  "*.weight": "q4_k_m",
  "token_embd.weight": "f16",
  "output.weight": "f16",
  "*.bias": "f32"
}

python convert-hf-to-gguf.py \
  --model-dir /path/to/hf-model \
  --outfile model.gguf \
  --quantization-config quant_config.json

The script validates tensor names against the architecture metadata, computes quantization scales, and writes the aligned binary. Conversion typically takes 2–10 minutes for 7B–70B models on a modern CPU.

Loading in llama.cpp

Loading a GGUF model requires minimal code:

#include "llama.h"

int main() {
    llama_backend_init();
    
    llama_model_params model_params = llama_model_default_params();
    model_params.n_gpu_layers = -1;  // offload all to GPU
    model_params.use_mmap = true;    // memory-map the file
    model_params.use_mlock = true;   // prevent swapping
    
    struct llama_model *model = llama_load_model_from_file(
        "model.gguf", model_params
    );
    
    if (!model) {
        fprintf(stderr, "Failed to load model\n");
        return 1;
    }
    
    llama_context_params ctx_params = llama_context_default_params();
    ctx_params.n_ctx = 8192;
    ctx_params.n_batch = 512;
    
    struct llama_context *ctx = llama_new_context_with_model(model, ctx_params);
    
    // ... inference loop ...
    
    llama_free(ctx);
    llama_free_model(model);
    llama_backend_free();
    return 0;
}

The use_mmap flag is critical for large models — it lets the OS page tensor data on demand rather than loading the entire file into RAM. With n_gpu_layers = -1, llama.cpp offloads all compatible tensors to the GPU via Metal, CUDA, or Vulkan, keeping only the KV cache and a few small tensors on CPU.

Python bindings expose the same knobs:

from llama_cpp import Llama

llm = Llama(
    model_path="model.gguf",
    n_gpu_layers=-1,
    n_ctx=8192,
    n_batch=512,
    use_mmap=True,
    use_mlock=True,
    verbose=False
)

output = llm("The capital of France is", max_tokens=32, stop=["."])
print(output["choices"][0]["text"])

Why GGUF matters for deployment

Single-file distribution eliminates dependency drift. A GGUF model produced today loads in llama.cpp from six months ago (forward compatibility within major version) and in any GGUF-compatible runtime. This is a stark contrast to PyTorch ecosystems where torch.load breaks across minor versions.

Memory-mapped loading enables running models larger than RAM. A 70B Q4_K_M model (~39 GB) runs on a 32 GB machine because only active layers are paged in. The alignment guarantee means zero-copy access on Linux, macOS, and Windows.

Per-tensor quantization lets you optimize the quality/size frontier. Typical recipe for a 7B model:

• Attention projections (q_proj, k_proj, v_proj, o_proj): Q4_K_M
• MLP gates/up/down: Q4_K_M
• Embeddings (token_embd, output): F16
• Norm weights: F32 (numerically sensitive)
• Biases: F32

This yields ~4.2 GB on disk with negligible perplexity loss versus F16 (~13 GB).

Common misconceptions

GGUF is not a quantization algorithm. It is a container format. The quantization schemes (K-quants, legacy quants) are implemented in llama.cpp and referenced by enum in the metadata. You can store F32 tensors in GGUF — it’s just inefficient.

GGUF does not require llama.cpp. The specification is open. Implementations exist in Rust (candle, llm), Go (go-llama.cpp), C# (LlamaSharp), and JavaScript (llama.cpp.wasm). Any parser that respects the header, KV store, and tensor layout can load GGUF.

GGUF v3 is not backward compatible with GGML. GGML files use a different magic (GGML), different header layout, and no typed KV store. Tools that claim “GGML/GGUF support” typically handle both by detecting the magic bytes and dispatching to separate parsers.

You cannot arbitrarily quantize a GGUF file. Re-quantization requires dequantizing to F32/F16 first, then applying a new scheme. The llama-quantize tool does this:

llama-quantize model.f16.gguf model.q4_k_m.gguf Q4_K_M

But feeding a Q4_K_M file back into llama-quantize for Q3_K_M dequantizes then requantizes, accumulating error. Always quantize from the highest-precision source available.

Tokenizer is embedded, not optional. The metadata includes the full vocabulary, merge rules, and special token IDs. You do not need a separate tokenizer.json. This is why llama.cpp can tokenize without any external files.

When to choose GGUF over alternatives

GGUF dominates local inference because it solves the full stack: serialization, quantization, tokenization, and hardware-agnostic execution in one file. For server workloads where throughput per dollar matters more than portability, calibrated FP8 or GPTQ on H100s still wins — but that’s a different deployment model.

Practical tips

• Verify before serving: run llama-gguf-dump model.gguf --verify to catch truncated downloads or corruption.
• Strip unused tensors: some converters include rope_freqs or positional embeddings that llama.cpp computes on the fly. The --no-lazy flag in conversion keeps them; omitting it saves ~100 MB on 70B models.
• Pin llama.cpp version: GGUF v3 is stable, but new quantization types (e.g., IQ4_XS) require matching loader support. Build llama.cpp from the same release branch used for conversion.
• Use llama-perplexity for quality checks: llama-perplexity -m model.gguf -f wikitext-2-raw.txt gives a quick sanity metric before deploying a new quant.

GGUF is the reason local LLM inference works on a laptop. It packages the model, tokenizer, and quantization decisions into a single memory-mappable file with a stable specification. If you’re shipping a model to run on user devices — or just want to run Llama-3-70B on your workstation without Docker — GGUF is the format you’ll use.