Quantization: run a big model on a small GPU

// the mechanism

A model is a giant pile of numbers — the weights. Quantization stores each one in fewer bits. Full precision is 2 bytes per weight (FP16); drop to 8-bit and it halves; drop to 4-bit and it halves again. Fewer bits, smaller model.

one weight: 0.7341 (16 bits) -> 0.7 (4 bits)

The memory math

Memory ≈ params × bytes-per-weight. So a 7-billion-parameter model shrinks fast — and drops under the line of a consumer GPU:

16-bit 14 GB # full precision — won't fit an 8GB card 8-bit 7 GB 4-bit ~4 GB # now it fits

This is what the formats do under the hood — GGUF (llama.cpp / Ollama), GPTQ, AWQ. Pull a model with Ollama and it's quantized by default.

Those numbers are weights only

This is the part that catches people out, and it's a fair objection to every "4-bit 7B fits in 4GB" chart on the internet — including the one above. Weights are the fixed cost. The other cost is the KV cache, and it grows linearly with your context length:

# bytes of KV cache per token 2 (K and V) × n_layers × n_kv_heads × head_dim × dtype_bytes

Put Llama-3-8B's numbers in — 32 layers, 8 KV heads (it uses grouped-query attention), head_dim 128, fp16 — and you get 128 KiB per token. That is small until you multiply it by a context window:

# Llama-3-8B, fp16 cache 8k ctx ~1 GB # fine 32k ctx ~4 GB # now equal to the 4-bit weights 128k ctx ~16 GB # four times the weights. this is what OOMs.

So the honest version of the memory math is weights + cache. Quantizing the weights to 4-bit and then asking for a 128k window doesn't fit a 24GB card — the thing you shrank stopped being the thing that costs. Two more details that decide whether this bites you:

Grouped-query attention is doing most of the saving. Llama-3-8B has 8 KV heads for 32 query heads. Older models without GQA (Llama 2 7B, 32 KV heads) pay roughly four times as much per token for the same context. Two models of the same parameter count can have wildly different cache costs.

Ollama allocates num_ctx up front, not on demand. Setting a 32k window reserves the memory whether your prompt is 200 tokens or 30,000. Raising num_ctx "just in case" is a real, immediate VRAM charge.

Quantize the cache too — but the flag has a dependency. OLLAMA_KV_CACHE_TYPE=q8_0 halves cache memory for very little quality cost. It silently does nothing unless OLLAMA_FLASH_ATTENTION=1 is also set, and it falls back to fp16 on architectures that don't support it. No error, no warning — you set the variable, get no saving, and still OOM. Set both, then check your actual VRAM use.

# both, or neither does anything OLLAMA_FLASH_ATTENTION=1 OLLAMA_KV_CACHE_TYPE=q8_0 # q4_0 also exists; more quality risk

Sources: Ollama PR #6279 — KV cache quantization · Ollama #13337 — the flash-attention dependency · smcleod.net — bringing KV context quantisation to Ollama

Thanks to richardhuelsberg, who raised exactly this under the video.

The catch

Fewer bits means lower quality, and it bites hardest at 4-bit. 8-bit is usually a nearly-free drop; naive 4-bit shows up as subtle slips — arithmetic, edge cases, reasoning:

what is 17% of 240? 16-bit -> 40.8 ✓ 8-bit -> 40.8 ✓ 4-bit -> 42 ✗ # subtle slips

Not the same thing as distillation or pruning — those are different methods that change the model itself. And the exact quality cost varies by model and method: GPTQ and AWQ protect "important" weights far better than naive rounding does.

The rule

16-bit top quality / research 8-bit the safe default 4-bit when you must fit

Use 8-bit when you can, 4-bit when you have to fit — and set your context window to what you actually use, because that is the other half of the bill. It's the lever that turns "I need a datacenter GPU" into "it runs on my laptop" — pairs directly with running models locally.

Sources: Hugging Face — Quantization concepts · llama.cpp / GGUF · GPTQ & AWQ papers

One concept a week. Free.

The deeper, copy-paste version of each ToolCall short — in your inbox.

// total: 0.00 · spam: void · unsubscribe: one click