Quantization stores a model’s weights with fewer bits, so it takes less VRAM. The usual rule: use the highest quantization that fits comfortably in your VRAM, and treat Q4_K_M as the default. This page shows the memory cost of each option.
Check your own card with the Can My GPU Run This AI Model? calculator.
Memory use by quantization (Llama 3.1 8B)
| Quantization | Bits per weight (approx.) | Weights | Total at 8K context (est.) |
|---|---|---|---|
| Q4_K_M | 4.9 bits | 4.9 GB | 7.0 GB |
| Q5_K_M | 5.7 bits | 5.7 GB | 7.8 GB |
| Q6_K | 6.6 bits | 6.6 GB | 8.7 GB |
| Q8_0 | 8.5 bits | 8.5 GB | 10.6 GB |
| FP16 | 16 bits | 16.1 GB | 18.1 GB |
The weight sizes match published GGUF files (Q4_K_M 4.92 GB, Q8_0 8.54 GB). Totals add the KV cache and about 1 GB of overhead.
Which one should you pick?
- 8 GB cards: Q4_K_M for 8B models. Q5_K_M only with a short context.
- 12 GB cards: Q8_0 of an 8B model fits (about 10.6 GB), or Q4_K_M of a 12B to 14B model.
- 16 GB cards: Q8_0 of 8B to 12B models, or Q4_K_M of a 14B to 20B model.
- 24 GB cards: Q4_K_M of a 24B to 32B model, or higher quantization for smaller models.
Quality versus size
Q8_0 is nearly lossless. Q6_K is very close. Q5_K_M and Q4_K_M give up a little quality in exchange for much smaller files. Going below Q4 saves more memory but quality drops faster, so a bigger model at Q4 is usually better than a smaller model at Q8 when both fit.
Speed
Smaller quantizations also generate faster, because each token reads less memory. An 8B model at Q4_K_M reads about 4.9 GB per token, versus 8.5 GB at Q8_0, so Q4 is roughly 1.7 times faster on the same card.
Frequently asked questions
What does K_M mean in Q4_K_M?
It is a llama.cpp quantization scheme that keeps more precision for the most sensitive weights. M stands for medium.
Should I use FP16?
Only if you have the memory. FP16 needs about 2 GB per billion parameters.
Is Q8 worth it over Q4?
For close-to-original quality, yes, if it fits. Otherwise Q4_K_M is the practical default.
Sources and method
- GGUF file sizes: bartowski’s Llama 3.1 8B GGUF repository, as reported by Markaicode.
- Model architecture: each model’s published
config.jsonon Hugging Face (Llama 3.1 8B and Gemma 3 are read from the unsloth mirrors because the original repositories require a login).
Estimates: weights = parameters × bits per weight; KV cache from the model’s published architecture; about 1 GB of runtime overhead. “Fits” means at most 92 percent of VRAM, “Tight” is 92 to 100 percent. Speed is memory bandwidth divided by model size at 60 to 80 percent efficiency. These are planning estimates, not our own benchmarks.
Last updated 3 Nov 2026.
