What GPU Do I Need for Gemma 3 12B Instruct?

Short answer: Gemma 3 12B Instruct at Q4_K_M needs about 9.3 GB of memory with an 8K context, so you want a GPU with at least 12 GB of VRAM. Context length, the quantization you choose and other GPU apps all change that number.

Check your own setup with the Can My GPU Run This AI Model? calculator. The numbers below come from the same data.

How much memory does Gemma 3 12B Instruct need?

Gemma 3 12B Instruct has 12.19 billion parameters, 48 layers and 8 KV heads of size 256. Weights take parameters × bits per weight ÷ 8. The KV cache adds memory as context grows, and the runtime adds about 1 GB.

Quantization Bits per weight Weights Total at 8K context (est.)
Q4_K_M 4.9 7.5 GB 9.3 GB
Q5_K_M 5.7 8.7 GB 10.6 GB
Q6_K 6.6 10.1 GB 11.9 GB
Q8_0 8.5 13.0 GB 14.8 GB

Which GPUs can run it?

At Q4_K_M and an 8K context:

GPU VRAM Result Speed (est., tok/s)
RTX 5090 32 GB Fits 144 to 192
RTX 5080 16 GB Fits 77 to 103
RTX 5070 Ti 16 GB Fits 72 to 96
RTX 5070 12 GB Fits 54 to 72
RTX 5060 Ti 16GB 16 GB Fits 36 to 48
RTX 5060 Ti 8GB 8 GB No –
RTX 5060 8 GB No –
RTX 5050 8 GB No –
RTX 4090 24 GB Fits 81 to 108
RTX 4080 Super 16 GB Fits 59 to 79
RTX 4080 16 GB Fits 58 to 77
RTX 4070 Ti Super 16 GB Fits 54 to 72
RTX 4070 Super 12 GB Fits 41 to 54
RTX 4070 12 GB Fits 41 to 54
RTX 4060 Ti 16GB 16 GB Fits 23 to 31
RTX 4060 Ti 8GB 8 GB No –
RTX 4060 8 GB No –
RTX 3090 24 GB Fits 75 to 100
RTX 3080 10GB 10 GB Tight 61 to 81
RTX 3070 8 GB No –
RTX 3060 Ti 8 GB No –
RTX 3060 12GB 12 GB Fits 29 to 39
RTX 3050 8GB 8 GB No –
RX 7900 XTX 24 GB Fits 77 to 103
RX 7900 XT 20 GB Fits 64 to 86
RX 7800 XT 16 GB Fits 50 to 67
RX 7600 XT 16 GB Fits 23 to 31
RX 7600 8 GB No –
RX 9070 XT 16 GB Fits 51 to 69
RX 9060 XT 16GB 16 GB Fits 26 to 34

Estimates: weights = parameters × bits per weight; KV cache from the model’s published architecture; about 1 GB of runtime overhead. “Fits” means at most 92 percent of VRAM, “Tight” is 92 to 100 percent. Speed is memory bandwidth divided by model size at 60 to 80 percent efficiency. These are planning estimates, not our own benchmarks.

Our pick

The smallest cards that fit it comfortably are the RTX 5070, RTX 4070 Super, RTX 4070, RTX 3060 12GB (12 GB). If you want longer context or a higher-quality quantization, step up to the next memory size. For other models, see the best AI models for 8 GB of VRAM and Q4 vs Q8.

Frequently asked questions

How much VRAM does Gemma 3 12B Instruct need?

About 9.3 GB at Q4_K_M with an 8K context; about 14.8 GB at Q8_0.

Can I run Gemma 3 12B Instruct on a CPU or with system RAM?

Yes, layers can be offloaded to system RAM, but generation gets much slower than keeping everything in VRAM.

Does longer context need more VRAM?

Yes. At 32K context the KV cache alone is about 2.5 GB for this model.

Sources and method

Last updated 10 Nov 2026.

Spynium


Leave a Comment