The RTX 3090 remains one of the most talked-about cards for local AI for one reason: 24 GB of VRAM. With 936 GB/s of bandwidth it is only slightly behind the RTX 4090 in memory-bound speed, and it fits the same models.
Check your own setup with the Can My GPU Run This AI Model? calculator. The numbers below come from the same data.
RTX 3090 specs that matter for local AI
- Memory: 24 GB GDDR6X on a 384-bit bus
- Memory bandwidth: 936 GB/s
- Note: check the card’s power requirements and condition before buying one used.
VRAM decides which models fit. Bandwidth decides how fast they generate text, because every new token means reading the whole model from memory.
Which AI models fit on the RTX 3090?
The table assumes Q4_K_M quantization (about 4.9 bits per weight) and an 8K context. Totals include the KV cache and about 1 GB of runtime overhead. Speed is an estimate: memory bandwidth divided by model size, at 60 to 80 percent efficiency, which matches published llama.cpp results on several RTX cards.
| Model | Weights | KV cache (8K) | Total (est.) | On 24 GB | Speed (est.) |
|---|---|---|---|---|---|
| Qwen3 4B | 2.5 GB | 1.2 GB | 4.7 GB | Fits | 228 to 304 tok/s |
| Llama 3.1 8B Instruct | 4.9 GB | 1.1 GB | 7.0 GB | Fits | 114 to 152 tok/s |
| Qwen3 8B | 5.0 GB | 1.2 GB | 7.2 GB | Fits | 112 to 149 tok/s |
| Gemma 3 12B Instruct | 7.5 GB | 0.9 GB | 9.3 GB | Fits | 75 to 100 tok/s |
| Qwen3 14B | 9.0 GB | 1.3 GB | 11.4 GB | Fits | 62 to 83 tok/s |
| OpenAI gpt-oss-20b | 12.8 GB | 0.2 GB | 14.0 GB | Fits | Varies (MoE) |
| Mistral Small 3.2 24B | 14.7 GB | 1.3 GB | 17.0 GB | Fits | 38 to 51 tok/s |
| Qwen3.6 27B | 17.0 GB | 0.5 GB | 18.6 GB | Fits | 33 to 44 tok/s |
| Gemma 3 27B Instruct | 16.8 GB | 1.1 GB | 18.9 GB | Fits | 33 to 45 tok/s |
| Qwen3 32B | 20.1 GB | 2.1 GB | 23.2 GB | Tight | 28 to 37 tok/s |
| Qwen3.6 35B-A3B | 22.0 GB | 0.2 GB | 23.2 GB | Tight | Varies (MoE) |
“Fits” means the estimate is at most 92 percent of VRAM. “Tight” means it is between 92 and 100 percent, so close other GPU apps. Mixture-of-Experts models (gpt-oss-20b and Qwen3.6 35B-A3B) need their full weights in memory, but they only use part of them per token, so speed is not estimated.
Best models for the RTX 3090
- Mistral Small 3.2 24B (Q4_K_M): about 17 GB at 8K.
- Gemma 3 27B and Qwen3.6 27B (Q4_K_M): about 19 GB at 8K.
- Qwen3 32B (Q4_K_M): about 23 GB at 8K, so it fits with a short context.
- gpt-oss-20b: about 14 GB.
How context length changes things
The KV cache grows with every token of context, so a model that fits at 4K can fail at 32K.
| Model (Q4_K_M) | Context | KV cache | Total (est.) | On 24 GB |
|---|---|---|---|---|
| Llama 3.1 8B Instruct | 4K | 0.54 GB | 6.5 GB | Fits |
| Llama 3.1 8B Instruct | 8K | 1.07 GB | 7.0 GB | Fits |
| Llama 3.1 8B Instruct | 16K | 2.15 GB | 8.1 GB | Fits |
| Llama 3.1 8B Instruct | 32K | 4.29 GB | 10.2 GB | Fits |
| Qwen3 14B | 4K | 0.67 GB | 10.7 GB | Fits |
| Qwen3 14B | 8K | 1.34 GB | 11.4 GB | Fits |
| Qwen3 14B | 16K | 2.68 GB | 12.7 GB | Fits |
| Qwen3 14B | 32K | 5.37 GB | 15.4 GB | Fits |
A Q8 KV cache (an option in Ollama and llama.cpp) roughly halves the cache cost if you need more context.
What does not fit
70B-class models need about 40 GB at Q4, so they do not fit on one card. Full FP16 weights work only for models up to about 10B.
The value pick for 24 GB. It gives you the same memory as an RTX 4090 and not much less speed. The catches are age, power draw and condition if you buy used.
Verdict
What speed to expect
A community llama.cpp dataset (May 2024) measured the RTX 3090 at 115.4 tokens per second on Llama 3 8B Q4_K_M and 47.4 on FP16, compared with 130.6 and 54.8 for the RTX 4090. The ceiling from bandwidth is about 190 tokens per second, and our estimate for 8B is 114 to 152. For a 27B model at Q4, expect roughly 33 to 45 tokens per second.
Frequently asked questions
Is the RTX 3090 still good for AI?
Yes. 24 GB is what decides which models fit, and the 3090 has it.
RTX 3090 vs RTX 4090 for AI?
Same memory size. The 4090 is faster (measured 130.6 vs 115.4 tokens per second on Llama 3 8B Q4_K_M), but the same models fit on both.
Can it run Qwen3 32B?
Yes at Q4_K_M with a short context, about 23 GB at 8K.
Sources and method
- RTX 3090 specs: Guru3D’s RTX 3080 and 3090 specs article (936 GB/s).
- Measured speeds: XiongjieDai’s GPU-Benchmarks-on-LLM-Inference on GitHub (llama.cpp, May 2024 snapshot).
- Linear-attention and sliding-window layers are handled with reduced KV cache; small fixed states in some newer models are not counted.
- Model architecture: each model’s published
config.jsonon Hugging Face (Llama 3.1 8B and Gemma 3 are read from the unsloth mirrors because the original repositories require a login). - VRAM and speed figures are estimates from our calculator, not our own benchmarks. Last updated 3 Nov 2026.
