The RTX 5090 is the top consumer card for local AI. Its 32 GB of GDDR7 and 1,792 GB/s of bandwidth give it both the memory to hold 32B models with long context and the speed to run them quickly. It is also the only mainstream consumer card that clears the 24 GB barrier.
Check your own setup with the Can My GPU Run This AI Model? calculator. The numbers below come from the same data.
RTX 5090 specs that matter for local AI
- Memory: 32 GB GDDR7 on a 512-bit bus
- Memory bandwidth: 1792 GB/s
- Board power: 575 W for the standard model (some factory-overclocked cards list 600 W)
VRAM decides which models fit. Bandwidth decides how fast they generate text, because every new token means reading the whole model from memory.
Which AI models fit on the RTX 5090?
The table assumes Q4_K_M quantization (about 4.9 bits per weight) and an 8K context. Totals include the KV cache and about 1 GB of runtime overhead. Speed is an estimate: memory bandwidth divided by model size, at 60 to 80 percent efficiency, which matches published llama.cpp results on several RTX cards.
| Model | Weights | KV cache (8K) | Total (est.) | On 32 GB | Speed (est.) |
|---|---|---|---|---|---|
| Qwen3 4B | 2.5 GB | 1.2 GB | 4.7 GB | Fits | 437 to 582 tok/s |
| Llama 3.1 8B Instruct | 4.9 GB | 1.1 GB | 7.0 GB | Fits | 219 to 291 tok/s |
| Qwen3 8B | 5.0 GB | 1.2 GB | 7.2 GB | Fits | 214 to 286 tok/s |
| Gemma 3 12B Instruct | 7.5 GB | 0.9 GB | 9.3 GB | Fits | 144 to 192 tok/s |
| Qwen3 14B | 9.0 GB | 1.3 GB | 11.4 GB | Fits | 119 to 158 tok/s |
| OpenAI gpt-oss-20b | 12.8 GB | 0.2 GB | 14.0 GB | Fits | Varies (MoE) |
| Mistral Small 3.2 24B | 14.7 GB | 1.3 GB | 17.0 GB | Fits | 73 to 97 tok/s |
| Qwen3.6 27B | 17.0 GB | 0.5 GB | 18.6 GB | Fits | 63 to 84 tok/s |
| Gemma 3 27B Instruct | 16.8 GB | 1.1 GB | 18.9 GB | Fits | 64 to 85 tok/s |
| Qwen3 32B | 20.1 GB | 2.1 GB | 23.2 GB | Fits | 54 to 71 tok/s |
| Qwen3.6 35B-A3B | 22.0 GB | 0.2 GB | 23.2 GB | Fits | Varies (MoE) |
“Fits” means the estimate is at most 92 percent of VRAM. “Tight” means it is between 92 and 100 percent, so close other GPU apps. Mixture-of-Experts models (gpt-oss-20b and Qwen3.6 35B-A3B) need their full weights in memory, but they only use part of them per token, so speed is not estimated.
Best models for the RTX 5090
- Qwen3 32B (Q4_K_M): about 23 GB at 8K context, leaving close to 9 GB free for a longer context or a higher-quality quantization.
- Gemma 3 27B, Qwen3.6 27B and Mistral Small 3.2 24B: 17 to 19 GB at Q4, so Q6_K versions also fit.
- Qwen3.6 35B-A3B (MoE): all weights must be stored, about 22 GB at Q4-class size, and only part of them runs per token.
- Smaller models at near-original quality: an 8B model at Q8_0 uses about 10.6 GB at 8K.
How context length changes things
The KV cache grows with every token of context, so a model that fits at 4K can fail at 32K.
| Model (Q4_K_M) | Context | KV cache | Total (est.) | On 32 GB |
|---|---|---|---|---|
| Llama 3.1 8B Instruct | 4K | 0.54 GB | 6.5 GB | Fits |
| Llama 3.1 8B Instruct | 8K | 1.07 GB | 7.0 GB | Fits |
| Llama 3.1 8B Instruct | 16K | 2.15 GB | 8.1 GB | Fits |
| Llama 3.1 8B Instruct | 32K | 4.29 GB | 10.2 GB | Fits |
| Qwen3 14B | 4K | 0.67 GB | 10.7 GB | Fits |
| Qwen3 14B | 8K | 1.34 GB | 11.4 GB | Fits |
| Qwen3 14B | 16K | 2.68 GB | 12.7 GB | Fits |
| Qwen3 14B | 32K | 5.37 GB | 15.4 GB | Fits |
A Q8 KV cache (an option in Ollama and llama.cpp) roughly halves the cache cost if you need more context.
What does not fit
70B-class models need roughly 40 GB at Q4 for the weights alone, so they do not fit entirely in 32 GB. Full FP16 weights only work for models up to about 13B.
The no-compromise option. If you want to run 27B to 35B models comfortably, with long context or high quantization, the 32 GB of VRAM is the main reason to buy it. For 8B to 14B models it is overkill.
Verdict
What speed to expect
With 1,792 GB/s, an 8B Q4 model has a ceiling near 365 tokens per second and 32B models near 90. We found no reliable measured result for this card: crowdsourced LocalScore.ai lists about 66 tokens per second for Llama 3.1 8B, which is far below what the bandwidth allows and probably reflects a software or system limit. Until good measurements exist, treat our estimate as a theoretical range: about 220 to 290 tokens per second for 8B and 54 to 72 for a 32B model.
Frequently asked questions
Can the RTX 5090 run a 70B model?
Not entirely on the GPU. Q4 weights are about 40 GB, so part of the model would sit in system memory and slow it down.
Can it run Qwen3 32B?
Yes. At Q4_K_M it needs about 23 GB at 8K context, so it has room for a much longer context.
Is the RTX 5090 worth it for 8B models?
Not for the VRAM. An 8B model fits easily on 8 GB cards. You would be paying for speed and headroom.
Sources and method
- RTX 5090 specs: PNY’s NVIDIA GeForce linecard, January 2026 (32 GB GDDR7, 1,792 GB/s, 575 W).
- Crowdsourced speed: LocalScore.ai Llama 3.1 8B leaderboard.
- Linear-attention and sliding-window layers are handled with reduced KV cache; small fixed states in some newer models are not counted.
- Model architecture: each model’s published
config.jsonon Hugging Face (Llama 3.1 8B and Gemma 3 are read from the unsloth mirrors because the original repositories require a login). - VRAM and speed figures are estimates from our calculator, not our own benchmarks. Last updated 3 Nov 2026.
