The RTX 4090 is one of the two most capable consumer cards for local AI (only the RTX 5090, with 32 GB and 1,792 GB/s, is higher). It combines 24 GB of GDDR6X with 1,008 GB/s of bandwidth, so it runs 8B models extremely fast and can hold 24B to 32B models that smaller cards cannot.
Check your own setup with the Can My GPU Run This AI Model? calculator. The numbers below come from the same data.
RTX 4090 specs that matter for local AI
- Memory: 24 GB GDDR6X on a 384-bit bus
- Memory bandwidth: 1008 GB/s
- Power limit: 450 W
- Note: the RTX 3090 also has 24 GB, but its bandwidth is 936 GB/s.
VRAM decides which models fit. Bandwidth decides how fast they generate text, because every new token means reading the whole model from memory.
Which AI models fit on the RTX 4090?
The table assumes Q4_K_M quantization (about 4.9 bits per weight) and an 8K context. Totals include the KV cache and about 1 GB of runtime overhead. Speed is an estimate: memory bandwidth divided by model size, at 60 to 80 percent efficiency, which matches published llama.cpp results on several RTX cards.
| Model | Weights | KV cache (8K) | Total (est.) | On 24 GB | Speed (est.) |
|---|---|---|---|---|---|
| Qwen3 4B | 2.5 GB | 1.2 GB | 4.7 GB | Fits | 246 to 328 tok/s |
| Llama 3.1 8B Instruct | 4.9 GB | 1.1 GB | 7.0 GB | Fits | 123 to 164 tok/s |
| Qwen3 8B | 5.0 GB | 1.2 GB | 7.2 GB | Fits | 121 to 161 tok/s |
| Gemma 3 12B Instruct | 7.5 GB | 0.9 GB | 9.3 GB | Fits | 81 to 108 tok/s |
| Qwen3 14B | 9.0 GB | 1.3 GB | 11.4 GB | Fits | 67 to 89 tok/s |
| OpenAI gpt-oss-20b | 12.8 GB | 0.2 GB | 14.0 GB | Fits | Varies (MoE) |
| Mistral Small 3.2 24B | 14.7 GB | 1.3 GB | 17.0 GB | Fits | 41 to 55 tok/s |
| Qwen3.6 27B | 17.0 GB | 0.5 GB | 18.6 GB | Fits | 36 to 47 tok/s |
| Gemma 3 27B Instruct | 16.8 GB | 1.1 GB | 18.9 GB | Fits | 36 to 48 tok/s |
| Qwen3 32B | 20.1 GB | 2.1 GB | 23.2 GB | Tight | 30 to 40 tok/s |
| Qwen3.6 35B-A3B | 22.0 GB | 0.2 GB | 23.2 GB | Tight | Varies (MoE) |
“Fits” means the estimate is at most 92 percent of VRAM. “Tight” means it is between 92 and 100 percent, so close other GPU apps. Mixture-of-Experts models (gpt-oss-20b and Qwen3.6 35B-A3B) need their full weights in memory, but they only use part of them per token, so speed is not estimated.
Best models for the RTX 4090
- Mistral Small 3.2 24B (Q4_K_M): about 17 GB at 8K context. Fits with room for longer context.
- Gemma 3 27B and Qwen3.6 27B (Q4_K_M): about 19 GB at 8K. Both use cache-saving attention designs, so context is cheap.
- Qwen3 32B (Q4_K_M): about 23 GB at 8K. It fits, but only just, so keep context short.
- gpt-oss-20b: about 14 GB with a very small KV cache.
- 8B to 14B models at higher quality: Q8_0 of an 8B model (about 8.5 GB) leaves plenty of space to spare.
How context length changes things
The KV cache grows with every token of context, so a model that fits at 4K can fail at 32K.
| Model (Q4_K_M) | Context | KV cache | Total (est.) | On 24 GB |
|---|---|---|---|---|
| Llama 3.1 8B Instruct | 4K | 0.54 GB | 6.5 GB | Fits |
| Llama 3.1 8B Instruct | 8K | 1.07 GB | 7.0 GB | Fits |
| Llama 3.1 8B Instruct | 16K | 2.15 GB | 8.1 GB | Fits |
| Llama 3.1 8B Instruct | 32K | 4.29 GB | 10.2 GB | Fits |
| Qwen3 14B | 4K | 0.67 GB | 10.7 GB | Fits |
| Qwen3 14B | 8K | 1.34 GB | 11.4 GB | Fits |
| Qwen3 14B | 16K | 2.68 GB | 12.7 GB | Fits |
| Qwen3 14B | 32K | 5.37 GB | 15.4 GB | Fits |
A Q8 KV cache (an option in Ollama and llama.cpp) roughly halves the cache cost if you need more context.
What does not fit
Models above about 32B parameters at Q4 need more than 24 GB. Full FP16 weights are also out of reach for anything beyond roughly 10B (FP16 needs about 2 GB per billion parameters).
A top-tier card for local AI. 24 GB is the difference between running 8B models and running 24B to 32B models. If you only need 8B to 14B models, a cheaper 12 GB or 16 GB card is enough. If you want 27B and 32B models without compromise, 24 GB is the practical starting point.
Verdict
What speed to expect
At 1,008 GB/s, an 8B Q4 model has a ceiling near 205 tokens per second. Measured results vary. A community llama.cpp dataset (May 2024) lists 130.6 tokens per second for Llama 3 8B Q4_K_M and 54.8 for FP16. Crowdsourced LocalScore.ai results for Llama 3.1 8B are lower, in the range of about 90 to 120 tokens per second, which shows how much real results depend on software and settings. Our estimate is 123 to 164 tokens per second for an 8B model and about 36 to 48 for 27B models.
Frequently asked questions
Can an RTX 4090 run a 70B model?
Not fully on the GPU. A 70B model at Q4 needs about 40 GB for the weights alone, so much of it would run from system memory and be slow.
Can it run Qwen3 32B?
Yes at Q4_K_M with a short context. The total is about 23 GB at 8K, which is close to the 24 GB limit.
Is the RTX 4090 faster than the RTX 3090 for AI?
Yes. Its bandwidth is higher (1,008 vs 936 GB/s) and measured llama.cpp results show it ahead, but the memory size is the same, so both fit the same models.
Sources and method
- RTX 4090 specs: TechPowerUp’s RTX 4090 Founders Edition review and Notebookcheck’s RTX 4090 page (1,008 GB/s, 384-bit, 450 W).
- Measured speeds: XiongjieDai’s GPU-Benchmarks-on-LLM-Inference on GitHub (May 2024 snapshot) and LocalScore.ai.
- Linear-attention and sliding-window layers are handled with reduced KV cache; small fixed states in some newer models are not counted.
- Model architecture: each model’s published
config.jsonon Hugging Face (Llama 3.1 8B and Gemma 3 are read from the unsloth mirrors because the original repositories require a login). - VRAM and speed figures are estimates from our calculator, not our own benchmarks. Last updated 3 Nov 2026.
