Can I Run Mistral Small 3.2 24B on an RTX 4090?

Yes. At Q4_K_M and an 8K context it needs about 17.0 GB, which is 71 percent of the RTX 4090’s 24 GB.

Check your own setup with the Can My GPU Run This AI Model? calculator. The numbers below come from the same data.

Memory needed on the RTX 4090

Quantization Weights Total at 8K (est.) On 24 GB Speed (est., tok/s)
Q4_K_M 14.7 GB 17.0 GB Fits 41 to 55
Q5_K_M 17.1 GB 19.4 GB Fits 35 to 47
Q6_K 19.8 GB 22.2 GB Tight 31 to 41
Q8_0 25.5 GB 27.9 GB No –

The highest quantization that still fits comfortably is Q5_K_M.

Context length matters

Context KV cache Total at Q4_K_M (est.) On 24 GB
4K 0.67 GB 16.4 GB Fits
8K 1.34 GB 17.0 GB Fits
16K 2.68 GB 18.4 GB Fits
32K 5.37 GB 21.1 GB Fits

What speed to expect

With 1008 GB/s of bandwidth, expect about 41 to 55 tokens per second at Q4_K_M. We did not find a reliable measured result for this exact card and model, so treat speeds as estimates that could be off by 10 to 20 percent.

Estimates: weights = parameters × bits per weight; KV cache from the model’s published architecture; about 1 GB of runtime overhead. “Fits” means at most 92 percent of VRAM, “Tight” is 92 to 100 percent. Speed is memory bandwidth divided by model size at 60 to 80 percent efficiency. These are planning estimates, not our own benchmarks.

Verdict

Run it at Q4_K_M and enjoy. For a wider view, see what GPU you need for Mistral Small 3.2 24B and everything the RTX 4090 can run.

Frequently asked questions

Can the RTX 4090 run Mistral Small 3.2 24B at Q8?

Q8_0 needs about 27.9 GB at 8K context: no on 24 GB.

How much context can the RTX 4090 handle with Mistral Small 3.2 24B?

At Q4_K_M, 32K context needs about 21.1 GB in total (fits on 24 GB).

Sources and method

  • Model architecture: Mistral Small 3.2 24B config.json on Hugging Face.
  • GPU specs: TechPowerUp RTX 4090 Founders Edition review (1008 GB/s, 384-bit, 450 W power limit).
  • GPU specs: Notebookcheck RTX 4090 page (1008 GB/s, 384-bit, 450 W).
  • All VRAM and speed figures are estimates, not our own benchmarks.

Last updated 10 Nov 2026.

Spynium


Leave a Comment