Thinking about running an AI model locally? Choose your graphics card and a model to see if it fits in VRAM, how much memory it needs and a rough speed estimate.
Can My GPU Run This AI Model?
Pick your graphics card and a local AI model to see if it fits in VRAM, how much memory it needs and roughly how fast it will run.
GPU and AI model FAQ
How much VRAM do I need to run an AI model?
Roughly the model file size plus the KV cache for your context length plus about 1 GB of runtime overhead. A 8B model at Q4 quantization needs around 5 GB for the weights, so an 8 GB card runs it comfortably at a moderate context.
What is quantization?
Quantization stores model weights with fewer bits. Q4_K_M uses about 4.9 bits per weight and cuts memory to roughly a third of FP16 with a small quality loss. Q8_0 is almost lossless but needs about twice the memory of Q4.
What is the KV cache?
The KV cache stores the attention state for every token in the conversation. It grows with context length, so a 32K context needs much more memory than 4K. Some models with sliding-window or linear attention layers need far less.
How is speed estimated?
Generating a token requires reading the whole model from memory, so speed is limited by memory bandwidth divided by model size. Real llama.cpp results usually reach 60 to 80 percent of that limit. Treat the number as a planning estimate.
Why is a Mixture-of-Experts model different?
MoE models keep all experts in memory but use only a few per token. They need the full weight memory, yet often generate faster than a dense model of the same size.
How the calculator works
Memory needed = model weights + KV cache + about 1 GB of runtime overhead. Weights equal the parameter count times the bits per weight of the chosen quantization. The KV cache is calculated from each model’s published architecture (layers, KV heads, head size) and your context length.
Where the data comes from
GPU memory size and bandwidth come from manufacturer and review-site specifications. Model architecture comes from each model’s published config on Hugging Face. Speed is an estimate based on memory bandwidth, not a benchmark of your exact setup.
Tips
- If a model is tight, lower the context length or use a Q8 KV cache.
- Q4_K_M is the usual best balance of size and quality.
- Running out of VRAM does not stop a model from working, but layers spill into system RAM and speed drops sharply.