# Best GPU for Ollama in 2026 Choosing a GPU for Ollama: supported backends by OS, how to check GPU placement, settings that change VRAM use, and cards that fit common models. - URL: https://homelabpicks.com/gpu-ai/best-gpu-ollama/ - Published: 2026-03-15 - Updated: 2026-09-30 - Evidence: research-backed > HomeLabPicks earns a commission on qualifying purchases made through https://homelabpicks.com/go/ links. Prices change; check the retailer before buying. Disclosure: https://homelabpicks.com/disclosure/ ## Verdict **[Used NVIDIA RTX 3090](https://homelabpicks.com/go/rtx-3090-used)** (Used-market option): The 24GB CUDA pick for Ollama: it keeps 32B models at Q4_K_M entirely on the GPU, provided used-card power, cooling, condition, and return risk are acceptable. ## Products ### Our Pick: [Used NVIDIA RTX 3090](https://homelabpicks.com/go/rtx-3090-used) (Used-market option) The RTX 3090 remains useful because 24GB changes which models and contexts fit on the GPU. It is not a low-risk default: inspect condition, cooling, connector history, and return terms. - VRAM: 24GB GDDR6X - Ollama backend: CUDA - Board power: 350W reference rating - Purchase: Used-only recommendation Pros: - 24GB keeps 32B models at Q4_K_M entirely on the GPU - Mature CUDA path in Ollama - Room for a larger context or parallel requests on smaller models Cons: - High power and cooling requirement - Used condition and memory thermals vary - Large card and PSU requirements raise complete-system cost ### Best New Value: [NVIDIA RTX 5060 Ti 16GB](https://homelabpicks.com/go/rtx-5060-ti-16gb) The default new-card choice when 16GB fits the target model and context. CUDA support simplifies setup, and buying new avoids used-card condition risks. - VRAM: 16GB GDDR7 - Ollama backend: CUDA - Board power: 180W reference rating - Purchase: Current new product Pros: - 16GB current-generation CUDA option - Lower board-power class than the RTX 3090 - New-product warranty path Cons: - The 16GB ceiling matters more than peak speed for larger models - Partner-card size and power connectors vary - Speed figures do not transfer across different Ollama builds and quantizations ### AMD Option: [AMD Radeon RX 9070 XT](https://homelabpicks.com/go/rx-9070-xt) A viable choice after validating the exact operating system, Ollama backend, and model path. Hardware value cannot compensate for an unsupported software configuration. - VRAM: 16GB GDDR6 - Ollama backend: ROCm on Linux, or Vulkan - Architecture: RDNA 4 - Purchase: Current new product Pros: - Current 16GB AMD option - Listed in Ollama's supported Radeon cards for Linux - Strong fit when the rest of the workload already favors AMD Cons: - CUDA-focused tooling and guides remain more common - OS and backend constraints require pre-purchase validation - Troubleshooting cost can outweigh hardware savings ## Comparison | Product | VRAM | Ollama backend | Board power | Fits fully on the GPU | Purchase type | | --- | --- | --- | --- | --- | --- | | [Used RTX 3090](https://homelabpicks.com/go/rtx-3090-used) (winner) | 24GB GDDR6X | CUDA | 350W | 32B at Q4_K_M | Used | | [RTX 5060 Ti 16GB](https://homelabpicks.com/go/rtx-5060-ti-16gb) | 16GB GDDR7 | CUDA | 180W | 14B at up to Q6_K | New | | [RTX 5070 Ti 16GB](https://homelabpicks.com/go/rtx-5070-ti-16gb) | 16GB GDDR7 | CUDA | 300W | 14B at up to Q6_K, with more bandwidth | New | | [Radeon RX 9070 XT](https://homelabpicks.com/go/rx-9070-xt) | 16GB GDDR6 | ROCm on Linux, or Vulkan | 304W | 14B at up to Q6_K | New | | [Intel Arc B580](https://homelabpicks.com/go/intel-arc-b580) | 12GB GDDR6 | Vulkan | 190W | 14B at Q4_K_M | New | Ollama makes local models easy to launch, but GPU shopping still begins with two questions: does the model fit, and does Ollama support the card on your operating system? This guide covers Ollama's supported backends, how to confirm a model is running on the GPU, the settings that change memory use, and which cards fit common model sizes. The [used RTX 3090](https://homelabpicks.com/go/rtx-3090-used) is the value recommendation when 24GB matters. The [RTX 5060 Ti 16GB](https://homelabpicks.com/go/rtx-5060-ti-16gb) is our recommendation for a new card when 16GB fits. Those are different risk and capacity choices, not merely a speed ranking. ## Which GPUs Ollama supports Ollama's GPU documentation lists four hardware paths: - **NVIDIA:** cards with compute capability 5.0 or newer and driver 550 or newer. Older cards in the 5.x and 6.x range need driver 570 or newer. - **AMD:** Radeon cards through ROCm v7 on Linux, including the RX 9070 XT, RX 9070, RX 9060 XT, and RX 7900 XTX. The Windows list is narrower. - **Apple:** Apple silicon through Metal. - **Vulkan:** Intel and AMD GPUs on Windows and Linux. Set `OLLAMA_VULKAN=0` to turn it off. To limit Ollama to specific cards, set `CUDA_VISIBLE_DEVICES` (NVIDIA), `ROCR_VISIBLE_DEVICES` (AMD), or `GGML_VK_VISIBLE_DEVICES` (Vulkan). ## Check where the model runs Load a representative model and prompt, then run `ollama ps`. The processor column reads "100% GPU" when the whole model is on the card. A split such as "48%/52% CPU/GPU" means part of the model is running from system RAM. Partial offload works, but it can slow generation enough that an interactive assistant turns into a batch job. ## Settings that change VRAM use The same model can fit or spill depending on a few Ollama settings, all documented in its FAQ: - **Context length.** The default is 4,096 tokens. Raise it with `OLLAMA_CONTEXT_LENGTH` or the `num_ctx` parameter, and the KV cache grows in proportion. For Qwen3-32B, 32K tokens of 16-bit cache takes about 8.6GB on top of the weights. - **KV cache type.** With `OLLAMA_FLASH_ATTENTION` enabled, `OLLAMA_KV_CACHE_TYPE=q8_0` stores the cache in about half the memory of the default, and `q4_0` in about a quarter. The setting applies to every model the server runs. - **Parallel requests.** `OLLAMA_NUM_PARALLEL` (default 1) sets how many requests a model serves at once. Memory scales with parallel requests multiplied by context length, so a household server needs more headroom than a single-user desktop. - **Loaded models.** `OLLAMA_MAX_LOADED_MODELS` limits how many models stay in memory together. Several resident models compete for the same VRAM. The [VRAM planning guide](https://homelabpicks.com/gpu-ai/how-much-vram-for-llms/) has weight sizes and KV-cache tables for sizing a card before you buy. ## Multiple GPUs If a model fits on one GPU, Ollama loads it there. If it doesn't, Ollama spreads it across all available GPUs. Two cards can therefore hold a model that neither holds alone, but a single card with enough VRAM is simpler to power, cool, and troubleshoot. ## CUDA simplifies software setup NVIDIA remains the easiest recommendation because CUDA support is broadly documented across local-AI tools. The RTX 5060 Ti 16GB is the current new entry point. The RTX 5070 Ti does not increase VRAM, so buy it only when more throughput inside the same 16GB is worth the extra system cost. The RTX 3090 gives up efficiency and warranty for 24GB, which keeps 32B models at Q4_K_M entirely on the GPU. Check the PSU, physical clearance, airflow, connector condition, memory temperatures, and seller return policy before treating the card price as the complete cost. ## AMD and Intel require stack validation The [Radeon RX 9070 XT](https://homelabpicks.com/go/rx-9070-xt) is on Ollama's supported list for Linux through ROCm, and it can also run through Vulkan. Verify your exact operating system and Ollama version before buying. The [Intel Arc B580](https://homelabpicks.com/go/intel-arc-b580) runs through Ollama's Vulkan backend. With 12GB it suits 8B and 14B models. Choose it if you are willing to configure the Vulkan backend before using the server. ## Bottom line Choose capacity first, software support second, and speed third. Buy the [used RTX 3090](https://homelabpicks.com/go/rtx-3090-used) for a risk-tolerant 24GB CUDA build. Buy the [RTX 5060 Ti 16GB](https://homelabpicks.com/go/rtx-5060-ti-16gb) for a current, lower-power, warrantied CUDA system. Buy AMD or Intel only after the exact Ollama backend for your card and operating system is confirmed in Ollama's documentation. ## FAQ ### How much VRAM does Ollama need? Enough for the model's weights, its KV cache at the context length you set, and some runtime headroom. Ollama's default context is 4,096 tokens, and raising it or allowing parallel requests increases memory use. As a guide, 16GB holds 14B models at up to Q6_K, and 24GB holds 32B models at Q4_K_M. ### How do I check whether Ollama is using my GPU? Run ollama ps after loading a model. The processor column reads 100% GPU when the whole model is on the card. A split such as 48%/52% CPU/GPU means part of the model is running from system RAM, which is much slower. ### Does Ollama require NVIDIA? No. Ollama's GPU documentation lists NVIDIA cards with compute capability 5.0 or newer, AMD Radeon cards through ROCm on Linux (with a narrower Windows list), Apple silicon through Metal, and Intel and AMD GPUs through its Vulkan backend. Check the current list for your exact card and operating system before buying. ### Can Ollama use two GPUs? Yes. If a model fits on one GPU, Ollama loads it there. If not, it spreads the model across all available GPUs. You can limit which GPUs it uses with CUDA_VISIBLE_DEVICES on NVIDIA or ROCR_VISIBLE_DEVICES on AMD. ### Why are there no tokens-per-second rankings here? Generation rate changes with model, quantization, context, prompt-processing state, backend, driver, Ollama version, and measurement method. Without a reproducible test record, a single number is misleading. ## Sources - [Ollama GPU support documentation](https://docs.ollama.com/gpu) (accessed 2026-09-30) - [Ollama FAQ (GPU placement, context length, KV cache, concurrency, multiple GPUs)](https://docs.ollama.com/faq) (accessed 2026-09-30) - [NVIDIA GeForce RTX 5060 family specifications](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5060-family/) (accessed 2026-09-29) - [NVIDIA GeForce RTX 5070 family specifications](https://www.nvidia.com/en-us/geforce/graphics-cards/50-series/rtx-5070-family/) (accessed 2026-09-29) - [NVIDIA GeForce RTX 3090 specifications](https://www.nvidia.com/en-us/geforce/graphics-cards/30-series/rtx-3090-3090ti/) (accessed 2026-09-29) - [AMD Radeon RX 9070 series launch announcement (specifications)](https://ir.amd.com/news-events/press-releases/detail/1238/amd-unveils-next-generation-amd-rdna-4-architecture-with-the-launch-of-amd-radeon-rx-9000-series-graphics-cards) (accessed 2026-09-30) - [Intel Arc B580 specifications](https://www.intel.com/content/www/us/en/products/sku/241598/intel-arc-b580-graphics/specifications.html) (accessed 2026-09-30) - [llama.cpp quantize tool README (bits per weight)](https://github.com/ggml-org/llama.cpp/blob/master/tools/quantize/README.md) (accessed 2026-09-30) - [Qwen3-32B model configuration (layer and KV-head counts)](https://huggingface.co/Qwen/Qwen3-32B/raw/main/config.json) (accessed 2026-09-30)