How to Choose a GPU for Running LLMs

Running a large language model on your own hardware comes down to one question first: will the model fit on the GPU? Speed, power and software matter too, but memory decides whether a model runs at all. This guide walks through what to check, in the order it usually matters.

1. GPU memory decides what fits

A model's weights have to sit in GPU memory (VRAM) while it runs. If they do not fit, the model has to be split across several GPUs or partly moved to slower system memory, and both cost speed.

You can estimate the memory the weights need with simple arithmetic:

memory for weights ≈ number of parameters × bytes per parameter

The bytes per parameter depend on the precision the model is stored in:

Lower precision (quantization) makes a model smaller, usually at some cost in output quality.

2. Leave room above the weights

The weights are only part of the story. While a model generates text it also needs memory for its context — the conversation or document it is working with — and for the software running it. That extra memory grows with longer context and with more users served at once, so choose a GPU with room to spare above the weights estimate.

3. Memory bandwidth shapes speed

When a model generates text one token at a time, each new token needs the weights read from memory. That makes memory bandwidth — how fast the GPU can read its memory — one of the main limits on how quickly answers appear. When you compare two cards with enough memory, compare their bandwidth next.

4. Check the number formats a card supports

GPU generations differ in which number formats they run natively, such as 16-bit or 8-bit formats. A model in a format the card does not support well may run slowly or need converting first. Check the formats your model uses against the card before you buy.

5. Software support

Most LLM tools are built first for NVIDIA's CUDA, with growing support for AMD's ROCm. Older cards can drop out of support in newer drivers and libraries, so check that the tools you plan to use still support the card.

6. Power, cooling and form factor

Data-center cards are built for servers. Some are passively cooled and rely on the server's fans to push air through them, and many draw more power than a typical desktop supplies. Before putting one in a workstation, check its power connectors, its cooling and whether it fits the case.

Where to start

Browse the GPU catalog for each card's memory size and full specs, and use Compare GPUs to put cards side by side. Every spec on the site shows where it came from and when it was checked.