Real-world performance
LLM Benchmarks
Compare real-world LLM performance across available GPUs and popular model families.
| NVIDIA GeForce RTX 3090 24GB | Unknown | Llama 2 7B (Llama 2) | 4-bit (Q4_0) | 158.2 | — | — | |
| NVIDIA Tesla P40 24GB | 24 GB | Llama 2 7B (Llama 2) | 4-bit (Q4_0) | 54.7 | — | — | |
| NVIDIA Tesla V100 PCIe 32GB | 32 GB | Not measured yet | — | — | — | — | |
| NVIDIA Tesla V100 SXM2 32GB | 32 GB | Not measured yet | — | — | — | — | |
| NVIDIA Tesla V100 16GB | 16 GB | Not measured yet | — | — | — | — |
Benchmark Methodology
Tokens per second is generation throughput measured on real hardware. Each result lists its test setup, source and measurement date; expand a row to see them. Results vary with model version, quantization method, context length and system configuration.
Need help choosing the right GPU?
Get expert advice on the right GPU for your LLM workload and budget.