GPU buying guide

Selecting the right GPU for local LLM deployments requires ignoring conventional gaming metrics. Synthetic rasterization scores and FP32 compute ratings do not dictate language model performance. Local inference operates as a capacity-bound workload during model initialization and shifts to a memory-bandwidth-bound workload during token generation.

To run open-weight models reliably, physical VRAM dictates the largest parameter class you can load into memory. Memory bandwidth determines how rapidly the card transfers weights into compute cores for every generated token.

How VRAM Requirements Are Calculated for Local Inference

Calculating the VRAM footprint starts with model parameter weights. At FP16 precision, every billion parameters requires roughly 2 GB of memory. Quantization reduces this footprint: 8-bit precision (Q8) halves base weight requirements to 1 GB per billion parameters, while 4-bit precision (such as Q4_K_M) drops the footprint to approximately 0.5 GB to 0.6 GB per billion parameters.

Base weights are only the baseline. As prompt context grows, the Key-Value (KV) cache consumes dynamic memory. An unquantized FP16 KV cache scales linearly with sequence length and batch size. While quantized KV caches reduce this burden, active context windows can demand gigabytes of extra allocation.

Attention architecture heavily influences KV cache sizing. Older architectures using Multi-Head Attention allocate dedicated key and value heads for every query head, causing severe cache growth over long sequences. Modern models implement Grouped-Query Attention (GQA), which shares key-value heads across multiple query heads to compress the dynamic cache footprint.

Runtime engines require workspace memory beyond raw weights. Our engine standardizes on a 15% runtime overhead buffer to account for CUDA execution contexts, attention scratchpads, and memory fragmentation. You can verify custom configurations using the precise VRAM requirements calculator.

VRAM Floors by Model Class: 7B, 32B, and 70B Parameter Workloads

Entry-level 7B parameter models running at Q4 quantization fit inside an 8 GB VRAM envelope for minimal sequence lengths. However, extending context beyond a few thousand tokens quickly exhausts 8 GB cards. A 12 GB pool provides safe headroom for 7B workloads running at Q8 or deep context windows.

Intermediate 32B parameter models occupy the modern mid-tier sweet spot. At Q4 quantization, 32B weights demand roughly 18 GB to 20 GB of memory before accounting for the KV cache and the 15% overhead. Sizing a 32B architecture requires 24 GB of addressable VRAM.

Large 70B architectures establish a strict 48 GB VRAM requirement for reliable local execution at Q4 quantization. A single consumer graphics card cannot host a 70B model at practical context lengths. Operators deploy dual 24 GB cards or high-capacity system memory to run this class.

Sparse Mixture of Experts (MoE) architectures introduce distinct sizing constraints. While an MoE model activates only a fraction of its total parameters per token during computation, all routing and expert weights must reside simultaneously in memory. You cannot size VRAM solely around active parameter counts.

Why Memory Bandwidth Governs Generation Speed, Not Core Counts

Auto-regressive text generation generates one token per step. For each generated token, the GPU must stream every single active model weight from VRAM into its compute units. Compute workloads per weight are small, which means the execution cores constantly stall while waiting for data transfer across the memory bus.

A wider memory bus width enables higher data transfer rates in gigabytes per second (GB/s). A GPU equipped with thousands of CUDA cores but constrained by a narrow memory bus will generate text slower than a card with fewer cores attached to a wider memory bus.

Prompt processing behaves differently from token generation. Ingesting prompt tokens during the prefill phase computes matrix products across all input tokens concurrently. This stage saturates compute units and benefits from raw tensor core throughput, while the subsequent token decoding phase remains strictly bandwidth-limited.

Evaluating Published Hardware: Budget Entries to Flagship Silicon

Budget builds often start with the RTX 3060 12GB at USD 280.00. Check the RTX 3060 12GB spec profile for full architectural metrics. Its 12.00GB pool comfortably hosts 7B models at Q8 or 13B models at Q4. In contrast, the RTX 4060 8GB at USD 300.00 provides only 8.00GB of VRAM, making it a restrictive choice despite newer architecture.

The mid-range tier features the RTX 4070 Ti Super 16GB priced at USD 800.00. Review the RTX 4070 Ti Super 16GB card details to see how its 16.00GB pool handles 14B and quantized 32B models.

For single-card consumer flagships, the RTX 4090 24GB at USD 1800.00 delivers 24.00GB of high-speed memory for 32B models. On the enterprise secondary market, the used Tesla T4 16GB at USD 200.00 delivers 16.00GB of VRAM at low upfront cost, trading bandwidth for capacity.

Operating system desktop environments consume addressable VRAM directly from the primary display card. Running multiple high-resolution displays or hardware-accelerated desktop compositors can reserve hundreds of megabytes before loading an inference runtime. Dedicated inference rigs frequently run headless operating systems to maximize available GPU allocation.

Used Datacenter Cards vs Consumer Hardware: Power, Thermals, and Software

Enterprise cards like the used Tesla T4 offer high memory density at low prices, but they present mechanical challenges. Datacenter cards use passive cooling intended for high-static-pressure server racks. Installing one in a desktop workstation requires 3D-printed fan shrouds and dedicated high-RPM blower fans.

Power delivery requires careful verification on repurposed enterprise boards. Datacenter cards often use standard EPS12V CPU power pinouts rather than standard PCIe auxiliary power cables. Motherboard firmware must also support Above 4G Decoding and Resizable BAR to map the full memory window into system space.

Consumer GeForce hardware includes active multi-fan coolers, but high-end cards generate substantial transient power spikes. Power supplies must handle rapid load swings. Additionally, ensure your motherboard provides adequate PCIe lane allocation (x16 or x8 electrical lanes) to prevent inter-device transfer bottlenecks.

Software integration is seamless on Nvidia hardware through CUDA. Open-weight runtime engines and FlashAttention target CUDA architectures natively, ensuring broad support across community tooling.

Common Hardware Mistakes: Box Labels, Missing Overhead, and Thermal Choke

Retail product names cause frequent purchasing errors. Graphics cards sharing similar names often feature different memory configurations. Always verify the exact retail box SKU to ensure you receive the intended memory pool, such as confirming a 12.00GB capacity rather than a cut-down 8.00GB variant.

Another common error is calculating capacity without accounting for context expansion. A model whose raw weights fit within 90% of available VRAM will crash with Out-Of-Memory errors once active conversations grow. Always budget a 15% runtime buffer.

Physical motherboard layout also trips up multi-GPU builders. Standard ATX motherboards place PCIe slots too close together for modern three-slot consumer coolers. Suffocating the intake fans of the primary card leads to severe thermal throttling during extended inference sessions.

Motherboard slot layout can also introduce hidden bandwidth penalties. Lower PCIe slots on consumer motherboards often route through the chipset rather than directly to CPU lanes, dropping link speeds to PCIe x4 or x1 electrical. Verify motherboard block diagrams before designing multi-GPU rigs.

Multi-GPU Topologies: llama.cpp Layer Splitting vs vLLM Tensor Parallelism

Distributing workloads across multiple GPUs depends heavily on the runtime architecture. The llama.cpp repository supports pipeline layer offloading. In this setup, model layers are split sequentially across distinct CUDA devices, allowing mismatched memory sizes like a 12.00GB card paired with a 16.00GB card.

Production serving platforms described in the vLLM documentation use Tensor Parallelism. This splits individual weight matrices across GPUs, requiring synchronous communication across PCIe buses. Mismatched cards or low PCIe bandwidth create latency bottlenecks under Tensor Parallelism.

Pipeline parallelism processes layers sequentially, meaning one GPU waits idle while another executes its assigned layer slice. Tensor Parallelism computes simultaneously across devices but demands frequent synchronization per layer. Without high-speed interconnects like NVLink, consumer systems incur PCIe transfer overhead during Tensor Parallel operations.

Navigating Sibling Tools and Hardware Profiles

Sizing hardware requires evaluating exact weight targets alongside your budget. Use the what hardware runs which model tool to cross-reference specific model architectures against target hardware pools. If you need precise memory values for non-standard quantizations, check the interactive widget on this page.