What can I run?

When operators ask what can I run on local hardware, the answer depends almost entirely on physical memory limits rather than gaming frame rates or GPU clock speeds. Open-weight language models run inside dedicated video RAM. If your card lacks sufficient space, execution fails instantly with CUDA memory allocation faults or collapses into severe latency from CPU offloading.

Determining true model compatibility requires balancing static weights, quantization schemes, expanding context buffers, and system background processes. Evaluating hardware limits before downloading large weight files prevents wasted bandwidth and repeated runtime crashes.

Disambiguating Local LLM Memory Sizing from Gaming Benchmarks

Search results for local software requirements are often cluttered with PC gaming benchmark numbers and rasterization frame rates. Modern games rely heavily on shader pipelines, ray tracing cores, and real-time floating-point operations.

Local language models operate differently. Text generation is primarily bound by memory bandwidth and total VRAM capacity. Every token generated requires the GPU to stream entire weight matrices through memory, making raw capacity the dominant bottleneck.

High compute throughput cannot compensate for a hard out-of-memory fault. If model parameters and runtime buffers exceed physical VRAM, execution halts immediately regardless of peak FLOPS ratings.

When a GPU runs out of VRAM, the driver may attempt unified memory paging, spilling active tensors over the PCIe bus into system RAM. System memory bandwidth is a fraction of GDDR6 or GDDR6X speeds, causing generation speed to drop from dozens of tokens per second to a crawl.

Furthermore, core clock speeds and tensor processing units remain idle while waiting for weight parameters to arrive over the system bus. Hardware profiling must prioritize bus bandwidth and memory capacity over theoretical compute throughput.

How the Engine Computes Model Fit and Memory Allocation

Evaluating whether an open-weight model fits a GPU requires calculating both static and dynamic allocations. Model parameter weights scale directly with precision, such as uncompressed FP16 or quantized formats like Q4_K_M and standard 4-bit quantization.

As text generation progresses, the engine accounts for KV cache overhead. Long context windows dynamically expand, consuming additional megabytes for every active token in the prompt buffer.

To ensure operational stability, the engine applies a standard 15% VRAM overhead buffer. This reserves space for CUDA VRAM allocation tasks, framework memory pools, and temporary context spikes during token generation.

The static footprint stems directly from array size and quantization density. Uncompressed 16-bit weights require two bytes per parameter, whereas 4-bit block quantization compresses weights down to approximately half a byte per parameter, plus quantization scale factors.

Dynamic allocation depends heavily on attention mechanisms and sequence length. Key-value tensors must store historical context across all attention heads for every layer. Without optimizations like FlashAttention or KV cache quantization, context memory can rival the baseline model weight size during deep prompts.

The 8 GB VRAM Limit: Desktop Overhead and Context Window Spikes

Operators using entry-level hardware like the RTX 4060 8GB (USD 300.00) frequently encounter early runtime failures. While a quantized model file might show a 6 GB footprint on disk, loading it onto an 8 GB graphics card leaves almost no margin.

Host operating systems, desktop display overhead, and background applications regularly consume between 1 GB and 2 GB of VRAM before any model loads. This baseline usage significantly reduces actual free memory.

When context windows expand during generation, memory requirements quickly exceed remaining capacity. This triggers an immediate OOM error or forces layers onto host system RAM.

Display server allocation varies by operating system and monitor setup. Multi-monitor desktop layouts running hardware-accelerated web browsers and system compositors routinely claim significant video memory. On an 8 GB card, losing system VRAM to desktop rendering leaves insufficient headroom for prompt processing.

When total allocation breaches physical hardware boundaries, CUDA memory allocators throw fatal runtime exceptions. Even if the inference framework attempts automatic layer offloading, traversing limited PCIe bus lanes introduces high latency that halts practical generation.

Evaluating VRAM Price Tiers and Capacity Tradeoffs

Selecting hardware for local execution requires weighing price against total memory buffer. The RTX 3060 12GB (USD 280.00) offers extra memory capacity compared to smaller 8 GB options, making it a common budget selection.

For dedicated server setups, the Used Tesla T4 16GB (USD 200.00) provides 16 GB of memory at a low price point, though it lacks active cooling and requires specialized chassis airflow.

Mid-to-high tier hardware like the RTX 4070 Ti Super 16GB (USD 800.00) balances high memory bandwidth with broad local compatibility across medium-sized models.

At the high end, the RTX 4090 24GB (USD 1800.00) provides 24 GB of fast memory, allowing higher precision quants and extended context windows. For deeper hardware reviews, consult our GPU buying guide.

Deploying enterprise server cards in standard desktop cases presents unique thermal and mechanical challenges. Passively cooled accelerators require high-static-pressure cooling shrouds, custom fan mounts, and specialized power adapters to prevent immediate thermal throttling under continuous inference loads.

Upgrading memory bus capacity enhances tokens-per-second generation performance. Higher memory bandwidth allows the execution engine to cycle parameters through compute cores faster, directly reducing latency during long output generation runs.

Fallback Options When Your Desired Model Exceeds Available VRAM

When a targeted open-weight model fails to fit within available VRAM, operators have three primary recovery options. The simplest adjustment is lowering model precision, such as stepping down from 8-bit to 4-bit quantization formats.

Another effective tactic is truncating the maximum context length within your inference runner. Restricting context caps KV cache expansion, keeping total allocation within hardware limits.

If memory capacity remains insufficient, offloading a portion of model layers to CPU RAM permits execution at reduced generation speeds. If physical hardware upgrades seem necessary, check our cost vs cloud calculator to compare local hardware investment against hosted API rates.

Partial layer offloading divides model execution across hardware boundaries. The inference runner loads critical initial layers into fast VRAM and assigns remaining layers to system memory. While this allows execution of oversized models, passing intermediate activations across PCIe lanes creates bottlenecking.

Quantization level selection involves explicit quality tradeoffs. Moving from 8-bit to 4-bit quantization cuts parameter memory requirements significantly while preserving overall output coherence. However, aggressive 2-bit quantization can introduce severe perplexity degradation and loss of reasoning capability.

Transitioning from Model Discovery to Detailed VRAM Profiling

The 'What Can I Run?' matching engine acts as an initial filter to surface compatible open-weight models based on estimated VRAM capacity. It helps narrow down candidates quickly without manual math.

However, complex deployments require exact resource profiling. When tuning precise prompt contexts, testing custom batch sizes, or modifying KV cache precision, switch to our detailed VRAM calculator.

Detailed profiling reveals exact megabyte allocations per layer, allowing operators to maximize context allocations right up to physical hardware limits.

Optimizing KV cache storage can free significant VRAM during long conversations. Advanced inference engines allow quantization of key-value caches to 8-bit or 4-bit precision, cutting context memory usage without modifying underlying model weights.

Enabling FlashAttention mechanisms reorganizes attention computation to reduce intermediate activation buffers. This architectural optimization prevents memory spikes during long prompt ingest phases, ensuring stable allocation during complex retrieval tasks.

Who Should Use This Tool Versus Advanced Hardware Profilers

This discovery tool targets individual operators, developers, and hobbyists building single-GPU workstations or desktop inference rigs. It provides fast compatibility checks without requiring deep infrastructure configuration.

Enterprise system architects and production engineers managing multi-GPU clusters should use dedicated tensor profiling tools. High-throughput serving engines like vLLM require specialized math for tensor parallelism and continuous batching workloads.

For single-card desktop deployments, establishing basic memory boundaries remains the most effective starting point for stable local execution.

Multi-GPU production setups rely on specialized memory management techniques like PagedAttention. By dividing KV cache allocations into non-contiguous memory blocks, production serving frameworks eliminate fragmentation and maximize concurrent user request throughput across server nodes.

In contrast, local desktop inference prioritizes single-stream, low-latency execution. Matching static model parameters and context limits to available single-card video memory remains the primary operational task for individual workstation setups.