llama.cpp

llama.cpp is a C/C++ inference engine built for single-user local model execution without Python or PyTorch runtime dependencies. Developed originally by Georgi Gerganov, it serves as the reference implementation for the GGUF model format. By eliminating virtual environment overhead, it lets operators run quantized language models directly on native hardware compute interfaces.

The tool gives operators low-level control over memory mapping, layer offloading, and physical batching. Rather than masking execution parameters behind high-level GUIs, it exposes explicit knobs to balance weight distribution between system RAM and dedicated VRAM.

What llama.cpp actually does

llama.cpp executes quantized language models using native C and C++ code via the ggml backend. It natively parses GGUF files, supporting integer and floating-point quantization schemes like INT4, INT8, and FP16. This architecture skips Python interpreter latency and PyTorch GPU memory overhead, running matrix multiplications directly on target hardware accelerators.

For terminal workflows, the project provides dedicated command-line utilities. You can launch interactive text sessions using llama-cli:

llama-cli -m ./model.Q4_K_M.gguf -ngl 99 -c 4096

To expose model endpoints to external applications or scripts, llama-server runs a lightweight HTTP service:

llama-server -m ./model.Q4_K_M.gguf -ngl 99 -c 8192 --port 8080

This binary serves OpenAI-compatible routes, including /v1/chat/completions, /v1/completions, and /v1/embeddings.

Installing it on Windows, macOS, and Linux

On macOS, installation works via Homebrew, which pulls pre-built binaries configured for Apple Silicon Metal acceleration:

brew install llama.cpp

To build for NVIDIA hardware, compile from source using CMake with CUDA support enabled:

cmake -B build -DGGML_CUDA=ON && cmake --build build --config Release -j

This command generates binaries compiled for your installed CUDA toolkit.

For non-NVIDIA GPUs or AMD cards without ROCm support, compiling with Vulkan support via -DGGML_VULKAN=ON provides broad hardware coverage. Ensure system Vulkan development headers are installed so CMake links the loader library correctly.

When compiling on memory-constrained machines, parallel build jobs can exhaust host RAM and crash the compiler. Limit parallel jobs with -j2 or expand swap space to stabilize compilation.

Building on Windows can trigger BLAS macro naming collisions depending on the MSVC environment. On Linux with AMD hardware, compilation failures usually stem from missing HIP development headers in ROCm package paths. Check the llama.cpp GitHub repository for platform-specific flags.

Hardware and VRAM behaviour

The engine includes compute backends for CUDA, ROCm, Metal, Vulkan, SYCL, and plain CPU execution.

Weight offloading uses the -ngl flag, which specifies how many transformer layers shift to GPU VRAM while leaving the rest in host RAM. The -ngl flag is the most useful knob when VRAM is tight.

Memory allocation scales with context window length. You can reduce KV cache memory overhead using Flash Attention with the -fa flag. Quantizing the KV cache with -ctk q8_0 and -ctv q8_0 cuts context memory footprint further without dropping conversation context.

For multi-GPU setups, llama.cpp splits memory allocation using the --tensor-split flag. Supplying comma-separated ratios balances tensor splits across cards with mismatched VRAM capacities.

Certain backends have memory accounting edge cases. Vulkan buffers can under-report free memory under heavy load, while SYCL backends on integrated Intel APUs can desynchronize memory tracking during continuous context evaluation.

Configuration that matters (the 3-5 settings people get wrong)

Understanding -ngl / --n-gpu-layers for layer offloading

Passing an integer to -ngl sets how many transformer layers offload to VRAM. Passing -ngl 99 offloads all model layers to the GPU. If VRAM is full, reduce the layer count to split execution between host RAM and VRAM.

Logical batch (-b) versus physical batch (-ub)

The -b parameter sets the logical batch size for prompt evaluation, while -ub controls the physical micro-batch size processed by the GPU in a single pass. If prompt processing causes GPU allocation spikes or driver crashes, lower -ub to keep peak VRAM usage under control.

Thread allocation (-t) and physical CPU core pinning

By default, CPU inference spawns threads matching logical core counts. Setting -t equal to physical CPU cores prevents cache thrashing. On asymmetric CPUs with mixed core types, restrict threads to performance cores to prevent thread migration stalls.

Multi-GPU orchestration with --tensor-split and --split-mode

When distributing a GGUF model across non-identical GPUs, pass proportional ratios to --tensor-split. Pair this setting with --split-mode layer to configure pipeline parallelism across separate GPU buses.

Server execution parameters (-c, --port)

The -c flag sets the total context window size in tokens, like -c 8192. Memory allocation scales with context size, so setting an oversized context reserves VRAM immediately on server startup. Use --port 8080 to bind llama-server to your port.

Common errors and how to fix them

PCIe bus saturation during partial offload

When running models with intermediate -ngl values, token generation slows because layer tensors cross the PCIe bus on every forward pass. Offloading all layers to VRAM eliminates host-to-device transfer overhead.

llama-server memory leaks on repeated requests

Long-running llama-server instances can leak memory over repeated incoming requests. Over extended inference runs, this accumulation leads to host OOM crashes. Fix this in production by running the binary under a systemd daemon set to restart periodically, or by setting up container recycling.

Antivirus false positives on Windows

Pre-compiled release binaries on Windows can trigger false-positive alerts in security software due to direct memory allocation calls. Resolve this by adding a folder exclusion rule or compiling locally from source using Visual Studio Build Tools.

Compilation OOMs and missing library headers

Compiler crashes during compilation usually stem from RAM exhaustion caused by multi-threaded build jobs. Limit build threads by adding -j2 or -j1 to the build command. On ROCm or SYCL setups, verify driver headers like hip/hip_runtime.h are present in your environment PATH before running CMake.

How it compares with the alternatives

Compared to high-throughput engines like vLLM, llama.cpp focuses on single-user latency and CPU/GPU hybrid execution rather than high-concurrency PagedAttention batching.

Against packaged options like Ollama or desktop tools like LM Studio, llama.cpp gives direct access to execution flags without abstraction layers or background daemon overhead.

For specialized self-hosting, tools like KoboldCpp build on llama.cpp to add storytelling APIs, while web interfaces like text-generation-webui and Open WebUI use llama.cpp or its server binary as an underlying inference backend. Raw weights can be downloaded from Hugging Face Hub before quantizing or running them locally.

Who should use it (and who should not)

llama.cpp is best for operators running local models on consumer GPUs, Apple Silicon, or mixed hardware. It is the right choice when you need direct control over physical micro-batching (-ub), context limits (-c), and layer offloading (-ngl).

It is not built for multi-tenant API serving where high parallel concurrency is required; engines like vLLM provide better throughput for concurrent loads. Users who want a single-click installer without command-line configuration should use a desktop app instead.

The short version

llama.cpp is the core C/C++ engine for bare-metal GGUF model execution across varied hardware. Tuning flags like -ngl for GPU offloading, -ub for micro-batch sizes, and --tensor-split for multi-GPU balancing enables efficient hardware utilization. Refer to the llama.cpp documentation for complete endpoint and flag details.