All Local AI
Pick VRAM for fit estimates
Models Tools Hardware Best Of Compare Calculators Stacks Guides Blog
inference engine AGPL-3.0 linux macos windows

KoboldCpp

Single-executable llama.cpp fork with a built-in web UI, aimed at storytelling and long-form generation.

GitHub stars

12,016

License

AGPL-3.0

Install

binary

Latest release

Oct 13, 2026

Source on GitHub Documentation

Best for

Running a model from one downloaded file with no install step, especially on Windows.

Model formats

  • GGUF

Hardware behaviour

Auto-detects a sensible layer offload and supports CUDA, ROCm, Vulkan, and CPU from the same binary, which makes it forgiving on older or mixed hardware.

Not sure what your card can hold? Run the fit check — the entry point in our listings is the RTX 4060 8GB at 8.00 GB.

Setup

Install KoboldCpp

Copy-paste starting points. Always read the project's own instructions before running anything with elevated permissions.

Windows

Windows

# Download koboldcpp.exe from the Releases page and run it

No installer, no dependencies.

Linux

KoboldCpp

curl -fLo koboldcpp https://github.com/LostRuins/koboldcpp/releases/latest/download/koboldcpp-linux-x64 && chmod +x koboldcpp

Launch with a model

KoboldCpp

./koboldcpp --model ./model.Q4_K_M.gguf --gpulayers 99 --contextsize 8192

Headless API

KoboldCpp

./koboldcpp --model ./model.gguf --port 5001 --skiplauncher

Quick start

Getting the first model running

Download one file, point it at a GGUF, and a browser tab opens with a working chat and story interface. There is no package manager, no Python environment, and no service to manage — which is exactly why it is a common recommendation for Windows users who just want to try a model.

It exposes both a KoboldAI-compatible API and an OpenAI-compatible endpoint, so existing front-ends can usually connect without modification.

FAQ

KoboldCpp questions people actually ask

Short answers to the problems that come up most often.

How is KoboldCpp different from llama.cpp?

It is a fork that bundles a web UI, a launcher, and multiple backends into one executable. The inference core is the same lineage, so GGUF compatibility and quantization behaviour match closely.

Does it work on AMD GPUs?

Yes. Builds support ROCm on Linux and Vulkan across platforms, which usually gets AMD and Intel cards working without a CUDA toolchain.

What is context shifting?

When a conversation exceeds the context window, KoboldCpp can drop the oldest tokens and reuse the remaining cache instead of reprocessing the whole prompt. It keeps long chats responsive at the cost of losing the earliest turns.

Can I use it as a backend for another front-end?

Yes. Run it with --skiplauncher and connect over the API. It speaks both the KoboldAI and OpenAI-compatible formats.

Alternatives

Compared with other inference engine options

Same job, different tradeoffs.

inference engine

Ollama

One-command local model runner with an OpenAI-compatible API and automatic GPU offload.

 inference engine

llama.cpp

The C/C++ engine that made CPU and mixed CPU/GPU inference practical, and the origin of the GGUF format.

 inference server

vLLM

High-throughput serving engine with PagedAttention and continuous batching, built for concurrent requests.

 desktop app

LM Studio

Desktop app for discovering, downloading, and chatting with local models, with a built-in OpenAI-compatible server.

 web UI

text-generation-webui

Gradio web interface that can drive several inference backends, with LoRA training and extensions built in.

Next step

A runner is only half the decision. Check which models fit your GPU with the VRAM calculator, then follow a setup guide or pick a ready-made stack.