Local AI tools

Inference engines, desktop apps, and OpenAI-compatible servers for running open models on your own hardware.

Stacks Grid Table

All desktop app download tool inference engine inference server web UI

inference engine

Ollama

One-command local model runner with an OpenAI-compatible API and automatic GPU offload.

Best for: Getting a model running in under five minutes, and serving it to other apps on localhost.

binary · MIT · ★ 183,582

GGUF

inference engine

llama.cpp

The C/C++ engine that made CPU and mixed CPU/GPU inference practical, and the origin of the GGUF format.

Best for: Squeezing a model onto limited hardware, and controlling exactly how many layers live on the GPU.

binary · MIT · ★ 131,476

GGUF

inference server

vLLM

High-throughput serving engine with PagedAttention and continuous batching, built for concurrent requests.

Best for: Serving one model to many simultaneous users or agents, where tokens per second across the fleet matters more than single-prompt latency.

pip · Apache-2.0 · ★ 93,984

safetensors · GGUF · AWQ · GPTQ

desktop app

LM Studio

Desktop app for discovering, downloading, and chatting with local models, with a built-in OpenAI-compatible server.

Best for: People who want a graphical interface, model search, and a local API without touching a terminal.

binary · MIT · ★ 5,354

GGUF · MLX

inference engine

KoboldCpp

Single-executable llama.cpp fork with a built-in web UI, aimed at storytelling and long-form generation.

Best for: Running a model from one downloaded file with no install step, especially on Windows.

binary · AGPL-3.0 · ★ 12,016

GGUF

web UI

text-generation-webui

Gradio web interface that can drive several inference backends, with LoRA training and extensions built in.

Best for: Experimenting across backends and quantization formats without maintaining separate installs for each.

manual · AGPL-3.0 · ★ 47,733

GGUF · safetensors · EXL2 · GPTQ

web UI

Open WebUI

Self-hosted, multi-user chat interface for Ollama and any OpenAI-compatible backend, with RAG and document upload.

Best for: Giving a household or a team a polished chat interface over a local model, with accounts and shared history.

docker · NOASSERTION · ★ 155,954

GGUF

download tool

Hugging Face Hub

Official client for downloading model weights, with resumable transfers and selective file patterns.

Best for: Pulling exactly the files you need from a repository instead of cloning tens of gigabytes you will not use.

pip · Apache-2.0 · ★ 3,988

safetensors · GGUF