Local AI tools

Inference engines, desktop apps, and OpenAI-compatible servers for running open models on your own hardware.

vLLM

High-throughput serving engine with PagedAttention and continuous batching, built for concurrent requests.

Best for: Serving one model to many simultaneous users or agents, where tokens per second across the fleet matters more than single-prompt latency.

pip · Apache-2.0 · ★ 93,987

safetensors · GGUF · AWQ · GPTQ