Ollama

Ollama is a background daemon built on top of llama.cpp. It provides an OpenAI-compatible HTTP endpoint on port 11434 alongside its own CLI.

The daemon handles model downloads and token streaming. It schedules model layers across GPU and CPU.

Hardware offloading to GPUs and system memory occurs automatically, without manual build steps or CUDA compilation flags.

The daemon exposes HTTP endpoints on localhost:11434. It provides native REST endpoints (/api/generate and /api/chat) and an OpenAI-compatible one (/v1/chat/completions).

Client SDKs and agent frameworks can route traffic to local models by changing the base URL.

It manages model distribution using a registry system. When a tag is requested, Ollama downloads the GGUF weights and saves them to an internal blob store, bundling prompt formatting and tokenization metadata into the tag.

Ollama runs headlessly without a bundled graphical user interface. It serves inference requests directly to terminal sessions or web frontends such as Open WebUI.

Supported platforms and installation

Ollama provides native binaries for Linux, macOS, and Windows. It uses underlying hardware acceleration where available, including on NVIDIA GPUs and Apple Silicon.

Linux and macOS binary installation

The installation script handles both Linux and macOS:

curl -fsSL https://ollama.com/install.sh | sh

On Linux, the installer sets up a systemd unit located at /etc/systemd/system/ollama.service. It starts the daemon immediately and configures it to bind to port 11434 on system boot.

On macOS, Ollama runs as an application bundle that registers the command-line binary in your path and adds a menu bar status controller.

Windows setup and Docker deployment

Windows users run a standalone installer that registers an auto-starting user service. The Windows binary automatically detects supported NVIDIA GPUs via installed display drivers.

For containerized environments, use the official Docker image:

docker run -d --gpus=all -v ollama:/root/.ollama -p 11434:11434 ollama/ollama

Once installed, test model execution directly from your terminal:

ollama run llama3.1:8b

Hardware and VRAM behaviour

Ollama offloads as many layers to the GPU as VRAM allows and quietly falls back to CPU for the rest, so an oversized model gets slow rather than crashing.

Token throughput drops when the model is not fully resident in VRAM, as activation tensors must traverse the PCIe bus on every generation cycle.

Inspect active layer distribution and memory usage by querying the daemon:

ollama ps

The output details the active model name and its split across system RAM and GPU VRAM.

Ollama retains models in VRAM for a period after the last request by default. Adjust this using the keep_alive parameter.

A value of -1 keeps weights permanently pinned, while 0 unloads them immediately after inference.

Key configurations

Overriding default context window (num_ctx)

Many models default to a limited num_ctx. Prompts exceeding this are truncated silently unless explicitly configured, even when the base model supports larger contexts.

Override this by specifying num_ctx inside a custom Modelfile:

FROM llama3.1:8b
PARAMETER num_ctx 8192

Build the new model tag with ollama create llama3.1-8k -f Modelfile. Alternatively, pass the option in API payloads:

curl http://localhost:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Summarize this text...",
  "options": {
    "num_ctx": 8192
  }
}'

Expanding context increases the size of the key-value cache, requiring additional VRAM allocation alongside the model weights.

Network binding and remote access (OLLAMA_HOST)

By default, the Ollama daemon binds exclusively to loopback interface 127.0.0.1:11434, rejecting incoming network connections.

To bind all network interfaces for LAN or remote access, set the OLLAMA_HOST environment variable when starting the service:

OLLAMA_HOST=0.0.0.0:11434 ollama serve

On Linux systems running systemd, persist this setting by running sudo systemctl edit ollama.service and adding the override below:

[Service]
Environment="OLLAMA_HOST=0.0.0.0:11434"

Apply changes with sudo systemctl daemon-reload && sudo systemctl restart ollama.

Storage location and custom GGUF imports (OLLAMA_MODELS)

Ollama stores weights in ~/.ollama/models on Linux/macOS and %USERPROFILE%\.ollama\models on Windows. Redirect storage by setting the OLLAMA_MODELS environment variable.

For example, to persist this in systemd on Linux and prevent filling root disk space, use:

Environment="OLLAMA_MODELS=/mnt/fast_nvme/ollama_models"

If you already have local GGUF files, create a Modelfile containing:

FROM /absolute/path/to/custom_model.gguf

Run ollama create custom-local -f Modelfile to import the weights directly into the local store.

Comparing Ollama with Alternatives

Choosing an inference engine often depends on concurrency targets and runtime control. While Ollama simplifies setup, raw command-line builds of llama.cpp offer more direct control over lower-level execution parameters.

For high-concurrency server deployments, vLLM offers PagedAttention and continuous batching.

Ollama focuses instead on single-user workflows and local desktop integrations. Those seeking a desktop application interface can consider LM Studio.

The codebase is open source, maintained under the MIT license on the Ollama GitHub repository.

The short version

To run models locally with automatic GPU offloading, install the binary and launch a model:

# Install Linux / macOS
curl -fsSL https://ollama.com/install.sh | sh

# Run a model interactively
ollama run llama3.1:8b

# Serve on the network
OLLAMA_HOST=0.0.0.0:11434 ollama serve

For headless operation, explicitly set num_ctx in your Modelfile and monitor layer distribution using ollama ps.