How to run locally
OTel-2.0-LLM-31B-IT
farbodtavakkoli · 31B · install path for the weights on your own GPU
Quick start
Hugging Face
huggingface-cli download farbodtavakkoli/OTel-2.0-LLM-31B-IT
vLLM
vllm serve farbodtavakkoli/OTel-2.0-LLM-31B-IT
Prefer the full runbook below if this is your first local load. Specs and VRAM fit live on the listing.
How to run OTel-2.0-LLM-31B-IT locally starts with a VRAM check, then a runner choice, then one clean load at Q4. This runbook is the install path — the listing page covers whether the weights belong on your card.
Check VRAM before you fetch OTel-2.0-LLM-31B-IT
Confirm the VRAM chart on the OTel-2.0-LLM-31B-IT listing first. If Q4 at 4k context does not fit your card, stop here and pick a smaller sibling instead of fighting a load that will spill into system RAM.
Weights live at farbodtavakkoli/OTel-2.0-LLM-31B-IT on Hugging Face. Prefer an official or well-known GGUF. Unofficial dumps often ship the wrong chat template.
Choose a runner for OTel-2.0-LLM-31B-IT
Ollama is the shortest path when you want a local chat UI quickly. Install it, confirm it sees the GPU, then point it at the local blob or a matching library tag.
llama.cpp gives you flags for layers, context, and sampler. Build or install a CUDA, Metal, or Vulkan binary, then run with -m at the GGUF and -c at a context you can hold.
vLLM is for an OpenAI-compatible HTTP API on a GPU server. Use it when you already serve other models that way, not for a first desktop test.
First load of OTel-2.0-LLM-31B-IT that actually fits
Start at Q4_K_M and 4k context. Raise context only after the card stops swapping. Drop quant if the OS starts using shared GPU memory — that is a failed fit even if the process stays alive.
Windows: prefer CUDA or Vulkan and close overlays that grab the GPU. macOS: use Metal and remember unified memory still has a ceiling. Linux with NVIDIA: confirm nvidia-smi sees the card before blaming the weights.
Lock the OTel-2.0-LLM-31B-IT setup once it answers
Write down quant, context, tokens per second, and the GPU used. That note beats a generic “it works” comment when you change drivers later.
Lock a short system prompt and a sampler preset so later A/B tests stay comparable. Re-run the same prompt after every runner or driver update.
Do not expose an unauthenticated local server to the public internet. Bind to localhost or a trusted LAN and add auth if anyone else can reach the port.