Intern-S2

How to run Intern-S2 locally on your own hardware requires picking the right execution framework, validating VRAM allocations, and launching the server with explicit parsing flags.

Check Hardware VRAM Limits Before Downloading Weights

Before pulling model weights from the InternLM Intern-S2 Hugging Face Collection, assess your hardware memory footprint. The flagship model features over 403 billion parameters (403,423,094,768). Running FP16 demands 901.721 GB of VRAM, FP8 takes 491.25 GB, and Q4_K_M requires 286.015 GB.

If you are deploying on a single desktop or dual-GPU workstation, target the 35B Preview or Mobius-v0 variants instead. Never attempt to load the 403B flagship on consumer rigs without multi-node cluster configurations, as execution will immediately fail due to out-of-memory errors.

Select LMDeploy or SGLang as Your MoE Inference Runner

Standard GGUF-based runners like desktop Ollama builds are not fully confirmed for this Mixture of Experts architecture. Instead, use LMDeploy or SGLang to serve the weights locally. These engines provide efficient FP8 and INT4 VRAM footprint management alongside robust multi-GPU tensor parallelism.

Install SGLang inside a clean PyTorch environment with CUDA support:

pip install "sglang[all]" --find-links https://flashinfer.ai/huggingface/torch/

Ensure your CUDA runtime and NVIDIA drivers match current FlashAttention requirements before starting the installation.

Execute the First SGLang Launch Command with Explicit Parsers

To launch the server without losing function calls or agent execution steps, you must pass the Qwen3 reasoning and tool parsers. Run the launch command directly in your terminal, targeting your downloaded local model weights or the Hugging Face repo ID:

python3 -m sglang.launch_server \
  --model-path internlm/Intern-S2 \
  --tp 4 \
  --reasoning-parser qwen3 \
  --tool-call-parser qwen3_coder \
  --host 0.0.0.0 \
  --port 30000

Adjust the --tp (tensor parallel) flag to match the exact number of GPUs installed in your machine.

Lock Sampling Default Parameters in API Payloads

Once the OpenAI-compatible server is active on port 30000, ensure all incoming API requests explicitly enforce the recommended sampling parameters. Leaving parameters unconstrained can lead to incoherent step-by-step reasoning or loop failures during long output generations.

Set your client payload parameters to stick strictly to the recommended baseline values:

{
  "model": "internlm/Intern-S2",
  "temperature": 0.8,
  "top_p": 0.95,
  "top_k": 50,
  "min_p": 0.0,
  "max_tokens": 4096
}

Verify that your client limits context windows to 4096 tokens to stay within the model's native execution limits. For further script configurations, consult the official documentation in the InternLM Official GitHub Repository.