vLLM, set up properly

vLLM is what you move to when one person chatting stops being the job. It is much faster when several requests arrive at once, and a worse experience for everything else: Linux, Docker, CUDA, and flags that refuse to load a model rather than quietly coping. This gets you from nothing to a working server, then to the configuration we actually run.

Is it worth it for you?

Worth settling before you spend an evening on it. The question is whether several requests will be in flight at once.

If it is you, one chat window, one question at a time, LM Studio gives you nearly the same single-stream speed with none of this. Our numbers make the case: a single request against our 80B model returns 118 tokens per second; four at once aggregate to 315. That second number is the entire argument for vLLM, and if you will never see it you are doing a lot of work for the first.

It earns its keep with more than one person or agent hitting the model, with agent frameworks that fan out requests, and with bulk jobs over thousands of documents.

Prerequisites

On Windows you want WSL2 with Ubuntu. Everything after that is identical to running on Linux directly, which is the point of doing it this way.

# Windows only, in an admin PowerShell. Reboot afterwards.
wsl --install -d Ubuntu

Then, inside Ubuntu (or on your Linux box), Docker and the NVIDIA Container Toolkit. The toolkit is the part that lets a container see the GPU, and skipping it is the most common reason the container starts and then reports no devices.

# Docker
curl -fsSL https://get.docker.com | sh
sudo usermod -aG docker $USER   # log out and back in after this

# NVIDIA Container Toolkit
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey \
  | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list \
  | sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' \
  | sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update && sudo apt install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Check the GPU is visible from inside a container before going any further. If this does not print your card, nothing below will work:

docker run --rm --gpus all nvidia/cuda:12.4.0-base-ubuntu22.04 nvidia-smi

You do not need the CUDA toolkit on the host. The driver plus the container toolkit is enough, and the container brings its own CUDA.

Quick start: one command

This pulls the official image and serves an 8B model. It is a large download the first time, both the image and the weights.

docker run --rm -it \
  --gpus all \
  --ipc=host \
  -p 8000:8000 \
  -v ~/.cache/huggingface:/root/.cache/huggingface \
  vllm/vllm-openai:latest \
  --model Qwen/Qwen3-8B \
  --max-model-len 16384 \
  --gpu-memory-utilization 0.90

Three of those arguments matter more than they look:

• --ipc=host - without it you will hit shared-memory errors as soon as the model is anything but tiny. If you would rather not share the host namespace, set --shm-size 16g instead.
• -v ~/.cache/huggingface - mounts the model cache outside the container. Skip it and every restart re-downloads tens of gigabytes.
• --max-model-len - caps context. vLLM reserves memory for the full window at startup, so an uncapped model can fail to load on a card that would otherwise hold it comfortably.

Gated models such as Llama need a token: add -e HF_TOKEN=hf_....

Check it works

The API is OpenAI-compatible, which is the other reason to run it - anything written against OpenAI works with a changed base URL.

curl http://localhost:8000/v1/models

curl http://localhost:8000/v1/chat/completions \
  -H "Content-Type: application/json" \
  -d '{
    "model": "Qwen/Qwen3-8B",
    "messages": [{"role": "user", "content": "Explain VRAM in two sentences."}],
    "max_tokens": 120
  }'

If the first returns your model and the second returns text, you have a working server. Anything pointed at http://localhost:8000/v1 with any API key string will now talk to it.

The compose file we run

The single command is fine for trying it. For anything you come back to, put it in compose with the model-specific parts in a .env, so switching models is an edit and a restart rather than a remembered incantation.

services:
  vllm:
    image: vllm/vllm-openai:latest
    ipc: host
    shm_size: 16g
    ports:
      - "8000:8000"
    volumes:
      - hf-cache:/root/.cache/huggingface
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    command: >
      --model ${MODEL}
      --served-model-name ${SERVED_NAME}
      --max-model-len ${MAX_MODEL_LEN}
      --kv-cache-dtype ${KV_CACHE_DTYPE}
      --gpu-memory-utilization ${GPU_UTIL}
      --tensor-parallel-size ${TP_SIZE}
      ${EXTRA_ARGS}

volumes:
  hf-cache:

With a matching .env:

MODEL=Qwen/Qwen3.6-35B-A3B-AWQ
SERVED_NAME=qwen35b
MAX_MODEL_LEN=32768
KV_CACHE_DTYPE=fp8
GPU_UTIL=0.90
TP_SIZE=1
EXTRA_ARGS=--enable-prefix-caching --max-num-batched-tokens 8192 --max-num-seqs 16

docker compose up -d
docker compose logs -f vllm

Swapping models

Keep a preset per model rather than editing flags by hand, because the flags that differ between models are exactly the ones that stop it loading. Ours is a small script that writes the .env and restarts the container - a minute or two once the weights are cached.

# write the preset, then restart
cp presets/qwen35b-awq.env .env
docker compose up -d --force-recreate vllm

One model at a time is the normal way to run this. vLLM claims most of the card at startup, so two servers on one GPU will fight over memory.

The flags that matter

--enable-prefix-caching --max-num-batched-tokens 8192 --max-num-seqs 16

docker run -d -p 3000:8080 \
  -e OPENAI_API_BASE_URL=http://host.docker.internal:8000/v1 \
  -e OPENAI_API_KEY=none \
  -v open-webui:/app/backend/data \
  --name open-webui \
  ghcr.io/open-webui/open-webui:main

environment:
  - NCCL_P2P_DISABLE=1