ZoophostDocs

Run vLLM

Serve a Hugging Face model with vLLM, tune GPU memory and concurrency, and put a private OpenAI-compatible API in front of it.

GPU & LLM hosting3 min read

vLLM is a high-throughput inference server. It handles continuous batching and paged attention, which makes it a better fit than Ollama when many requests arrive at once.

Use vLLM for a production endpoint; use Ollama for a quick local model.

1. Prepare the machine

nvidia-smi
apt update && apt install -y python3-venv python3-pip

You need a recent NVIDIA driver and enough disk for the model weights. A 7B model in half precision is about 14 GB, plus the CUDA graph and KV cache.

2. Install vLLM

Install into a virtual environment so the system Python stays clean:

python3 -m venv /opt/vllm
/opt/vllm/bin/pip install --upgrade pip
/opt/vllm/bin/pip install vllm
/opt/vllm/bin/python -c "import vllm; print(vllm.__version__)"

If the install tries to build CUDA kernels, confirm the driver and CUDA toolkit versions are compatible with the vLLM release you chose. Pinning a known-good version is safer than always taking the latest.

3. Download the model

Public models download on first use. Gated models need a Hugging Face token:

export HF_TOKEN=hf_xxxxxxxxxxxxxxxxx
/opt/vllm/bin/huggingface-cli login --token "$HF_TOKEN"

For a repeatable setup, pre-download into a fixed cache directory and point the service at it with HF_HOME.

4. Test it in the foreground

/opt/vllm/bin/vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 127.0.0.1 \
  --port 8000 \
  --gpu-memory-utilization 0.90 \
  --max-model-len 8192

Then:

curl http://127.0.0.1:8000/v1/models

The server exposes an OpenAI-compatible API, so POST /v1/chat/completions works with OpenAI SDKs.

Tuning the flags

Flag Effect
--gpu-memory-utilization Fraction of VRAM vLLM may use. Lower it if the machine also runs other GPU work
--max-model-len Maximum context. Larger values reserve more KV cache
--tensor-parallel-size Split a model across multiple GPUs. On a single card leave it at 1
--quantization Use a quantised checkpoint (for example AWQ or GPTQ) to fit a larger model
--max-num-seqs Cap concurrent sequences; lower it to protect latency

Start with the defaults, watch nvidia-smi and the server log, then adjust. If startup fails with a KV-cache error, reduce --max-model-len or --gpu-memory-utilization.

5. Run it as a service

Create /etc/systemd/system/vllm.service:

[Unit]
Description=vLLM OpenAI-compatible server
After=network-online.target
Wants=network-online.target

[Service]
User=root
Environment="HF_HOME=/var/lib/huggingface"
ExecStart=/opt/vllm/bin/vllm serve meta-llama/Llama-3.1-8B-Instruct \
  --host 127.0.0.1 --port 8000 \
  --gpu-memory-utilization 0.90 --max-model-len 8192
Restart=always
RestartSec=10
TimeoutStartSec=900

[Install]
WantedBy=multi-user.target
systemctl daemon-reload
systemctl enable --now vllm
journalctl -u vllm -f

Model loading can take a few minutes; the generous TimeoutStartSec prevents systemd from killing it too early.

6. Put a private API in front

vLLM has no built-in authentication. Keep it bound to 127.0.0.1 and proxy through nginx with TLS and a bearer token, as shown in Run Ollama.

curl https://llm.example.com/v1/chat/completions \
  -H "Authorization: Bearer change-me" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "meta-llama/Llama-3.1-8B-Instruct",
    "messages": [{"role": "user", "content": "Summarise this ticket in one line: ..."}]
  }'

Troubleshooting

Startup fails with “out of memory” — reduce --max-model-len, lower --gpu-memory-utilization, or use a quantised checkpoint.

First request is very slow — the model is loading and compiling CUDA graphs. Warm it with one request before adding it to a load balancer.

Requests queue for a long time — the GPU is saturated. Raise --max-num-seqs only if latency is acceptable, or add a second instance and balance between them.

Model download is blocked — the model is gated. Accept the licence on Hugging Face and use a token with access.

Next

  • vllm
  • inference
  • openai compatible
  • huggingface
  • throughput
  • systemd

Ready to use this

A whole card with root access, not a shared slice or a per-token meter.

  • 8 GB RTX 2060 SUPER from the entry tier
  • CUDA 12 ready on every plan
See GPU plans
Still stuck? Open a ticket from Support tickets in the console, or email [email protected]. Include the service name, the exact error and what you already tried.