Run vLLM
Serve a Hugging Face model with vLLM, tune GPU memory and concurrency, and put a private OpenAI-compatible API in front of it.
vLLM is a high-throughput inference server. It handles continuous batching and paged attention, which makes it a better fit than Ollama when many requests arrive at once.
Use vLLM for a production endpoint; use Ollama for a quick local model.
1. Prepare the machine
nvidia-smi
apt update && apt install -y python3-venv python3-pip
You need a recent NVIDIA driver and enough disk for the model weights. A 7B model in half precision is about 14 GB, plus the CUDA graph and KV cache.
2. Install vLLM
Install into a virtual environment so the system Python stays clean:
python3 -m venv /opt/vllm
/opt/vllm/bin/pip install --upgrade pip
/opt/vllm/bin/pip install vllm
/opt/vllm/bin/python -c "import vllm; print(vllm.__version__)"
If the install tries to build CUDA kernels, confirm the driver and CUDA toolkit versions are compatible with the vLLM release you chose. Pinning a known-good version is safer than always taking the latest.
3. Download the model
Public models download on first use. Gated models need a Hugging Face token:
export HF_TOKEN=hf_xxxxxxxxxxxxxxxxx
/opt/vllm/bin/huggingface-cli login --token "$HF_TOKEN"
For a repeatable setup, pre-download into a fixed cache directory and point the service at it with HF_HOME.
4. Test it in the foreground
/opt/vllm/bin/vllm serve meta-llama/Llama-3.1-8B-Instruct \
--host 127.0.0.1 \
--port 8000 \
--gpu-memory-utilization 0.90 \
--max-model-len 8192
Then:
curl http://127.0.0.1:8000/v1/models
The server exposes an OpenAI-compatible API, so POST /v1/chat/completions works with OpenAI SDKs.
Tuning the flags
| Flag | Effect |
|---|---|
--gpu-memory-utilization |
Fraction of VRAM vLLM may use. Lower it if the machine also runs other GPU work |
--max-model-len |
Maximum context. Larger values reserve more KV cache |
--tensor-parallel-size |
Split a model across multiple GPUs. On a single card leave it at 1 |
--quantization |
Use a quantised checkpoint (for example AWQ or GPTQ) to fit a larger model |
--max-num-seqs |
Cap concurrent sequences; lower it to protect latency |
Start with the defaults, watch nvidia-smi and the server log, then adjust. If startup fails with a KV-cache error, reduce --max-model-len or --gpu-memory-utilization.
5. Run it as a service
Create /etc/systemd/system/vllm.service:
[Unit]
Description=vLLM OpenAI-compatible server
After=network-online.target
Wants=network-online.target
[Service]
User=root
Environment="HF_HOME=/var/lib/huggingface"
ExecStart=/opt/vllm/bin/vllm serve meta-llama/Llama-3.1-8B-Instruct \
--host 127.0.0.1 --port 8000 \
--gpu-memory-utilization 0.90 --max-model-len 8192
Restart=always
RestartSec=10
TimeoutStartSec=900
[Install]
WantedBy=multi-user.target
systemctl daemon-reload
systemctl enable --now vllm
journalctl -u vllm -f
Model loading can take a few minutes; the generous TimeoutStartSec prevents systemd from killing it too early.
6. Put a private API in front
vLLM has no built-in authentication. Keep it bound to 127.0.0.1 and proxy through nginx with TLS and a bearer token, as shown in Run Ollama.
curl https://llm.example.com/v1/chat/completions \
-H "Authorization: Bearer change-me" \
-H "Content-Type: application/json" \
-d '{
"model": "meta-llama/Llama-3.1-8B-Instruct",
"messages": [{"role": "user", "content": "Summarise this ticket in one line: ..."}]
}'
Troubleshooting
Startup fails with “out of memory” — reduce --max-model-len, lower --gpu-memory-utilization, or use a quantised checkpoint.
First request is very slow — the model is loading and compiling CUDA graphs. Warm it with one request before adding it to a load balancer.
Requests queue for a long time — the GPU is saturated. Raise --max-num-seqs only if latency is acceptable, or add a second instance and balance between them.
Model download is blocked — the model is gated. Accept the licence on Hugging Face and use a token with access.
Next
- vllm
- inference
- openai compatible
- huggingface
- throughput
- systemd
Ready to use this
A whole card with root access, not a shared slice or a per-token meter.
- 8 GB RTX 2060 SUPER from the entry tier
- CUDA 12 ready on every plan

