ZoophostDocs

Run Ollama

Install Ollama on a GPU instance, pull a model, expose a private OpenAI-compatible endpoint and keep it running with systemd.

GPU & LLM hosting3 min read

Ollama runs open models with a single binary and exposes both its own API and an OpenAI-compatible endpoint. It is the fastest way to get a private model answering requests.

The steps below assume an Ubuntu 24.04 GPU instance with root access. GPU instances are handed over with root access and nothing pre-installed, so start at step 1.

1. Confirm the GPU is visible

nvidia-smi

You should see the card, its memory and the driver version. If the command is missing, install the driver that matches the image:

apt update
apt install -y nvidia-driver-550
reboot

2. Install Ollama

curl -fsSL https://ollama.com/install.sh | sh

The installer creates a ollama system user and a systemd service. Check it:

systemctl status ollama --no-pager
ollama --version

3. Pull a model

ollama pull llama3.1:8b
ollama list

Start with a model that fits comfortably in VRAM. On a 24 GB card, an 8B model at 4-bit leaves room for context; a 70B model will not fit.

Test it locally:

curl http://127.0.0.1:11434/api/generate -d '{
  "model": "llama3.1:8b",
  "prompt": "Explain what a reverse proxy does in two sentences.",
  "stream": false
}'

4. Expose an API safely

By default Ollama listens on 127.0.0.1:11434, which is correct. Do not open 11434 to the internet — there is no authentication on it.

Put a reverse proxy in front and terminate TLS there. A minimal nginx site:

server {
    listen 443 ssl;
    server_name llm.example.com;

    ssl_certificate     /etc/letsencrypt/live/llm.example.com/fullchain.pem;
    ssl_certificate_key /etc/letsencrypt/live/llm.example.com/privkey.pem;

    location / {
        # Require a bearer token before proxying
        if ($http_authorization != "Bearer change-me") {
            return 401;
        }
        proxy_pass http://127.0.0.1:11434;
        proxy_set_header Host $host;
        proxy_read_timeout 300s;
    }
}

Use a long random token, and keep the port 443 firewall rule open while leaving 11434 closed. If the endpoint is only for other Zoophost services, keep it on the private network and skip the public proxy altogether.

5. Use the OpenAI-compatible endpoint

Ollama serves /v1/chat/completions with the same shape as OpenAI:

curl https://llm.example.com/v1/chat/completions \
  -H "Authorization: Bearer change-me" \
  -H "Content-Type: application/json" \
  -d '{
    "model": "llama3.1:8b",
    "messages": [{"role": "user", "content": "Write a haiku about backups."}]
  }'

Most SDKs work by setting a custom base URL:

from openai import OpenAI

client = OpenAI(
    base_url="https://llm.example.com/v1",
    api_key="change-me",
)

reply = client.chat.completions.create(
    model="llama3.1:8b",
    messages=[{"role": "user", "content": "Hello"}],
)
print(reply.choices[0].message.content)

6. Keep it running

The installer already enabled the service:

systemctl enable --now ollama
journalctl -u ollama -f

Models are stored under /usr/share/ollama/.ollama/models. That directory is on the instance disk; it is not backed up. Copy important weights to a Z2 bucket if re-downloading would be expensive.

Troubleshooting

nvidia-smi shows no process — the model may be running on CPU. Check the Ollama log for “no compatible GPUs”. Install the driver and reboot.

Out of memory — pick a smaller or more quantised model, or lower context. ollama ps shows what is loaded.

Slow first response — the model is loading from disk. Later requests are faster while it stays resident.

Connection refused from another machine — Ollama is bound to loopback. Use a proxy; do not rebind it to 0.0.0.0 on a public interface.

Next

  • ollama
  • llm
  • inference
  • openai compatible
  • systemd
  • gpu

Ready to use this

A whole card with root access, not a shared slice or a per-token meter.

  • 8 GB RTX 2060 SUPER from the entry tier
  • CUDA 12 ready on every plan
See GPU plans
Still stuck? Open a ticket from Support tickets in the console, or email [email protected]. Include the service name, the exact error and what you already tried.