Ollama Docker Compose Setup: Models, GPUs, and Network Safety

services:
  ollama:
    image: ollama/ollama
    container_name: ollama
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama:/root/.ollama
    restart: unless-stopped

volumes:
  ollama:

docker compose up -d

docker compose ps
curl http://127.0.0.1:11434/api/tags

services:
  ollama:
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]

docker compose -f compose.yml -f compose.nvidia.yml up -d

services:
  ollama:
    image: ollama/ollama:rocm
    devices:
      - /dev/kfd
      - /dev/dri

docker compose -f compose.yml -f compose.amd.yml up -d

Pull a model and verify the API

Pull one model after the container is running:

docker compose exec ollama ollama pull llama3.2

Run a quick local test:

docker compose exec ollama ollama run llama3.2

You can also test the HTTP API directly:

curl http://127.0.0.1:11434/api/generate \
  -d '{"model":"llama3.2","prompt":"Say hello in five words.","stream":false}'

A successful /api/tags response confirms that the API is reachable and the model list can be read. A successful generation response confirms that Ollama can load the selected model. If the first request is slow, allow time for model loading and hardware initialization.

Once verification succeeds, compare your deployment with OpenAlt’s Ollama profile, browse related AI & LLM tools, or find more projects in the self-hosting directory.

Network safety

Do not assume that exposing port 11434 publicly adds authentication. It does not. A public binding can allow unauthorized clients to list models, submit prompts, consume compute, and potentially access sensitive local workflows.

The safest default is:

ports:
  - "127.0.0.1:11434:11434"

This permits access from the Docker host but not directly from other machines. If another application runs in the same Compose project, prefer an internal Docker network and call the service by its Compose name instead of publishing the port.

For controlled LAN access, bind to a private host address, restrict traffic with the host firewall, and avoid exposing the service to the internet. If remote access is necessary, place Ollama behind a reverse proxy or gateway that provides authentication, encryption, request limits, and access logging. Keep the Ollama port private behind that layer.

Network isolation is separate from model security. Prompts, generated content, logs, and client credentials still deserve normal secrets-management and access-control practices.

Backups and reproducible re-pulls

The ollama volume contains the downloaded model data. Backing it up preserves the exact local state and avoids downloading models again after disk failure or migration. The tradeoff is storage: model files can be large, and backups may take considerable time and space.

A reproducible re-pull strategy saves less backup space. Recreate the volume, install the same Ollama image version, and pull the required models again. This is simpler, but it depends on download availability, network bandwidth, registry access, and the exact model reference remaining usable.

For an important host, keep a written model inventory and back up the volume or its underlying storage. For an experimental host, re-pulling may be sufficient.

Updates and rollback

Pin the Ollama image to a known release tag in production instead of relying indefinitely on the moving ollama/ollama tag. Record the selected version in version control, update deliberately, and keep the previous image reference available.

Before upgrading, confirm that the model volume is backed up or recoverable. Then recreate the service with the new image and run the same checks: docker compose ps, /api/tags, and a small generation request.

If behavior regresses, stop the service, restore the previous image reference, and start it with the existing volume. Keeping model data separate from the container makes image rollback independent from model re-downloads.

Troubleshooting

Symptom	Likely check
Port connection fails	Confirm the container is running and port 11434 is bound on the expected address.
Models disappear	Verify that /root/.ollama is mounted to the intended named volume.
API responds but generation fails	Confirm the model was pulled successfully and inspect container logs.
NVIDIA acceleration is missing	Check the host driver, NVIDIA Container Toolkit, Docker integration, and GPU reservation.
AMD acceleration is missing	Confirm /dev/kfd, /dev/dri, the ROCm image, permissions, and host compatibility.
Remote client cannot connect	Check loopback binding, firewall rules, private routing, and proxy configuration.

FAQ

Does Ollama need a GPU?

No. Ollama can run in CPU mode. A compatible NVIDIA or AMD GPU can improve performance, but CPU mode is the simplest baseline for testing and lightweight workloads.

Where are Ollama models stored?

With this Compose setup, models are stored in the persistent ollama volume mounted at /root/.ollama inside the container.

Which port does the Ollama API use?

The local API uses port 11434. Binding 127.0.0.1:11434:11434 keeps direct access on the Docker host.

Is the API safe to expose publicly?

Not by default. Public exposure does not provide authentication. Use loopback or private networking, or place the API behind an authenticated and encrypted proxy.

How do upgrades avoid re-downloading models?

Keep models in the persistent volume and change only the Ollama image version. The container can be replaced while the mounted model data remains available.