LLM Harbor
Containerized LLM toolkit: manage backends, APIs, and frontends via one CLI
Overview
LLM Harbor is a Docker-based orchestration toolkit that lets you spin up any combination of LLM backends (Ollama, LocalAI, vLLM), API proxies, chat front-ends, and auxiliary services with a concise CLI. It abstracts docker-compose complexity behind simple commands like harbor up ollama open-webui, enabling rapid switching between LLM stacks for experimentation. All configuration is file-based and version-controllable.
Where it falls short of OpenAI API
- Niche tool primarily aimed at power users; limited documentation for beginners
- No built-in UI beyond what the composed services provide
- Community is small; issues may go unanswered compared to larger projects
- Not suitable for production multi-user deployments without significant additional hardening
We list the gaps honestly so you can decide if the trade-off is worth owning your data.
Tags
Claim this listing to keep it accurate, add a deploy template, or feature it on relevant pages.
Embed the LLM Harbor difficulty badge in your README — it links back here.
[](https://openreplace.com/llm-harbor)Similar open-source projects
Other self-hostable tools in the same space worth comparing.
Run large language models locally with a simple CLI and REST API
Feature-rich self-hosted chat UI for Ollama and OpenAI-compatible APIs
Modern AI chat framework with multi-provider support and MCP marketplace
All-in-one local AI app with RAG, agents, and no-code agent builder