Blog
Notes on llmman, OCI model distribution, and running local inference.
- An aggregation of llmman
llmman serve daemons on several machines can now pool their hardware. Name the peers and a request to any node is served by whichever one has the model loaded, or the most room to load it.
- Verify the model, not the hub
llmman now checks cosign signatures before it will run a model, against a trust policy you write with keys you hold.
- Mirror Hugging Face models into any OCI registry
If you want somewhere other than Hugging Face to keep model weights, the boring answer is the registry infrastructure you already run.
- Introducing llmman
A command-line tool that treats LLM models as ordinary OCI artifacts, and serves them over the Ollama, OpenAI and Anthropic APIs your tools already speak.