Blog
Notes on llmman, OCI model distribution, and running local inference.
- Find and pull models from the web UI
The web UI has a Models page now: popular models from Docker Hub and Hugging Face, a search of both, and a card per model with every tag or quantization you can pull, its size, and whether it fits your machine.
- llmman now speaks System One
POST /v1/systemone turns a state and a few typed questions into a probability per answer, read from a local model's own token probabilities on ggml. Nothing is generated, so there is no text to parse.
- llmman now runs Qwen-Image 2.1
llmman run qwen-image-2.1 turns a prompt into a 1024×1024 picture, straight from the model's Diffusers safetensors, on ggml. Done in collaboration with Unsloth.
- GPU acceleration for Adreno on Windows on ARM
llama.cpp's only prebuilt GPU binary for Windows on ARM is an OpenCL one, not Vulkan. llmman now probes for it, so machines having it stop silently falling back to CPU.
- llmman now speaks Gemini
Antigravity's CLI talks to models over the Gemini API. llmman now translates that protocol, so AGY can run against a local model or a hosted one.
- llmman launch dsh: Run DeepSeek Harness on any local or hosted model
DeepSeek Harness treats the model as a plugin. llmman runs any model on your own hardware, in one command.
- llmman log: git log for your prompts
llmman serve now keeps a history of the prompts it is sent, and llmman log shows it the way git log shows commits.
- llmman now generates images, audio and video
A diffusion model is a model like any other: llmman run ltx-2.3 turns a prompt into a picture, an mp4 with a soundtrack, or a wav, on whatever GPU you have.
- Hybrid inference: a local model with a hosted one behind it
One model name can now carry a local model and a hosted one. llmman serve picks a side per request, and everything that fits stays on your machine.
- An aggregation of llmman
llmman serve daemons on several machines can now pool their hardware. Name the peers and a request to any node is served by whichever one has the model loaded, or the most room to load it.
- Verify the model, not the hub
llmman now checks cosign signatures before it will run a model, against a trust policy you write with keys you hold.
- Mirror Hugging Face models into any OCI registry
If you want somewhere other than Hugging Face to keep model weights, the boring answer is the registry infrastructure you already run.
- Introducing llmman
A command-line tool that treats LLM models as ordinary OCI artifacts, and serves them over the Ollama, OpenAI and Anthropic APIs your tools already speak.