Pass extra flags to llama-server with LLMMAN_LLAMA_ARGS
Pass any llama-server flag through llmman with one variable, LLMMAN_LLAMA_ARGS, whether llama-server runs on your machine or in a container.
When llmman serve loads a GGUF model, by default it starts llama-server from llama.cpp with flags based on your settings: context size, parallel slots, KV-cache type, among others. But llama-server has far more options than that, and to use the rest, you put them in LLMMAN_LLAMA_ARGS.
Say you’re running an agent against a local model. Every request resends the same long prompt (the system prompt, the tool definitions and the conversation so far). llama-server has flags that control how it reuses what it has already processed, such as --slot-prompt-similarity and --cache-ram. But llmman starts llama-server for you, so you can’t add those flags to its command line yourself. LLMMAN_LLAMA_ARGS lets you add them anyway.
Extra flags for each engine
llmman runs models with several inference engines: llama-server, vLLM, SGLang and ds4. Each one has its own variable for passing extra flags: LLMMAN_LLAMA_ARGS for llama-server, LLMMAN_VLLM_ARGS for vLLM, LLMMAN_SGLANG_ARGS for SGLang and LLMMAN_DS4_ARGS for ds4. With LLMMAN_LLAMA_ARGS, you can use any llama-server flag on your own workload without waiting for llmman to add a dedicated setting for it.
The variable
LLMMAN_LLAMA_ARGS="--slot-prompt-similarity 0.5 --cache-ram 16384" llmman serve
This starts the daemon with LLMMAN_LLAMA_ARGS set. Every llama-server it launches from then on gets those two flags appended to its command line: --slot-prompt-similarity sets how closely a new prompt must match a slot’s cached one for that slot to be reused, and --cache-ram sets the size of the prompt cache in RAM, in MiB.
If
llmman serveis already running, restart it to pick up a new value.
Those two are only examples. Any flag llama-server accepts works here, except the few covered under What it refuses, and what to avoid. For the full list, see the llama-server parameters or run llama-server --help.
How llmman applies LLMMAN_LLAMA_ARGS
Set LLMMAN_LLAMA_ARGS once and every llama-server that llmman starts gets the same flags, however and wherever it runs:
- Local or in a container: it doesn’t matter whether
llama-serverruns as a local binary or from theghcr.io/ggml-org/llama.cppimage under--runtime dockerorpodman. Both get exactly the same arguments. The one catch is file paths; see Paths in a container. - Quoted like the others: the value is split the same way
LLMMAN_VLLM_ARGSis, so a quoted argument can hold a space. Put the quotes around the value, not the whole flag:--flag 'a b', not--flag='a b'. - Works under
launchtoo: whenllmman launchstarts the daemon for you, the daemon inherits your environment, so allmman launch dshsession served by a local GGUF picks up the variable as well.
What it refuses, and what to avoid
A few mistakes are caught when llmman serve starts, so they never show up as a failed load later:
- An unclosed quote stops the daemon, with an error that names the variable but not its value.
--portor--hostalso stop it.llmmansets both itself, and a second copy would only conflict.
Some flags already have their own llmman variable, so set them there. For instance, set context size (--ctx-size) with LLMMAN_CONTEXT_LENGTH, parallel slots (--parallel) with LLMMAN_NUM_PARALLEL, and so on. llmman already passes these flags to llama-server, so if you add one to LLMMAN_LLAMA_ARGS as well, llama-server receives it twice and logs a warning that it was specified more than once.
Paths in a container
Flags such as --chat-template-file take a file path. llmman passes the path through unchanged, and llama-server opens the file itself, so whether the path works depends on where llama-server runs:
- Local binary:
llama-serverruns on your machine, so any path on your machine works. - Container:
llama-servercan only see files inside the container, andllmmanmounts just two directories into it: the model’s, at/models, and the multimodal projector’s, at/mmproj. A path like/home/you/template.jinjadoesn’t exist there, sollama-serverwon’t find the file.
For reference, docs/configuration.md lists every llmman variable, including the full quoting rules, and docs/backends.md covers how llmman runs llama-server locally and in a container. Questions are welcome at github.com/llmmanorg/llmman.