llmman now transcribes speech natively
A Canary GGUF posted to llmman serve comes back as text. The whole pipeline, from the WAV bytes to the tokens, runs in llmman on the same ggml library as its image and video models, and matches the reference token for token.
A 10.56 s WAV goes in, and 1.7 seconds later one line of text comes out.
Until now, llmman sent transcription to llama-server, which handles speech by feeding audio into an existing chat model. NVIDIA’s Canary 1B v2 has its own decoder, which is the part that writes the text and is trained only on speech, and llama.cpp, which is the project behind llama-server, has no code for it. llmman now runs Canary itself, on the ggml library it already uses for image and video models.
Try it
llmman serve
curl -s http://127.0.0.1:17434/v1/audio/transcriptions \
-F model=hf.co/handy-computer/canary-1b-v2-gguf:Q8_0 \
-F file=@clip.wav
{"text":"..."}
llmman serve speaks the OpenAI audio API, and the first request pulls the GGUF file that holds the model from Hugging Face, in its Q8_0 (8-bit) or F16 (16-bit) version. A request takes four fields:
file: a 16-bit mono WAV at 16 kHz, the format Canary was trained on.model: the model reference.language:enby default.response_format:json, the only format.
Other fields and values are refused, and so is every endpoint other than /v1/audio/transcriptions. llmman run does not transcribe yet.
Files are capped at 4 MiB, about 131 s of audio. Anything longer than 30 s is split into equal clips and the texts are joined, since NeMo itself chunks audio longer than 40 s.
Why a pipeline of its own
Each GGUF file records which kind of model it holds. For a Canary file, llmman serve starts a separate process that runs only the speech model.
A model is made of the numbers that it learned in training, called weights and stored in the GGUF, and the code that runs them. NVIDIA’s code for Canary is in NeMo, which is NVIDIA’s Python toolkit for speech, so running Canary anywhere else needs a port of that code.
A C++ port, transcribe.cpp, exists, but it bundles its own patched ggml, which would put two copies of the library in one process, where they could clash. Every operation Canary needs is already in the ggml llmman loads, so llmman’s port is Rust on that library, like its video (LTX-2.3) and image (Qwen-Image) pipelines.
The pipeline runs four stages, in order:
- The front end turns samples into mel bands, which are numbers that describe how much energy the sound has at each pitch.
- The encoder turns mel bands into audio vectors, which are lists of numbers that each describe a short stretch of sound.
- The decoder turns those vectors into token ids, which are numbers that each stand for a word or a piece of a word.
- A table lookup turns the ids into text.
The numbers that follow come from one 10.56 s test clip. Frames, vectors and tokens grow with the length of the audio, up to a 30 s clip.
Front end: from samples to mel bands
Cutting the samples into frames
A WAV holds the pressure readings that a microphone takes every 62.5 µs, called samples. The model works on frames, which are short slices of samples over which a vowel or a consonant stays roughly stable. The recipe comes from NeMo and from the GGUF’s front-end metadata:
- Lift the consonants (pre-emphasis): each sample has 0.97 times the previous sample subtracted from it, which makes the quieter consonants stand out.
- Pad the edges: 256 zeros are added on each side to centre the first frame on the first sample, as NeMo does, while the transcribe.cpp port mirrors the edge samples and its notes record that this choice alone visibly changes the mel bands.
- Cut into frames: the samples are cut into frames of 400 samples (25 ms), with a new frame starting every 160 samples (10 ms), so the frames overlap and a sound cut at the edge of one frame stays whole in the next.
- Fade the frame edges (Hann window): each frame’s edges are faded to zero, because a sharp cut would add energy that the sound never had.
Turning each frame into mel bands
A Fourier transform, which is a standard calculation that finds the pitches a sound contains, splits each frame’s range of 0 to 8 kHz into 257 bins, which are slices 31.25 Hz wide, and measures the frame’s energy in each one. The whole front end runs in plain Rust, because ggml has no Fourier transform.
To give you an example, think of a man’s voice and a woman’s voice. They sit only a few bins apart, and you can tell them apart at once, yet a gap of the same size in the hiss of an “s” goes unheard, because our ears are much worse at telling high pitches apart. That is why the front end groups neighbouring bins into 128 mel bands, which are frequency slices narrow at low pitches and wide at high ones, and blends each band’s energy into one number. Going from 257 numbers per frame to 128 loses detail that we barely hear.
The front end ends with two adjustments: a logarithm, which turns each energy into a number that tracks loudness the way we hear it, and a rescaling of each band over the whole clip, so that its values centre on zero and vary by a similar amount in every band. Together they make a quiet recording and a loud one look alike to the model, which was trained on input prepared the same way.
The front end hands on a log-mel spectrogram, which for the test clip holds 1,056 frames of 128 mel bands each, one frame every 10 ms.
Encoder: from mel bands to audio vectors
The encoder turns the spectrogram into a short sequence of vectors, one per 80 ms of audio, each describing what was said around that moment. It first shrinks the sequence and then refines it.
Shrinking 1,056 frames to 132
The encoder passes the frames through three convolutions, one after another, each with its own weights.
A convolution reads a group of three neighbouring frames and combines them into one new frame. To do that, it multiplies each number in the group by one of the model’s weights and adds up the results. Each group starts two frames after the previous one, so the groups overlap. Frames 1 to 3 make the first new frame, frames 3 to 5 the second, frames 5 to 7 the third. Starting a new group every two frames gives half as many frames out as went in.
The first convolution turns 1,056 frames into 528, the second 528 into 264, and the third 264 into 132. Each also halves the 128 bands, down to 16, each now holding many learned values. A last step, called a projection, multiplies each of the 132 new frames by another set of weights to give a vector of 1,024 values.
Refining the 132 vectors through 32 encoder blocks
The encoder runs 32 blocks over the 132 vectors, one after another. A block is a fixed sequence of steps with its own learned weights, and all 32 share the same steps and differ only in their weights. Each block keeps the vectors’ count and size, and updates every vector three ways:
- Attention: each vector looks at all 132 vectors and draws on the ones that matter to it, such as the rest of its word or sentence. Because every vector looks at every other one, the work grows with the square of their number, so shrinking 1,056 frames to 132 vectors makes this step about 64 times cheaper.
- Convolution: the same kind of operation as in shrinking, over groups of 9 vectors (one vector and its 4 neighbours on each side, about 0.7 s of audio) that start one vector apart, so all 132 stay. It catches where one speech sound ends and the next begins.
- Feed-forward: a small network, which is a short chain of calculations that has its own weights, refines each vector on its own, using what attention and convolution gathered.
Attention plus feed-forward is a Transformer block, which is the same block inside every LLM. Adding the convolution makes it a Conformer block, and FastConformer is NVIDIA’s version with the 8x shrink in front.
Decoder: from audio vectors to text
The decoder is a Transformer like the one in a chat model. It writes the transcript one token at a time, where a token is a word or a piece of a word from a vocabulary of 16,384. Each of its 8 blocks has two kinds of attention:
- Self-attention looks at the tokens written so far, so the text stays coherent.
- Cross-attention looks at the 132 audio vectors the encoder produced, so the text follows the audio.
Before writing any word, the decoder reads a prompt of ten special tokens from the GGUF metadata. Canary learned them in training as instructions for the task, and they sit at the start of the token sequence as if the decoder had written them, so self-attention sees them at every step:
▁: the context slot, left empty.<|startofcontext|>: the context marker. With the slot empty, there is no earlier text to continue.<|startoftranscript|>: the transcript begins.<|emo:undefined|>: no emotion tag.<|en|>: the source language, here English.<|en|>: the target language, the same for transcription.<|pnc|>: punctuation and capitals on.<|noitn|>: numbers stay as spoken words, such as “twenty”.<|notimestamp|>: no timestamps.<|nodiarize|>: no speaker labels.
The first real word becomes the 11th token, after the 10 above. Each step takes the most likely next token, until the model emits its end-of-text token or a length cap is reached.
Last, the speech process maps each token id to its piece of text through the GGUF vocabulary and joins the pieces. The prompt asked for punctuation and capitals, so they are already there. The 1.7 s is the time inside the speech process alone. Sent through llmman serve, which also loads the model first, the clip takes 3.6 s.
Validating against NeMo
A correct transcript for one clip proves little about the port. A small mistake in one stage, such as padding the samples differently from NeMo, can leave that transcript unchanged and still give wrong text on another recording. So we compared the output of every stage with NeMo’s, on one English clip and with both GGUF versions of Canary, Q8_0 and F16.
We ran the clip through NeMo and saved every stage’s activations, which are the numbers that one stage hands to the next. They are the mel spectrogram, the 1,024-value vectors after the shrink and after several encoder blocks, and the decoder’s score for every possible next token. Then we ran the same clip through llmman and compared the two sets of numbers position by position.
For each stage, the gap between llmman’s numbers and NeMo’s is a percentage of a typical NeMo number. For instance, 0.1% means the two agree to about one part in a thousand.
| Stage | F16 | Q8_0 |
|---|---|---|
| Mel spectrogram | at most 0.013% | at most 0.013% |
| Encoder | 0.04% to 0.12% | 0.38% to 3.8% |
| Decoder | 0.008% to 0.08% | 0.11% to 2.1% |
| Prompt, output tokens and text | identical | identical |
Q8_0 and F16 are the two published versions of Canary, with 8-bit and 16-bit weights. The front end uses no weights, so it gives the same numbers with both. With the F16 file, the CPU rounds the numbers inside every multiplication by a weight, from the encoder on, to half precision, which is a 16-bit format with about three significant digits. Half precision alone causes a gap of about 0.05%, and the F16 encoder sits near it, while Q8_0 keeps less detail and ends up about ten times further away. Both still produce NeMo’s 33 tokens and text.
Next
This first version transcribes audio files on the CPU. Next, we plan to:
- Run on GPUs: make llmman transcribe on a GPU through Vulkan. The ggml library already has every operation Canary needs there, and we still have to test it.
- Transcribe live audio: transcribe while someone is still speaking. For now, llmman waits for the whole file.
- Add timestamps: return the time at which each word was spoken, next to the text.
- Handle long recordings better: overlap the 30 s clips, so a word that falls on a cut comes out right.
- Improve decoding: try several candidates at each step, a method called beam search, for a slightly better transcript. Today each step keeps only the most likely token.
- Translate: let llmman ask Canary to translate speech into another language. So far it only asks it to transcribe.
- Support Parakeet: NVIDIA’s other speech model shares Canary’s front end and encoder, so llmman mostly needs to add its decoder.
References
The work is tracked in the project’s tracking issue. The model is described on its model card, and its reference implementation is in NeMo. The compute library is ggml, shared with llama.cpp. The C++ port is transcribe.cpp, which also publishes the Canary GGUF used here. Questions are welcome at github.com/llmmanorg/llmman.