Engineering

Throughput

Three platforms, measured against the same engines on the same files. Every number was produced by the command beside it, every round is shown, and none of it is rounded in llamay's favour. The short version: llamay's x86 prefill beats llama.cpp on three of four models; its decode loses on every platform.

x86-64 — Linux, 18 threads, AVX2, no GPU

Four models, A/B/A/B interleaved, scripts/modal/matrix_ci.py --action throughput. Medians.

Model	llamay pp	llama.cpp pp		llamay tg	llama.cpp tg
qwen2.5-0.5b	94.3	67.1	1.41×	7.6	30.0	0.25×
llama-3.2-1b	52.8	34.2	1.54×	3.0	18.4	0.16×
gemma-3-1b	68.1	33.7	2.02×	5.5	17.2	0.32×
gpt2-124m	261.6	1060.4	0.25×	24.5	213.1	0.11×

Prefill is llamay's, on everything but gpt2 — up to twice llama.cpp's, which is the payoff from the register-tiled K-quant kernels and the batched GEMM. Decode is not, by a factor of three to six.

Apple Silicon — M4, Metal

qwen2.5-0.5b-instruct Q4_K_M, 462.96 MiB. One blob, shared: llamay and llama.cpp were pointed at Ollama's own file rather than at a separate download, so nothing turns on one engine having a differently quantised copy.

Engine	prefill tok/s	decode tok/s
llamay, Metal	625 · 794 · 868 → 794	48.8 · 47.2 · 52.6 → 48.8
llama.cpp, Metal	1217 ± 12.7	71.5 ± 2.3
Ollama	1309 · 1117 · 1118 → 1118	87.8 · 59.9 · 75.8 → 75.8
llamay, CPU only	149.4	24.0

llamay is at 65% of llama.cpp's prefill and 68% of its decode; 71% and 64% of Ollama's.

llamay      llamay bench -m <blob> -prompt 512 -gen 128 -gpu
llama.cpp   llama-bench -m <blob> -p 512 -n 128 -r 3
Ollama      POST /api/generate, num_predict 128, timings from the response

NVIDIA — Tesla T4, CUDA

Engine	prefill tok/s	decode tok/s
llamay, CUDA	3153	152.2
llama.cpp, `-ngl 99`	8259 · 8278 · 8173 → 8237	294 · 292 · 293 → 293

38% of prefill and 52% of decode. After the attention rewrite the tensor-core matmul is 47% of a prefill, and the remaining difference is the one llama.cpp's MMQ has and this kernel does not: staging the next weight tile behind the current multiply. Decode is 47% weight reads.

Windows

Not measured. There is no benchmark harness on a Windows runner, and the release job's /VERYSILENT install check proves the installer works, not what the engine does afterwards. The binary is the same Go with the same AVX2 kernels as the Linux x86 row, so that row is the closest available estimate — but it is an estimate, and it is not written in the table as though it were a measurement.

Two ways this measurement lies if taken casually

Both were found by taking it.

Ollama's prompt cache answers a repeated prompt without prefilling it

The same prompt three times running reports 1212, then 63366, then 57322 tok/s. The last two are a cache hit divided by a duration. Every round here carries a different random nonce, so nothing is a prefix of anything already resident.

A short prompt measures request overhead

Ollama's default was a 37-token prompt and it reported 839 then 3331 tok/s on identical work. The prompts here are 786 to 827 tokens.

And one about llama.cpp: `-ngl 0` is not a CPU run

Not when the binary carries the CUDA backend. It keeps the weights off the card but still sends large prompt matmuls through cuBLAS, which is why -ngl 0 on the T4 reports 1824 to 2254 tok/s of prefill against 9 to 12 of decode. The x86 table is from a container with no GPU at all.

gpt2 is the same defect twice

gpt2-124m is the one x86 row llamay loses, and it loses badly — 0.25× prefill and 0.11× decode. It is also the one model whose perplexity is 4.67% off llama.cpp's while every other model is inside 1.3%, and both llamay kernel sets agree with each other exactly on it, which puts the fault in the gpt2 block rather than in a SIMD path.

Two independent measurements pointing at the same architecture is worth more than either alone, and it is the next thing to fix rather than a footnote.

What this does not say

Four models, three platforms, one quantisation. A 0.5–1 B model at Q4_K_M is not a claim about a 70 B at Q8.
Ollama's decode spread is 60 to 88 tok/s across three rounds on an idle machine, where llama.cpp's is ±2.3. Any single-round comparison against Ollama can be made to say almost anything.
Nothing here measures quality. Correctness carries perplexity and greedy-text agreement, which is the axis where llamay's numbers are good.

Why publish a number that loses

Because it is the number. llamay's argument is that a context is an object you can fork, snapshot and move, and that the engine is auditable — not that it multiplies matrices faster than a project that has been optimising them for years. A page that omitted the decode column would invite the first person who runs llama-bench to distrust every other row on it.

llama-server remains the right answer when raw single-stream throughput is all that matters. Shipping both is a feature.