Engineering

Correctness and validation

llamay verify compares llamay to llamay. It cannot see a wrongly wired architecture, because a model rotated the wrong way is rotated the wrong way consistently — Qwen was rotated the wrong way for the life of this repository and every check passed throughout. So there are three checks, and this page is what each catches and what it misses.

Three checks, not one

Each check exists because of what the one before it cannot see. Every architecture in llamay was landed against all three.

1 · `llamay verify`

llamay verify -m model.gguf
make check          # fmt, vet, tests, a generated model, and the invariants

It was written before any optimisation. Batched prefill is an optimisation of sequential decode, so the two must agree; a benchmark of an engine where they do not is a benchmark of two different models.

Row	Asserts
prefill ≡ decode	Feeding n tokens as a batch and feeding them one at a time produce the same logits.
fork exactness	A forked context continues identically — a logit difference of zero, not a small one.
snapshot resume	Writing a context to disk and reading it back changes nothing.
tokenizer round trip	Text → ids → text is the input, byte for byte.
thread invariance	The worker count does not change the output.
batch invariance	Who else is in the batch does not change the output.
vq8 bound	Mean total-variation distance between the exact and quantised cache distributions is under 5%. The bound.
vq8 bound has teeth	The same comparison against a one-bit cache, which must fail — so the row above it can never be vacuous.

2 · Greedy text against llama.cpp

scripts/crosscheck.sh

Both engines read the same file — llamay pulls it, and llama.cpp is pointed at the blob llamay's own store wrote — on the same container, on the CPU, with no GPU anywhere in the harness. Temperature 0, seed 0, against llama-simple.

A crosscheck of a YaRN model has to state its budget

llama-simple derives n_ctx from -n, and gpt-oss's rope scaling is a function of the context it is configured for. The same prompt through the same reference at -n 32, -n 88 and -n 424 produces three different continuations that agree with each other for eight characters. An earlier run recorded 9 of 34 characters agreeing; at a budget close to llamay's own window, all 111 agree. Both were correct measurements of two different reference configurations.

3 · Perplexity

llamay ppl -m model.gguf -f wiki.test.raw -ctx 512


reference	llama.cpp tag `b6510`, built from source, CPU only
runner	Modal, 16 vCPU x86-64 (AVX2), Ubuntu 22.04
text	the first 262,144 bytes of `wiki.test.raw` from wikitext-2-raw-v1, md5 `1d92c7cb8f7e4572f40c7e153a818a56`
window	512 tokens, 8 windows, second half of each scored — the scheme `llama-perplexity` uses, so the two numbers are comparable

A tenth of a percent of perplexity is two correct implementations disagreeing about float rounding. A whole unit is a bug.

Where the architectures stand

Twenty-one pinned GGUF files, reproduced by make matrix-ci. Eight clear the gate, four miss it by a fraction, and two are broken — stated in that order because that is the order it matters in.

Verdict	Architectures	Against llama.cpp's perplexity
Passing	`qwen2` `qwen3` `gemma2` `gemma3` `phi3` `gpt2` `falcon` `bert`	0.014% – 0.488%
Near	`llama` `stablelm` and the mixture	1.2% – 1.4%
Explained	`gpt-oss`	+2.067%, against a reference whose own error bar is ±44.2
Cause fixed, not re-measured	`gptneox` `phi2`	+32.001% and −3.090%
Not run	`gemma` `azmx` `azmx-code` `azmx-ocr` `qwen2moe` `qwen3moe`	no pinned file, or a separate tier

What "cause fixed, not re-measured" means, exactly

gptneox and phi2 both failed the tokenizer round trip on a real file, and both now pass it. A byte-level vocabulary is not uniformly byte-level: Pythia's carries 23 entries of literal spaces marked USER_DEFINED, and llamay put them through the byte decoder, which drops a real space. Pythia additionally declares tokenizer.ggml.pre = "olmo", a name preFor did not know, so it fell through to the legacy rules and split the text 17.9% finer than llama.cpp.

Pythia now makes 29,668 tokens of the matrix's 128 KiB of wikitext, which is llama.cpp's count to the token. What has not happened is a re-run of the perplexity rows, which need the weights and a llama.cpp build. The first number should be expected to move; the second should not, because Phi-2's token counts already agreed exactly and only its decoder was broken.

What this does not cover

Nothing here ran on a GPU. The device graphs decline the parallel residual and the mixture by name, and the i-quants have no device kernel at all.
llamay quantize will not write an i-quant. Every i-quant file measured was produced by llama.cpp and only read.
Five architectures have no pinned file — the mixture mechanism is covered by the Mixtral row, which is the same router and the same stacked expert tensors, but the Gemma 1 block and the AZMX files are covered by fixtures and by nothing else.

gpt-oss, with its methods

Kept in full because it is the row most likely to be quoted out of context. Token ids: identical to tiktoken's o200k over all 45,235 of a 173 kB corpus, where llama.cpp differs. Greedy, 24 tokens, at two prompts: all 106 compared characters agree on one, and all 111 on the other. Perplexity over four windows of 512: 437.42 against 428.57, which is +2.07% and outside this page's gate — against llama.cpp's own error bar of ±44.2 on the same measurement.

The mechanism behind a mixture's sensitivity here is worth knowing: where the router's distribution is flat, a summation-order difference of 1e-06 flips which expert wins a near-tie, and then the whole continuation diverges.