Concepts

Models and formats

Seventeen architectures reached by thirty-nine declared names, twenty-three tensor formats, three tokenizer families and three KV cache precisions. Anything outside that is refused at load with a message naming what does work — never loaded partially and run with the missing piece skipped.

Architectures

Declared	Covers	What makes it different
`llama` `mistral` `azmx`	Llama 1–3, Mistral, AZMX One and Code, most fine-tunes	The baseline: GQA, interleaved rotary, RMSNorm, SwiGLU.
`qwen2` `qwen3`	Qwen 2, 2.5 and 3	The neox rotary convention; qwen3 adds per-head query and key norms.
`gemma` `gemma2` `gemma3`	Gemma 1, 2 and 3	Scaled embeddings, GELU gate, post-attention and post-feed-forward norms, logit softcapping, sliding window on 1-in-2 or 5-in-6 layers with a second rotary base on the local ones.
`phi3`	Phi-3 and Phi-3.5	One fused attention projection, and a feed forward with the gate stacked on the up projection, split at load.
`stablelm`	StableLM 2	Layer normalisation with biases, and only a quarter of each head rotated.
`gpt2`	GPT-2, and AZMX's BIJA models	Layer normalisation, learned absolute positions, an ungated feed forward.
`falcon` `gptneox` `phi2`	Falcon, Pythia, the GPT-NeoX lineage, Phi-2	The parallel residual: attention and the feed forward both read the block's input, and both land on the residual.
`qwen3moe` `qwen2moe`, and `llama` with `expert_count`	Qwen3-MoE, Qwen1.5-MoE, Mixtral	A router picks k of n per token, experts stored as stacked 3-D tensors, a shared expert where the architecture has one.
`gpt-oss`	gpt-oss 20B and 120B	A per-head attention sink in the softmax denominator, MXFP4 experts, YaRN scaling, a clipped SwiGLU gate whose linear half is offset by one, a bias on every expert projection, the o200k pre-tokenizer.
`bert`	all-MiniLM and the sentence transformers; the ms-marco cross-encoders	Not a decoder at all: bidirectional attention, post-norm blocks, WordPiece, mean pooling. With a classification head and a pair input, a reranker.
`azmx-ocr`	The AZMX recogniser	A CTC head over a script alphabet.

Tensor formats

Twenty-three read: F32, F16, BF16, Q4_0/1, Q5_0/1, Q8_0, Q8_1, Q2_K through Q8_K, MXFP4, and the i-quants IQ4_NL, IQ4_XS, IQ2_XXS, IQ2_XS, IQ2_S, IQ3_XXS, IQ3_S. llamay info -m file.gguf -tensors reports what one file is, including whether any tensor in it went unread.

Family	Read	Written by `llamay quantize`	Accelerated
F32, F16, BF16	yes	yes	every backend
Q4_0/1, Q5_0/1, Q8_0 and the K-quants	yes	yes	assembly on both CPU architectures, and every device backend
MXFP4	yes	fixtures only	portable Go only — the codebook needs a byte shuffle, not a shift
IQ2, IQ3, IQ4	yes	no	portable Go only — a codebook lookup is a gather, which is what NEON and AVX2 are worst at
IQ1_S, IQ1_M	refused outright	no	—

Why the i-quants are read and not written

Picking grid points well is a search against an importance matrix, and a bad i-quantiser is worse than none — it produces a file that loads, runs, and is quietly worse than the one it replaced. llamay imatrix exists to measure which columns a model drives; the quantiser that would use that measurement does not.

The KV cache, in three precisions

The format that matters for memory at run time is the cache, and it is offered in three precisions rather than one. The split between them is a measurement rather than a preference.

Mode	Cost	What it quantises	When
`f32`	100%	Nothing. Exact.	The default, and what to use when a model's `verify` row is red.
`vq8`	66%	Values. Keys stay exact.	The recommended trade on everything that clears the bound.
`q8`	31%	Both.	Offered as a note, not a recommendation.

On a real prompt, quantising keys alone shifts final logits by 11.9% of their range and quantising values alone shifts them 1.8% — because a key feeds a dot product that a softmax then exponentiates. That asymmetry is the whole reason vq8 exists as a separate mode.

Two models need the exact cache

gpt-oss moves 12.7% of the sampler's mass under vq8 and phi3-mini moves 5.6%, both over the 5% bound. Run those with -kv f32. llamay verify reports it per file, so this is a thing you are told rather than a thing you notice.

The bound, and why it is the one it is

The check used to fail two real files, and the check was wrong rather than the files. It asked whether the largest logit difference anywhere in the vocabulary was a small fraction of the peak-to-peak logit range. Both halves are extreme-value statistics over the whole vocabulary and neither is about the top of the distribution: the numerator landed at ranks 4,671 to 226,526 on real files — nowhere near a token any model would emit — and the denominator is the distance down to an untrained embedding row.

What it asserts now is the mean total-variation distance between the exact-cache and quantised-cache distributions over every position, bounded at 5%: the probability a sampler notices. It is the mean and not the worst position, because a mixture reroutes an expert on a 1e-3 error and worst-position cannot separate a healthy cache from a destroyed one at all.

Cache bits	Measured TV distance across 14 fixtures and 7 real files
8	0.17 – 4.6%
4	3.0 – 10.1%
2	7.6 – 33.4%

A permanent vq8 bound has teeth row runs the same comparison against a one-bit cache and fails if that passes, so the row above it can never be vacuous.

Tokenizers

Byte-level BPE with the GPT-2, Qwen 2, Llama 3 and o200k pre-tokenizers — hand-written, since RE2 has no lookahead — plus SentencePiece and WordPiece. A file whose tokenizer.ggml.pre names a rule set llamay does not implement still loads, under a guess that Tokenizer.PreKnown records rather than hides.

Why the round trip is a gating test

AZMX's vocabulary beats every comparable tokenizer on Indic fertility by 2.1× to 6.5×. An engine that reproduces its input only approximately gives that advantage back, and does so silently — the text comes out looking almost right. Two real architectures failed exactly this: a byte-level vocabulary is not uniformly byte-level, Pythia's carries 23 entries of literal spaces marked USER_DEFINED, and putting them through the byte decoder drops a real space.

Where models come from

Source form	Meaning
`<name>[:<tag>]`	An Ollama-style registry, `registry.ollama.ai` by default.
`hf.co/<owner>/<repo>[:<quant>]`	A Hugging Face repository; the tag picks the quantisation.
`hf:<owner>/<repo>/<file.gguf>[@rev]`	An exact file, optionally pinned to a commit.
`https://…`	Any URL serving a GGUF — an internal mirror, an object store, a colleague's machine.
A path	A file that exists always wins over a name. No store on the machine can change what a path means.

llamay fetches from registries and hosts nothing. There is no llamay registry and there will not be one: hosting weights is a bandwidth business with nothing defensible in it, and an engine that requires its own registry has made itself the gatekeeper of its own models. That is why llamay pull exists and POST /api/push is refused by name.

What is refused

Mamba and the state-space models — declined at load rather than approximated.
Vision encoders — not implemented on any backend.
DeepSeek-V2 and V3 — refused by name, and the reason is not a missing router: multi-head latent attention caches one shared compressed latent per position instead of a key and a value per head, so it changes what a KV page holds. That is the paged allocator, the copy-on-write fork and the snapshot format, not the attention alone.
The OCR encoder — the decoder is here, because on that model the decoder is where the accuracy is. Frames come in from the recogniser; the encoder graph follows.
LoRA hot-swap — the adapter format has been read off a real file and the arithmetic costed at 0.78% of overhead at rank 16. No code exists, because an adapter has to reach 41 call sites in pkg/model and three device backends, and a loader that parses the file and cannot apply it would be worse than nothing.