Concepts

Architecture

Thirteen packages, six of which carry the weight. This page follows one GGUF file from the disk to a token on a socket, and then follows one HTTP request through the same machinery from the other end.

The package map

PackageWhat it owns
cmd/llamayThe CLI, and the argv[0] trick that makes llamay-server serve with no subcommand.
pkg/storeThe model store: content-addressed blobs and manifests. A blob is only ever installed under the digest of its own finished bytes.
pkg/ollamaReading a local Ollama store, if there is one. A courtesy, not a dependency.
pkg/ggufThe file format: parse, mmap, write. 23 tensor formats read; IQ1_S and IQ1_M deliberately absent, so a file carrying them is refused rather than read as garbage.
pkg/archWhich block shape a file describes, and what a model identifies itself as when the architecture field names only the vehicle. 17 entries reached by 39 declared names.
pkg/quantBlock layouts, dequantisation, and the fused W4A8 kernels every one of them is tested against.
pkg/tensorMatrices, MatVec for decode, MatMul for prefill, and the worker pool that stays off the efficiency cores.
pkg/tokByte-level BPE, SentencePiece and WordPiece, with four hand-written pre-tokenizers — hand-written because RE2 has no lookahead.
pkg/modelWeight binding and the forward pass. State holds one conversation's scratch; Batcher decodes several at once.
pkg/kvThe state engine: paging, the radix prefix tree, copy-on-write forks, versioned snapshots.
pkg/sampleSamplers and the constraint interface they call into.
pkg/constrainJSON, JSON Schema, GBNF and lexicon constraints, compiled to byte-level pushdown automata.
pkg/serveThe HTTP surface, the scheduler, continuous batching and the context routes.

Loading a file

A GGUF file is memory-mapped rather than read. The weights are the vast majority of the bytes, they are read once per token in a pattern the kernel's page cache is good at, and copying six gigabytes into the heap to then read it linearly buys nothing.

mmap the whole file header + KV metadata, tensor index pkg/arch lookup 39 names → 17 blocks bind weights by name, by shape tensor audit did anything go unread? Refused, by name, with what does work an architecture with no block · IQ1_S or IQ1_M · a Mamba/SSM file · DeepSeek-V2 latent attention never loaded partially and run with the missing piece skipped
The audit at the end is the unusual step: llamay info -m file.gguf reports whether any tensor in the file went unread, which is how a half-wired architecture announces itself at load rather than as strange text an hour later.

The forward pass

Prefill and decode are the same code. Batched prefill is an optimisation of sequential decode, so if the two ever disagree, a benchmark of the engine is a benchmark of two different models. This is the first invariant llamay verify asserts, and it was written before any optimisation was.

ONE BLOCK, REPEATED n_layer TIMES embed norm RMS / LN attention Q·K·V, rotary, GQA sink, sliding window pkg/kv read all · write this position + norm feed forward SwiGLU · GELU or a router and k experts + out residual residual The parallel residual is one variant of this Falcon, GPT-NeoX and Phi-2 have both branches read the block's input, and both land on the residual
The KV cache is the only thing in the block with memory across calls, which is why it is a separate package with its own invariants rather than a buffer inside the model.

Two constraints shape the loop rather than the maths. There is no allocation inside the token loop — scratch buffers are allocated once per State — and the worker count avoids efficiency cores, because a pool sized to the core count schedules a third of its work onto cores a fifth as fast and then waits for them.

A request, end to end

POST /v1/chat/… queue bounded → 429 admit KV acquired here prefill prefix matched first, then chunked decode batched across sequences, weights read once stream out SSE, or NDJSON for Ollama first token
Prefill is chunked so one long prompt cannot stall everybody else; decode is batched across sequences so N generations read the weights once rather than N times.

The step that matters is where the KV context is acquired: at admission, not at submission. That is what makes the queue safe to have at all. A queued request is a channel and a few hundred bytes, so the memory ceiling is the batch width rather than the queue depth, and a burst of a thousand requests costs a thousand small structs instead of a thousand caches.

The prefix cache is on the same path

Before prefill runs, Cache.Match(tokens) walks the radix tree and returns a context holding the longest prefix already computed. Only the new suffix is prefilled, and the response reports llamay_cached_prompt_tokens beside the usual usage — the number that says whether the prefix cache is earning its memory. How the tree works.

Why Go

Every hot loop is destined to be assembly or a GPU shader, so the language's job is orchestration. There, Go's static single binary, its cross-compilation and its scheduler are worth more than a marginal codegen advantage over C — and they are what make make cross six targets from one machine.

The costs are real and planned for rather than discovered: no allocation inside the token loop, a worker count that stays off efficiency cores, and hand-written assembly for all eight block formats on both architectures. The portable Go reference kernels stay in the tree and are correct — 2.8× to 7.6× slower per dot product on an M4 — and LLAMAY_BACKEND=portable runs the whole suite through them on a machine that has SIMD, because building a fallback is not the same as running it.