Documentation
llamay
The AZMX inference engine: a local-first runtime for GGUF models, written from scratch in Go, whose distinguishing idea is that a model's context is an object you can fork, save and move — not a buffer that dies with the process.
What llamay is
One static binary. It runs a model on your machine, serves an OpenAI-,
Anthropic- and Ollama-compatible API on 127.0.0.1:11435, and
never sends anything anywhere unless you configure a provider yourself.
The default build has no cgo and no dependency outside the Go standard
library; each GPU backend is one build tag.
Quickstart
Install, pull a model, make a request. Five minutes, three commands, no configuration file.
Start here →Architecture
How a GGUF file on disk becomes a token on a socket, package by package, with the lifecycle of one request drawn out.
The map →The state engine
Paging, the prefix tree, copy-on-write forks, snapshots that refuse the wrong weights. The part that is not a faster llama.cpp.
The idea →HTTP API
Every route across four API families, with the body it takes and the shape it answers in.
Reference →The shape of the thing
Six packages carry the weight, each with one job. A file is memory-mapped rather than read; a loader decides which block shape the file describes; the model runs the forward pass; a backend does the multiplies; the KV cache holds what has been computed; the server turns HTTP into all of it.
The three claims, and where each is checked
| Claim | What it means | Where it is checked |
|---|---|---|
| A context is an object | Fork it, snapshot it, restore it on another machine. A fork of a 32-page context eight ways allocates zero pages. | State engine — a reference implementation run in lockstep under randomised operations |
| It is deterministic | The same input gives the same output regardless of thread count or batch composition, and a fork continues with a logit difference of zero. | llamay verify, in CI on every push |
| It is correct against llama.cpp | Identical token counts on eight real models, and within 1.3% of its perplexity on eleven of thirteen measured. | The matrix — greedy text and perplexity on pinned files |
What it does not do
Named here rather than discovered later. Each of these is expanded on the page it belongs to.
- It decodes slower than llama.cpp — 68% of its rate on an M4, and a factor of three to six behind on x86. Prefill is the other way round on three of four models. Both columns.
- No Mamba or SSM, and no vision encoder. State-space models are declined at load rather than approximated. DeepSeek-V2 and V3 are refused by name.
- The i-quants are read, not written, and not accelerated.
IQ1_SandIQ1_Mare refused outright. - No LoRA hot-swap. The format is read and the arithmetic costed; no code exists, because a loader that parses an adapter and cannot apply it is worse than nothing.
- Two grammar constructs are refused — left recursion and token terminals — at load, by name, rather than looping at sample time.
Conventions in these pages
- A number with no source is not in here. Every measurement names the command that produced it and the machine it ran on.
llamayis lowercase everywhere — never Llamay, never LLAMAY.- A teal box in a diagram is a thing that is proved by a test; an amber one is the thing the diagram is about; a dashed one is optional, deferred, or not yet written.