Reference

CLI reference

Seventeen subcommands. llamay on its own prints them; llamay <command> -h prints that command's flags. This page is what each one is for.

The commands

Command	What it does
`run`	Generate text from a prompt.
`serve`	Serve the OpenAI-, Anthropic- and Ollama-compatible endpoints. Guide.
`info`	Describe a GGUF file: architecture, shapes, quantisation, and a tensor audit that says whether anything in the file went unread.
`pull`	Download a model into llamay's store. Five source forms; resumes; verifies.
`models`	List what can be named instead of a path, with size and architecture.
`rm`	Remove a model from the store.
`cp`	Give a stored model a second name, sharing one copy of the weights.
`bench`	Prefill and decode measured separately, plus fork, snapshot and restore.
`verify`	The invariants: prefill ≡ decode, fork exactness, snapshot resume, tokenizer round trip, thread-count invariance, the quantised-cache bound. What each row means.
`ppl`	Perplexity over a text file, in the scheme `llama-perplexity` uses so the two numbers are comparable.
`embed`	Turn text into vectors with an encoder.
`rerank`	Score documents against a query with a cross-encoder.
`ctx`	`build`, `fork` and `restore` contexts. Guide.
`quantize`	Rewrite a model in another quantisation. Will not write an i-quant or MXFP4.
`imatrix`	Measure which columns a model drives, to guide quantisation.
`ocr`	Decode recogniser frames, optionally against a lexicon.
`version`	What this build is: tags, kernel sets, GPU backends compiled in, formats, architectures. The first command to run when a number looks wrong.

Two names, one binary

A binary named llamay-server serves with no subcommand: the program reads argv[0] and preselects serve, so tooling written for llama-server needs no change. --version is answered before that rename, because it is the question deployment tooling asks first and it asks it of whatever binary it was handed.

The flags every model command shares

Flag	Default	Meaning
`-m`	—	A model name or a path to a GGUF. A file that exists always wins over a name.
`-t`	`0`	Worker threads. 0 picks a default that avoids efficiency cores.
`-kv`	`f32`	Cache precision: `f32`, `vq8` (values quantised, keys exact) or `q8` (both).
`-kv-page`	`64`	Positions per KV page.
`-gpu`	off	Run the whole layer stack on the device, in one command buffer.
`-ngl`	`-1`	llama.cpp's spelling of `-gpu`: 0 is CPU only, anything at or above the layer count is the whole model on the device.
`-no-repack`	per device	Keep weights in their file format instead of rewriting them for a faster kernel. On by default for the CPU and unified memory, off for a discrete GPU.
`-lexicon`	—	A newline-separated word list for lexicon-constrained decoding.

Worked examples

Getting a model

llamay pull qwen2.5:0.5b                        # an Ollama-style registry
llamay pull hf.co/ggml-org/tiny-llamas:q8_0     # Hugging Face, quant as the tag
llamay pull hf:owner/repo/file.gguf@revision    # an exact file, pinned
llamay pull https://example.com/model.gguf      # any URL serving a GGUF
llamay cp qwen2.5:0.5b small:latest             # a second name, not a second copy
llamay rm qwen2.5:0.5b

Generating

llamay run -m azmx-one-q4.gguf -p "भारत के बारे में बताइए" -n 256
llamay run -m azmx-code-q4.gguf -p "write a binary search in Go" -temp 0.2
llamay run -m azmx-one-q4.gguf -p "extract the fields" -json   # always parseable

Samplers: greedy, temperature, top-k, top-p, min-p, typical-p, repetition/presence/frequency penalties, DRY, logit bias, banned tokens. -stats reports what a constraint mask cost, which a synthetic benchmark cannot.

Inspecting and converting

llamay info     -m model.gguf -tensors
llamay quantize -i model-f16.gguf -o model-q4.gguf -type q4_0
llamay verify   -m model.gguf
llamay bench    -m model.gguf -prompt 2048 -gen 256
llamay ppl      -m model.gguf -f text.txt -ctx 512

OCR

llamay ocr -frames page.f32 -alphabet devanagari.txt -lexicon hi.txt -compare

llamay ocr -compare

greedy    120µs  भरत एक दश ह
beam      890µs  भरत एक देश है
lexicon   940µs  भारत एक देश है

Greedy collapse discards the model's second choice at every frame, and a lexicon-aware beam search resolves those choices against words that actually exist. On AZMX OCR that is worth several points of word accuracy without touching a weight.

Environment

Variable	Meaning
`LLAMAY_MODELS`	The store. Defaults to `~/.llamay/models`.
`LLAMAY_API_KEY`	Required on every route but `/healthz`. Better than `-api-key`, which is visible in the process list.
`HF_TOKEN`	Gated Hugging Face repositories, and a higher rate limit.
`LLAMAY_BACKEND=portable`	Force the reference Go kernels — the way to tell a SIMD bug from a block bug.
`LLAMAY_NO_BLAS=1`	Make an `accelerate` build bit-exact again.
`LLAMAY_GPU`	Force a device backend on or off.
`LLAMAY_NO_SERVICE=1`	Read by the installer: install the CLI and register no service.
`OTEL_EXPORTER_OTLP_ENDPOINT`	Same as `-otlp`.