Guides

Forking a context

Build a context once, fork it N ways, write it to disk, restore it somewhere else. These routes have no counterpart in any other engine, and they are the reason llamay exists. How they work underneath.

The problem they solve

Four workloads, one shape. An agent exploring alternatives from a shared history. Best-of-N sampling. A document processed against a common instruction. A session that has to survive a closed laptop. Every one of them re-reads the whole prompt on every other engine.

Over HTTP

curl -X POST localhost:11435/v1/contexts \
  -d '{"prompt":"<the long system prompt>"}'
# {"id":"ctx_1","tokens":14,"pages":1}

curl -X POST 'localhost:11435/v1/contexts/ctx_1/fork?n=4'
# 4 branches, each holding all 14 tokens, sharing the parent's page

curl localhost:11435/v1/contexts/ctx_1/snapshot -o base.llamayctx
curl -X POST localhost:11435/v1/contexts/restore --data-binary @base.llamayctx

POST /v1/contexts takes an optional {"prompt": …} or {"messages": [...]} and prefills the context with it, which is the point of building one: pay for the shared prefix once, then fork it.

Route	Does
`POST /v1/contexts`	Create one, optionally prefilled from a prompt or a message list.
`GET /v1/contexts`	List them.
`POST /v1/contexts/{id}/fork?n=`	Fork n ways, sharing every page below the branch point.
`GET /v1/contexts/{id}/snapshot`	Download it: pages, the weight digest, and a CRC over everything before it.
`POST /v1/contexts/restore`	Restore one. Refuses a snapshot made against different weights.
`DELETE /v1/contexts/{id}`	Release it.

Using one in a request

Requests carry llamay's extensions namespaced, so a client that does not know them is unaffected:

{
  "messages": [...],
  "llamay_session": "ctx_3",
  "llamay_retain_prefix": true
}

Responses report llamay_cached_prompt_tokens alongside the usual usage, which is the number that says whether the prefix cache is earning its memory.

From the CLI

llamay ctx build   -m model.gguf -p "$(cat long-system-prompt.txt)" -o base.llamayctx
llamay ctx fork    -m model.gguf -i base.llamayctx -n 8 -o branches/
llamay ctx restore -m model.gguf -i base.llamayctx -p "now answer:" -n 200

llamay ctx fork

source                    base.llamayctx, 46 tokens, 1 pages
branches                  8 in 19µs (2µs each)
pages allocated           0
pages a copy would need   8
memory held               128.00 KiB (a copy would be 1.00 MiB)

Two rules that are not negotiable

A restore refuses another model

The digest of the weights that computed the KV is recorded in the snapshot and verified on load. KV computed under different weights is not detectably wrong at generation time — it produces fluent, confident nonsense — so it has to fail at load or it never fails at all. The refusal is ErrSnapshotModel, distinct from the five other reasons a snapshot can be refused, so a caller can tell a file from the wrong model apart from one that arrived down a broken pipe.

A fork is exact, not close

The test asserts a logit difference of zero between a forked context and its parent, not a small one. A branch that drifts by a rounding error produces agents that quietly disagree about their own history, which is a class of bug nobody debugs successfully.

Moving one between machines

A context serialises position-major rather than page-major, so a snapshot is independent of the page size that produced it: a cache built with 64-position pages restores one written by a cache using 128. Snapshots are written at full precision even from a q8 cache — the trade is deliberate, since a snapshot outlives the settings that produced it, but it does mean the file is larger than the memory it came from.

There is no remote leg yet. Pushing snapshots to object storage is the obvious next step and is not written.