Concepts

The state engine

Inference engines treat a context as memory inside a process. That is the right model for a chatbot and the wrong one for an agent, because agents do not run forward in a line — they branch, backtrack and resume. Three operations fix it, and they build on each other.

Paging

Positions live in fixed-size pages rather than one flat allocation (kv.DefaultPageTokens = 64). A context is an ordered list of page pointers. This buys two things: a context grows without reallocating, and — far more importantly — two contexts can point at the same page. Everything below depends on that one fact.

Layout inside a page is layer-major because that is the order the forward pass reads it in. On disk it is position-major instead, so a cache built with 64-position pages can restore a snapshot written by one using 128.

The prefix tree

Agent traffic is overwhelmingly repetitive: the same system prompt, the same tool definitions, the same conversation re-sent with one more turn on the end. Cache.Match(tokens) walks a radix tree over token sequences and returns a context holding the longest prefix already computed, along with how many tokens matched. Only the new suffix is prefilled.

Eviction has to walk the tree rather than a list, because an interior node created by an edge split holds no context of its own and must not be considered a victim.

Copy-on-write fork

Context.Fork() copies the page-pointer slice and increments refcounts. Nothing else. A page is cloned only when a branch writes to a page whose refcount is above one — in Context.writable, which is the only place in the package where copying happens.

One function copies, and it is the only one. That is what makes the property testable rather than a claim about discipline across four call sites.

Two properties are asserted in tests rather than assumed:

Isolation. A branch that writes is not visible to its parent. Getting this wrong produces cross-talk between agents that reads like a model problem, not a memory one.
Exactness. A forked context continues identically — the test asserts a logit difference of zero, not a small one. A branch that drifts by a rounding error produces agents that quietly disagree about their own history.

Snapshots

A context serialises position-major rather than page-major, so a snapshot is independent of the page size that produced it. Two checks are non-negotiable and both are in the format itself.

Files are written through a temporary and renamed, because a partly written snapshot that looks loadable is worse than no snapshot. The on-disk format has not been bumped: nothing a previous build wrote is newly rejected.

91×faster to restore a 512-token context than to re-prefill it — and the ratio grows with length

0 pagesallocated by forking a 32-page context eight ways

1,654 linesof pointer-sharing arithmetic, checked against a reference that shares nothing

How it is proved

The package is pointer arithmetic, and the way it is checked is a reference implementation run beside it in lockstep. The reference is deliberately stupid: every context holds its own full copy of every position, nothing is shared at all, and the prefix tree is a flat list scanned linearly for the longest key that is a prefix of the query. It is obviously correct because it does nothing clever, so a sequence of operations where the two disagree is a bug in the clever one.

TestModelledOperations drives random sequences of every operation the package offers — allocate, extend, overwrite, fork, truncate, close, retain, match, under an arena and without one, at each of the three precisions — and asserts the whole invariant set after every single step:

A page's refcount is exactly the number of contexts that point at it. A refcount of one has to mean precisely one holder, because that is the fact writable acts on when it writes in place.
No page is held by a context after the cache freed it, none sits in the cache at or below zero, and allocated − freed equals the number of live pages.
Every arena slot the arena believes is out is backing a live page, and no two pages claim the same slot.
A position returns the same value forever unless this context writes it — fork isolation stated as an invariant rather than as a test case.
The tree's entry set and eviction order match the flat-list model exactly, and its structure is a well-formed radix tree.
Closing every context and clearing the tree returns the cache and the arena to empty.

At 500 seeds of 400 operations against each of the six configurations, all of it holds; the committed run is smaller so the suite stays fast. Around that: forks at a page boundary and one position either side, twelve-deep forks of branches that have themselves been forked and written, every truncation point of a shared context, every single-byte corruption and every truncation of a snapshot, and the concurrency the scheduler actually creates — sixteen goroutines forking one parent, snapshot-and-restore racing a decode that is rewriting the shared prefix, eviction racing allocation — under -race.

Four things that were wrong

Kept here because each one is the reason an invariant above is phrased the way it is.

The bug	How it presented	The fix
The prefix tree leaked arena slots	Nothing looked wrong: counters balanced, pages were collected, the entry bound held. Only the arena went on believing slots were occupied, so a server doing exactly what the tree is for drained its device arena over hours and silently stopped using the accelerated attention path.	One release path instead of four hand-written copies. Forty retains against a four-entry bound had left eighty slots out for eight live pages.
Recycled arena slots were handed out dirty	The heap path gets zeroed storage from `make` and the arena path got whatever the last page left, so the same program produced different bytes depending on whether a GPU arena happened to be attached — and an unzeroed slot wrote another conversation's KV into a snapshot.	Slots are zeroed on issue. The snapshot is a file whose entire purpose is to be shipped elsewhere, which is what made this more than an aesthetic problem.
Eviction left the node standing	Five hundred twelve-token inserts against a four-entry bound left five hundred and one nodes, each holding a full copy of its token edge — tens of megabytes retained for prefixes evicted precisely because nobody wanted them.	Eviction prunes upward from the victim, and a node with one child and no context — a split that no longer splits anything — collapses into its child.
`Restore` believed the length it was told	The position count is four bytes; the positions it promises are not. A one-megabyte body could claim several gigabytes before the lie ran into the end of the stream and the checksum that would have caught it.	Token ids are read into a plain slice before anything is allocated for them, and a declared length beyond sixteen million positions is refused by name.

What this composes into

Best-of-N sampling falls out for free: N branches from one position, sharing every page below it.
Agents exploring alternatives from a shared history pay for that history once.
A session survives a closed laptop — snapshot on exit, restore on start, no re-prefill.
A context can move to another machine, which is the local half of a disaggregated prefill/decode split.
kosa8 fork can fork the VM and its live model context together.

What is not done

Retain and Snapshot read the context they are given without the cache lock, so the single-owner rule for a Context covers them too. This is documented rather than enforced.
The prefix tree is bounded by entry count rather than by bytes, so a handful of very long prefixes can pin far more memory than many short ones.
Snapshots are written at full precision even from a q8 cache. The trade is deliberate — a snapshot outlives the settings that produced it — but the files are larger than the memory they came from.
No eviction under memory pressure.
No remote leg. Pushing snapshots to object storage beside kosa8's VM snapshots is the obvious next step and is not written.