Guides

Tools and schemas

A server that accepts tools and ignores them fails in the worst available way: it returns a well-formed assistant message containing prose about the function it would have called, the framework parses it as an answer, and nothing anywhere reports a problem. llamay implements tool calling on top of constrained decoding rather than on top of hope.

The constraint is the point

pkg/constrain compiles a JSON Schema to a byte-level pushdown automaton. A token is allowed only if every byte it contributes keeps the automaton alive, so the model can produce the wrong value but not a syntax error — and, because the automaton carries a richer state than a JSON grammar does, it also cannot produce a missing key, a string where a number belongs, or a function nobody offered.

Three automata cost ten to fifteen per cent more than one, not three times as much, because AnyOf drops an alternative as soon as a token rules it out — and the function name in the first few tokens rules out all but one.

Tool calling

tools and tool_choice work in all three API shapes: OpenAI's tool_calls, Anthropic's tool_use blocks, and Ollama's tools. When tool_choice names a function or is "required", the tool's own schema is compiled and the call cannot be malformed. Several tools become a union of automata rather than a wish.

curl localhost:11435/v1/chat/completions -d '{
  "messages": [{"role":"user","content":"weather in pune?"}],
  "tools": [{"type":"function","function":{"name":"get_weather",
    "parameters":{"type":"object",
      "properties":{"city":{"type":"string"},"unit":{"enum":["c","f"]}},
      "required":["city","unit"]}}}],
  "tool_choice": "required"}'

Streaming holds the call back: a partial tool call is not a thing a caller can do anything with, so the deltas carry text and the call arrives complete.

Structured output

Request field	Guarantee
`response_format: {"type":"json_schema", …}`	The full schema constraint.
`llamay_schema`	The same, under llamay's own namespace.
Ollama's `format`, carrying a schema	The same.
`llamay_format: "json"`	The weaker guarantee: any valid JSON.
`llamay_lexicon: true`	Decoding constrained to a word list, with skeleton indexing.

What is enforced, and what is not

	Keywords
Enforced	Object properties and their types, `required`, closed objects, nested objects, arrays with `items` and `minItems`/`maxItems`, the six scalar types, `enum`, `const`, and `minLength`/`maxLength` on strings. A number has to be a JSON number — `1.`, `1e` and `01` are refused, and generation cannot stop in the middle of one.
Ignored, and it says so	`minimum`, `maximum`, `multipleOf` and string `pattern`. A value can still come out of range; it will be the right type under the right key.
Refused at compile time	`$ref`, `anyOf`, `oneOf`, `allOf`, `not`, `patternProperties`, `if`/`then` — with a message naming the keyword. Ignoring one of these changes which documents are legal by so much that the caller would be getting a different schema than the one they sent, with nothing saying so.

Two deliberate departures from the JSON Schema defaults

An object that declares properties and says nothing about additionalProperties is closed. The spec's default is the opposite, and following it would mean a schema with three named properties constrains almost nothing.
Structural whitespace is budgeted, sixty-four characters per document. Whitespace is the only byte that returns the automaton to the state it came from, so with no bound a greedy sampler whose best legal token is a space emits spaces until the token budget runs out and the document is never closed. This is measured, not feared: the fixture model did exactly that.

GBNF grammars

llama.cpp's grammar files are read by a hand-written parser and compiled onto the same byte-level automaton the schemas use; its own grammars/ are in testdata/grammars as tests. Three things are refused, at load, by name:

A left-recursive rule. The automaton expands a rule reference before it consumes a character, so a rule that reaches itself without consuming one never stops expanding. llama.cpp does not check for this and loops when it samples one.
Token terminals (<think>, <[1000]>). They name entries in one model's vocabulary, and this constrains bytes.
A repetition bound above 4096, since each count is a separate state.

What is still unverified

One trained model. Qwen2.5-1.5B-Instruct, one quantisation. The convention is Hermes', so Hermes and Qwen fine-tunes should behave; Llama 3.1's own tool dialect is different and untested here.
Parallel calls under auto. The parser reads several <tool_call> blocks; no case here produced more than one.
Streaming with tools against a trained model. The fixture tests cover the deltas.