Integrating llamay

~/llamay $ llamay serve -m qwen2.5:0.5b -addr localhost:11439
time=2026-09-18T08:51:03.682-04:00 level=INFO msg="device graph attached" max-tokens=256
time=2026-09-18T08:51:03.682-04:00 level=INFO msg="llamay serving" addr=localhost:11439 mode
l=qwen2.5:0.5b arch=qwen2 max-loaded=2 keepalive=5m0s params="0.36 GB" context=32768 threads
=7 kv=f32 lexicon=0

  OpenAI      POST http://localhost:11439/v1/chat/completions
  Anthropic   POST http://localhost:11439/v1/messages
  Embeddings  POST http://localhost:11439/v1/embeddings     (start with -embed <encoder.gguf
>)
  Rerank      POST http://localhost:11439/rerank            (start with -rerank <reranker.gg
uf>)
  Contexts    POST http://localhost:11439/v1/contexts        (then /{id}/fork?n=8, /{id}/sna
pshot)
  Models      POST http://localhost:11439/api/pull          (and /api/copy, /api/delete)
  Stats       GET  http://localhost:11439/v1/stats


~/llamay $ # the server stays up; the rest of this is a second terminal
~/llamay $ # the official OpenAI client, in a clean virtualenv
~/llamay $ python3.12 -m venv build/demo/work/venv && build/demo/work/venv/bin/pip install -
q openai \
>   && build/demo/work/venv/bin/python -c 'import openai; print(openai.__version__)'

[notice] A new release of pip is available: 25.0.1 -> 26.2.1
[notice] To update, run: python3.12 -m pip install --upgrade pip
3.15.0
~/llamay $ cat scripts/site/demo_client.py
#!/usr/bin/env python3
"""A script written against OpenAI, run against llamay.

Nothing here knows what it is talking to. `OpenAI()` reads OPENAI_BASE_URL and
OPENAI_API_KEY from the environment, which is the documented way to point the
official client at a compatible server, and llamay is one. This file is in the
repository so the demo on /docs/migrate/ can `cat` the thing it then runs.
"""

import os

from openai import OpenAI

client = OpenAI()

print("base_url:", client.base_url, "\n")

chat = client.chat.completions.create(
    model=os.environ.get("MODEL", "qwen2.5:0.5b"),
    messages=[{"role": "user", "content": "In one sentence: what is a KV cache for?"}],
    max_tokens=60,
)
print("chat:", chat.choices[0].message.content.strip())
print("usage:", chat.usage.prompt_tokens, "prompt +",
      chat.usage.completion_tokens, "completion\n")

print("stream: ", end="", flush=True)
for chunk in client.chat.completions.create(
    model=os.environ.get("MODEL", "qwen2.5:0.5b"),
    messages=[{"role": "user", "content": "Count from one to six."}],
    max_tokens=40,
    stream=True,
):
    piece = chunk.choices[0].delta.content
    if piece:
        print(piece, end="", flush=True)
print("\n")

print("models:", ", ".join(m.id for m in client.models.list().data))
~/llamay $ # one environment variable is the entire migration
~/llamay $ OPENAI_BASE_URL=http://localhost:11439/v1 OPENAI_API_KEY=unused \
>   MODEL=qwen2.5:0.5b build/demo/work/venv/bin/python scripts/site/demo_client.py
base_url: http://localhost:11439/v1/

chat: The Cache (KV) Cache is a cache used to store key-value pairs for quick access to data
.
usage: 19 prompt + 21 completion

stream: Sure, here's the sequence from 1 to 6:

1, 2, 3, 4, 5, 6

You can continue counting by following the pattern of increasing

models: all-minilm:latest, gemma2:2b, gemma3:270m, gpt2:q8_0, qwen2.5:0.5b, smollm2:135m, az
mx-code-test:latest

~/llamay $ llamay serve -m qwen2.5:0.5b -addr localhost:11439
time=2026-09-18T08:49:13.209-04:00 level=INFO msg="device graph attached" max-tokens=256
time=2026-09-18T08:49:13.210-04:00 level=INFO msg="llamay serving" addr=localhost:11439 mode
l=qwen2.5:0.5b arch=qwen2 max-loaded=2 keepalive=5m0s params="0.36 GB" context=32768 threads
=7 kv=f32 lexicon=0

  OpenAI      POST http://localhost:11439/v1/chat/completions
  Anthropic   POST http://localhost:11439/v1/messages
  Embeddings  POST http://localhost:11439/v1/embeddings     (start with -embed <encoder.gguf
>)
  Rerank      POST http://localhost:11439/rerank            (start with -rerank <reranker.gg
uf>)
  Contexts    POST http://localhost:11439/v1/contexts        (then /{id}/fork?n=8, /{id}/sna
pshot)
  Models      POST http://localhost:11439/api/pull          (and /api/copy, /api/delete)
  Stats       GET  http://localhost:11439/v1/stats


~/llamay $ # the server stays up; the rest of this is a second terminal
~/llamay $ # OpenAI
~/llamay $ curl -s http://localhost:11439/v1/chat/completions -H 'content-type: application/
json' \
>   -d '{"model":"qwen2.5:0.5b","messages":[{"role":"user","content":"Say hi."}]}' | jq -r '
.choices[0].message.content'
Hello! How can I assist you today?
~/llamay $ # Anthropic
~/llamay $ curl -s http://localhost:11439/v1/messages -H 'content-type: application/json' \
>   -d '{"model":"qwen2.5:0.5b","max_tokens":40,"messages":[{"role":"user","content":"Say hi
."}]}' | jq -r '.content[0].text'
Hello! How can I assist you today?
~/llamay $ # Ollama
~/llamay $ curl -s http://localhost:11439/api/chat -H 'content-type: application/json' \
>   -d '{"model":"qwen2.5:0.5b","stream":false,"messages":[{"role":"user","content":"Say hi.
"}]}' | jq -r '.message.content'
Hello! How can I assist you today?
~/llamay $ curl -s http://localhost:11439/api/tags | jq '.models[].name'
"all-minilm:latest"
"gemma2:2b"
"gemma3:270m"
"gpt2:q8_0"
"qwen2.5:0.5b"
"smollm2:135m"
"azmx-code-test:latest"