Guides

Embeddings and reranking

Retrieval needs an encoder as much as generation needs a decoder, and a deployment that runs one engine for the answer and a Python service for the lookup has not actually been made local. Both stages run in the same process, on the same address, from the same binary.

The two stages

corpus 100,000 documents bi-encoder vectors precomputed once POST /v1/embeddings top 20 cheap, imprecise cross-encoder one forward pass per pair POST /rerank 5 Why two stages and not one An embedder never sees the query and the document together. A cross-encoder reads them concatenated — [CLS] query [SEP] document [SEP] — so bidirectional attention puts every query token beside every document token. Nothing can be precomputed, which is why it runs over twenty candidates rather than the corpus.
The reranker is where the precision is and the embedder is where the scale is; running both means the expensive one only ever sees twenty pairs.

Embeddings

llamay embed -m all-minilm.gguf -p "the capital of France"
llamay embed -m all-minilm.gguf -f sentences.txt -sim

BERT-family encoders run with bidirectional attention, WordPiece and mean pooling — -pool overrides that with mean, cls or last. Vectors come out unit-length, so a dot product is a cosine similarity.

Over HTTP: POST /v1/embeddings in OpenAI's shape, including the base64 encoding_format its official client asks for by default, and Ollama's POST /api/embed and /api/embeddings. An embedding endpoint reachable only by executing a CLI is not an endpoint; it is a subprocess.

Reranking

llamay rerank -m ms-marco-MiniLM-L6-v2.gguf \
  -q "what is the capital of France?" -f candidates.txt -top 5

Served at POST /rerank, /v1/rerank and /rerank/v1/rerank, in both shapes RAG stacks emit: Jina's and Cohere's documents/relevance_score, and text-embeddings-inference's texts/score. Whichever key the body carries decides the answer's shape.

Cross-encoders declare themselves badly

Of the three real files this was built against — bge-reranker-v2-m3, bge-reranker-base and ms-marco-MiniLM-L-6-v2 — none writes the pooling_type code that means "rank": two omit the key and one writes 1, which means mean. So llamay honours that declaration when it is there and otherwise recognises the classification head, under either of the two tensor namings in use. Scored against llama.cpp on a real file, ms-marco-MiniLM-L6-v2 agrees to 3.5e-4 of a logit.

Serving them beside a decoder

# generation and both retrieval stages on one address, in one process
llamay serve -m azmx-one-q4.gguf -embed all-minilm.gguf -rerank ms-marco.gguf

# an encoder on its own; the generation routes refuse rather than 404
llamay serve -m all-minilm.gguf

# a cross-encoder on its own; generation and /v1/embeddings both refuse, by name
llamay serve -m ms-marco-MiniLM-L6-v2.gguf

A route that exists but cannot be served refuses by name rather than answering 404. A 404 says "this server does not have that feature", which sends the caller looking for a different server; a named refusal says "this server has it and was not given an encoder", which sends them to the flag.