Running on open models

Agents here are not written against a particular model. Everything the agent layer needs — chat, embeddings, tool calls — goes through a gateway that speaks the OpenAI-compatible HTTP shape, so the model behind it can be a hosted frontier API, an open model you run yourself, or several at once.

This page is the engineering detail behind that claim. It exists because "supports open models" is easy to say and hard to verify, and the person evaluating a platform is right to ask.

The gateway

Backends are rows in a table, scoped per tenant. Each carries a base URL, an API key, a model name, a kind (chat or embedding) and a weight. Adding or swapping one is a configuration change — no deployment, no agent rewrite.

There is no vendor SDK anywhere in the dependency tree. The gateway issues plain HTTP against the OpenAI-compatible surface, which is why anything that implements that surface — vLLM, Ollama, llama.cpp servers, most commercial APIs — works without adapter code.

Capabilities are probed, not assumed

At startup the gateway calls /v1/models on an enabled backend and reads back what that model actually offers: context window, embedding dimension, declared capabilities.

Nothing about the model is hardcoded. This matters more than it sounds: a platform that assumes a frontier-sized context window will silently overflow an open model with a 32K window, and the failure surfaces as a truncated or nonsensical answer rather than an error.

Five ways small open models break

These are not hypothetical. Each one is handled because it showed up in practice.

1. Tool calls arrive as text

Many open models ignore the structured tool_calls field and write the call into the reply body instead. Nine distinct shapes have arrived in production so far, and the count is the honest part of this section: each one was found by reading an answer that had gone out badly, never by reading code, and there is no reason to think the ninth is the last. Two are common:

<tool_call>
  <function=search_docs>
    <parameter=query>refund policy</parameter>
  </function>
</tool_call>

<tool_call>
{"name": "search_docs", "arguments": {"query": "refund policy"}}
</tool_call>

The other seven have no fence at all: a bare JSON object; a call written as one line of code, doc_update_section({"key": …}); a call: or call:namespace:tool prefix; a JSON object truncated halfway by the token limit; and — most recently — an entire reply consisting of the seven characters {{ query_knowledge_table }}.

Left unhandled, nothing executes and the end user sees raw markup in the chat. The tool loop parses these forms, dispatches the tool normally, and — as a backstop — strips any residual call syntax from the text before it is sent onward. Raw tool-call syntax must never reach an end user, even if parsing fails.

Two rules keep that backstop from becoming its own bug, and both were learned by breaking them. Parse before stripping: a call written as one line of code is executable, and an early version stripped it as debris — so the document was never edited while the model cheerfully reported that it had been. A strip that empties the reply must fall back: if removing the syntax leaves nothing, the turn is answered again rather than sent blank.

2. Streamed tool calls arrive in fragments

OpenAI-compatible servers disagree about how to split tool calls across streaming chunks. A single call can arrive as several partial deltas, sometimes interleaved when the model requests more than one tool. Fragments are accumulated by index and reassembled into whole calls before any tool runs.

3. Context budget must come from the real window

The usable input budget is derived, not configured:

input_budget =
    probed_context_window − output_reservation − safety_margin

with a floor so an aggressive reservation can never drive the budget to zero. On a 32K model this produces a very different budget than on a million-token model, which is the entire point — the same agent definition runs correctly on both.

4. Reasoning consumes the answer

On models that emit reasoning before the final answer, an output budget sized for the answer alone gets eaten by the reasoning, and the user receives a truncated reply. The output allowance is raised to account for reasoning tokens so the answer itself survives.

5. The tool that should have been called, isn't

The failures above are shape problems — the call is there, in the wrong form. This one is different: the model simply answers in prose when it should have called something, and nothing looks broken anywhere. The reply is fluent, the log is clean, and the customer gets a description of products instead of the products.

Forcing the call at the protocol level looks like the obvious answer and is a trap on self-hosted servers: where tool_choice: required is implemented as a decoding constraint, a model that did not want to call anything fights the constraint and falls into a repetition loop — measured at about one turn in four on one backend, one of which spent two minutes emitting the same fragment before anything else could happen. Upstream has open reports of the same shape.

What moved the number without that cost was giving the model less to decide: the platform works out which tool the request needs, composes the instruction itself in that tool's own wording, and shows the turn only that tool. Measured on a self-hosted 35B: 16% → 70% on the same requests, at the same latency. The full mechanism is in Tools and MCP.

Two related defences sit underneath it, because a loop can still happen for other reasons. Output that gets stuck repeating is detected mid-stream and cut, rather than at the end of the turn — by then it is already on the reader's screen, and a long enough one will freeze their browser. And a turn that was supposed to produce something but produced prose is asked once more, which recovers a good share of them.

The obvious third defence — the sampler's repetition penalties, which most self-hosted servers ship switched off — we measured and did not enable. DRY penalises repeated sequences, and a JSON tool call is repeated sequences: at the threshold these servers suggest, structured output stopped working entirely, and at gentler thresholds it still cost accuracy while preventing no loop we could observe. It also cannot reach this particular failure: where a decoding constraint has already fixed which tokens are legal, a penalty on the remaining distribution has nothing left to move.

We built a per-backend setting for it, then removed it. An unexercised switch whose only imagined use is measurably harmful is a liability rather than an option — the measurement is the thing worth keeping.

Routing and failover

Backends of the same kind are selected by weight, so traffic can be split — for example most requests to a small self-hosted model, the remainder to a stronger hosted one. When a request fails in a way that is retryable, the gateway tries the other eligible backends before falling back to exponential backoff against the one it has.

The practical pattern is to split by consequence rather than by difficulty: read-only and low-stakes work on the cheaper model, anything that writes or commits on the stronger one.

What this does not do

Being honest about the edges is more useful than a longer feature list:

• It does not make a small model as capable as a large one. Tool selection — deciding which tool to call and with what arguments in a messy multi-step situation — is where model quality shows most. Narrowing the decision (above) recovers a large part of that gap for requests the platform can recognise, and nothing for the ones it cannot. Test your own tasks before moving high-consequence work to a smaller model.
• It does not manage your inference infrastructure. If you self-host, the GPU capacity, the model server and its uptime are yours. The gateway will route around a backend that is down; it cannot make one faster.
• It does not certify specific models. The compatibility work is about the shapes models produce, not a tested list. Any OpenAI-compatible endpoint is expected to work; which model is right for your workload is an evaluation you should run.

Related

• Checks on the answer — the stage that backstops these breakages on the way out
• Bring your own model — the same subject, for the person deciding rather than integrating
• Tools and skills — what the agent can actually call
• Security — isolation, encryption and what leaves your environment