Running on open models

Agents here are not written against a particular model. Everything the agent layer needs — chat, embeddings, tool calls — goes through a gateway that speaks the OpenAI-compatible HTTP shape, so the model behind it can be a hosted frontier API, an open model you run yourself, or several at once.

This page is the engineering detail behind that claim. It exists because "supports open models" is easy to say and hard to verify, and the person evaluating a platform is right to ask.

The gateway

Backends are rows in a table, scoped per tenant. Each carries a base URL, an API key, a model name, a kind (chat or embedding) and a weight. Adding or swapping one is a configuration change — no deployment, no agent rewrite.

There is no vendor SDK anywhere in the dependency tree. The gateway issues plain HTTP against the OpenAI-compatible surface, which is why anything that implements that surface — vLLM, Ollama, llama.cpp servers, most commercial APIs — works without adapter code.

Capabilities are probed, not assumed

At startup the gateway calls /v1/models on an enabled backend and reads back what that model actually offers: context window, embedding dimension, declared capabilities.

Nothing about the model is hardcoded. This matters more than it sounds: a platform that assumes a frontier-sized context window will silently overflow an open model with a 32K window, and the failure surfaces as a truncated or nonsensical answer rather than an error.

Four ways small open models break

These are not hypothetical. Each one is handled because it showed up in practice.

1. Tool calls arrive as text

Many open models ignore the structured tool_calls field and write the call into the reply body instead. Two shapes are common:

<tool_call>
  <function=search_docs>
    <parameter=query>refund policy</parameter>
  </function>
</tool_call>

<tool_call>
{"name": "search_docs", "arguments": {"query": "refund policy"}}
</tool_call>

Left unhandled, nothing executes and the end user sees raw markup in the chat. The tool loop parses both forms, dispatches the tool normally, and — as a backstop — strips any residual <tool_call> markup from the text before it is sent onward. Raw tool-call syntax must never reach an end user, even if parsing fails.

2. Streamed tool calls arrive in fragments

OpenAI-compatible servers disagree about how to split tool calls across streaming chunks. A single call can arrive as several partial deltas, sometimes interleaved when the model requests more than one tool. Fragments are accumulated by index and reassembled into whole calls before any tool runs.

3. Context budget must come from the real window

The usable input budget is derived, not configured:

input_budget =
    probed_context_window − output_reservation − safety_margin

with a floor so an aggressive reservation can never drive the budget to zero. On a 32K model this produces a very different budget than on a million-token model, which is the entire point — the same agent definition runs correctly on both.

4. Reasoning consumes the answer

On models that emit reasoning before the final answer, an output budget sized for the answer alone gets eaten by the reasoning, and the user receives a truncated reply. The output allowance is raised to account for reasoning tokens so the answer itself survives.

Routing and failover

Backends of the same kind are selected by weight, so traffic can be split — for example most requests to a small self-hosted model, the remainder to a stronger hosted one. When a request fails in a way that is retryable, the gateway tries the other eligible backends before falling back to exponential backoff against the one it has.

The practical pattern is to split by consequence rather than by difficulty: read-only and low-stakes work on the cheaper model, anything that writes or commits on the stronger one.

What this does not do

Being honest about the edges is more useful than a longer feature list:

• It does not make a small model as capable as a large one. Tool selection — deciding which tool to call and with what arguments in a messy multi-step situation — is where model quality shows most. Test your own tasks before moving high-consequence work to a smaller model.
• It does not manage your inference infrastructure. If you self-host, the GPU capacity, the model server and its uptime are yours. The gateway will route around a backend that is down; it cannot make one faster.
• It does not certify specific models. The compatibility work is about the shapes models produce, not a tested list. Any OpenAI-compatible endpoint is expected to work; which model is right for your workload is an evaluation you should run.

Related

• Bring your own model — the same subject, for the person deciding rather than integrating
• Tools and skills — what the agent can actually call
• Security — isolation, encryption and what leaves your environment