llama.cpp Is Quietly Becoming the Local Agent Runtime, One API Compatibility Fix at a Time

The most important local-AI releases increasingly look like schema chores. That sounds dismissive until an agent stack breaks because a JSON field is missing. llama.cpp release b9763 adds an item-level id field to llama-server Responses API function-call outputs. Six additions, three deletions, one changed file in the relevant PR. Small patch, large signal: local inference servers are becoming agent runtimes, and agent runtimes live or die on boring protocol compatibility.

The release was published on June 22 at 21:32 UTC. The relevant change is PR #24882, summarized in the release as “server: Add id to tool call responses api.” Before the patch, llama-server could emit tool-call output compatible enough for some clients but not strict enough for others. The reported real-world failure came from Rig 0.39 using Qwen3-Coder-Next GGUF through llama-server /v1/responses: the client failed while parsing tool calls with missing field id. After the change, Rig executed the tool call and returned the final answer.

That is the whole story if you only care about the diff. It is not the whole story if you are building local agents.

“Mostly OpenAI-compatible” is not a runtime contract

Tool calling is not just a model emitting function arguments. It is a protocol. A client receives model output, identifies function-call items, executes external tool code, correlates tool results, sends those results back, and expects the model to continue from the right state. The OpenAI Responses API shape distinguishes the item-level id from call_id. The latter correlates a function call with the tool output returned later. The former identifies the output item itself. Some clients can limp along without both. Others deserialize against the documented schema because strictness is how stateful systems avoid ambiguity.

Strict clients are not being pedantic. Agent traces can include concurrent tool calls, retries, streaming outputs, reasoning items, partial failures, and human approval steps. Identifiers become load-bearing. If the server omits one because “the model still produced the right arguments,” the breakage may appear only when a downstream framework expects the full contract. That is exactly the kind of bug that makes local agents feel flaky even when the model is fine.

This is why llama.cpp’s server work deserves more attention than the usual tokens-per-second discourse. The project is no longer only the tool people use to run a quantized model on a laptop and post a screenshot. llama-server is an OpenAI-compatible API surface used by local coding agents, private assistants, eval rigs, IDE plugins, test harnesses, and experimental control planes. Once it exposes /v1/responses, function calling, model status, routing, and service-style behavior, it becomes part of the same runtime contract ecosystem as hosted APIs.

The local stack is growing service boundaries

The broader b9763 comparison from b9754 includes nine commits and 35 changed files. Beyond the id field, the release includes a server edit-file crash fix, model status and load progress via /models/sse, router work moving model downloading to a dedicated process, input-file schema refactoring, SYCL BF16 support, and top-n-sigma sampler changes. That mix is instructive. It is not just inference optimization. It is service ergonomics, observability, compatibility, and recovery behavior.

That is where local AI is heading. The old mental model was “weights on disk plus a fast runtime.” The new model is “a local service boundary agents can depend on.” That boundary has to answer practical questions: Is the model loaded? How far along is the download? What schema does the server speak? What happens when a tool call streams? Can the router recover? Does the server crash on edit-file paths? Can the client pin behavior across versions? How closely does the local implementation follow the hosted API it claims to replace?

Local inference advocates often sell privacy, cost control, offline operation, and independence from hosted vendors. Those are real advantages. But an agent stack does not become reliable merely because it is local. Local systems still need contracts. In some ways, they need better contracts because the ecosystem is more fragmented: different clients, different GGUF conversions, different model families, different OpenAI-compatible servers, and different interpretations of what “compatible” means.

The Qwen3-Coder-Next / Rig failure is a perfect example. The model was not the issue. The local server was not useless. The client was not wrong to expect the documented field. The break sat at the boundary between “compatible enough for demos” and “compatible enough for an agent framework.” This is the seam where many local-agent stacks will fail over the next year.

Practitioners should test protocol fidelity like they test model quality

If you run local OpenAI-compatible servers behind coding agents or internal assistants, add contract tests. Not vibes tests. Contract tests. Exercise /v1/responses, function-call item shapes, call_id round trips, streaming outputs, error objects, multi-tool-turn loops, parallel tool calls if supported, cancellation behavior, and malformed tool results. Run those tests against pinned server versions before upgrading. Keep at least one strict client in the test matrix because permissive clients can hide missing fields until production finds them for you.

Also test the actual models and tools you intend to use. Tool calling with one chat template and one model family does not prove compatibility for another. A local coding stack using Qwen, DeepSeek, Llama, or a specialist code model may expose different edge cases in argument formatting, tool-call ordering, or continuation behavior. The server’s API contract should absorb that variation, not leak it into every client.

For teams comparing local agents with hosted options, this changes the buying criteria. Do not ask only whether local is cheaper or private. Ask whether the runtime exposes the same semantics your orchestration layer expects. Ask whether observability covers model status, downloads, routing, tool calls, and failures. Ask whether the project fixes schema mismatches quickly. llama.cpp’s response here is encouraging because the patch is small, direct, and aligned with the documented API shape. That is what runtime maturity looks like: unromantic fixes merged because downstream systems depend on them.

The larger lesson is that the model layer and the agent layer are collapsing into one operational surface. A local model server that supports tool calls is no longer “just inference.” It is part of the agent’s control plane. A missing field can stop a tool call. A weak error object can hide a failed mutation. A streaming mismatch can break traces. A model-status gap can make orchestration brittle. Schema fidelity is not paperwork; it is how software knows what happened.

llama.cpp’s b9763 will not be remembered as a glamorous release. It should be remembered as a useful one. The future of local agents will be decided by tokens per second, yes — but also by JSON fields, server events, routing processes, crash fixes, and the discipline to make “OpenAI-compatible” mean something testable.

Sources: llama.cpp GitHub release, PR #24882, OpenAI function-calling guide, llama.cpp repository, llama-server REST API changelog tracker