xAI’s Context Compaction and WebSocket Mode Make Agent Costs a Runtime Problem

The least glamorous part of AI agents is becoming the part that decides whether they survive contact with production: context management. xAI’s newly refreshed developer docs add two primitives for the Responses API — Context Compaction and WebSocket Mode — that are easy to mistake for plumbing. They are plumbing. That is the point. The expensive failures in agent systems rarely come from a single bad answer anymore; they come from long loops, growing context, repeated tool output, hidden retries, and a bill that looks like someone accidentally trained a small model in the background.

Context Compaction gives developers a way to collapse a long conversation into a single opaque compaction item. WebSocket Mode keeps a response chain alive over one long-lived connection so follow-up turns can send only new input plus a previous_response_id. Together, they are xAI’s answer to a very practical problem: agent workloads are not normal chat workloads. They are stateful, repetitive, tool-heavy, and brutally sensitive to token accounting.

The token landfill finally gets a garbage collector

The Context Compaction API is exposed at POST /v1/responses/compact. Developers send the conversation they want compacted, and xAI returns a response.compaction object containing one output item of type compaction. That item includes an encrypted_content blob, which xAI explicitly tells developers to treat as opaque: do not parse it, modify it, prune it, or reorder it. Store it, pass it back verbatim, and append new turns after it.

The docs say compaction preserves salient state including system prompts, attached files, prior reasoning, and a compressed record of previous turns while dropping verbose tool output and back-and-forth. The sample response shows the shape of the economics: input_tokens: 12000, output_tokens: 800, reasoning_tokens: 240, total_tokens: 12800, and dropped_message_count: 45. That example is not a benchmark, but it is a clear product signal. xAI wants developers thinking about conversations as state that can be checkpointed, not endless transcripts that must be replayed forever.

There is an important caveat buried in the docs: compaction cannot rescue a request that is already over the context limit. The conversation must still fit before compaction runs. That makes compaction less like an emergency parachute and more like routine maintenance. Production systems should compact before the cliff, not after the model starts throwing context_length_exceeded.

The opaque design is both smart and limiting. It prevents developers from “cleaning up” a summary and accidentally corrupting state the model needs later. It also means teams are trusting xAI’s compaction process to decide what matters. For serious agent deployments, that should trigger test coverage. Build fixtures where critical details appear early, after noisy tool output, inside attached files, and inside corrections from the user. Compact the conversation and verify that the agent still behaves correctly. Treat the blob like a lossy checkpoint with vendor-managed internals, not a magic memory crystal.

WebSockets make agent loops less wasteful

WebSocket Mode attacks the other side of the agent tax. Instead of opening a fresh HTTP request for every turn, clients connect to wss://api.x.ai/v1/responses and send response.create messages over a single socket. After the first response, follow-up turns include only the new input items and previous_response_id. The server keeps prior state in memory on the open connection.

xAI says this works with Zero Data Retention and store=false because the continuation state can live in the connection cache rather than persistent storage. That is a meaningful enterprise detail, not a footnote. Many teams want lower latency and less repeated context transfer, but they do not want every intermediate turn persisted in a vendor system. The WebSocket path offers a compromise: hot state for the live chain, no storage fallback if you choose not to persist.

The docs claim internal benchmarks show up to roughly 20% lower end-to-end latency for agentic workloads with many tool calls compared with repeated HTTP requests using the same previous_response_id chain. “Up to” numbers deserve the usual adult supervision, but the direction makes sense. Coding agents, research agents, and orchestration loops do not do one clean request and stop. They ask, inspect, call a tool, receive output, reason again, call another tool, retry, and eventually produce something useful. Avoiding repeated connection setup and state rehydration is exactly the kind of boring optimization that compounds.

The constraints matter. A single WebSocket connection processes turns serially; it queues rather than multiplexes. Parallel branches require multiple connections. A connection can stay open for up to 25 minutes, after which the server closes it. If a previous_response_id falls out of the connection cache, store=true may allow rehydration from persisted state, but store=false or ZDR has no fallback and returns previous_response_not_found. A failed turn can also evict state so a retry does not continue from a broken chain.

That is not a dealbreaker. It is an architecture contract. If you are building on this, response IDs belong in your state machine. Socket age should be a metric. Reconnect handling should be boring code, not a TODO. If your agent can run longer than 25 minutes, compact periodically, persist whatever your policy allows, and design a full-context recovery path for the cases where the hot chain disappears.

Cost control is now an agent-runtime feature

The pricing page makes the operational story sharper. xAI currently lists grok-4.3 at $1.25 per million input tokens and $2.50 per million output tokens. Server-side tools are priced separately: Web Search, X Search, and Code Execution at $5 per thousand calls; file attachments at $10 per thousand calls; Collections Search at $2.50 per thousand calls. The tool usage docs distinguish attempted tool_calls from successful, billable server_side_tool_usage, and they warn that max_turns limits agent-loop turns, not individual tool calls.

That distinction is where budgets get wrecked. A single user request can trigger several model passes, multiple searches, code execution, image or file analysis, retries after failed calls, and a final response that looks deceptively small. The completion may be cheap; the trajectory is not. Traditional chat billing dashboards are not enough for agents because the cost driver is the loop, not the final message.

Teams should respond by defining context budgets the same way they define CPU and memory budgets. Set compaction thresholds by workflow: every N turns, after a token threshold, after large tool outputs, before a long-running phase, or before a socket approaches its 25-minute limit. Log input_tokens, output_tokens, reasoning_tokens, dropped_message_count, cached-token counts where available, tool attempts, billable tool usage, retry reasons, socket age, and response-chain IDs. If that sounds like observability work, good. That is what production agents require.

There is also a product-management implication. Prompt caching, context compaction, WebSocket state, per-tool billing, and rate limits are becoming part of model selection. OpenAI’s prompt-caching documentation, for example, says exact prefix caching can cut latency by up to 80% and input token costs by up to 90%. xAI is not alone in realizing that model intelligence is only one half of the agent platform. The other half is whether the runtime can keep long jobs affordable, inspectable, and recoverable.

The right practitioner takeaway is not “turn on compaction and call it done.” It is to make context lifecycle a first-class design surface. Decide what must be remembered, what can be discarded, what must be auditable, and what can live only in hot connection state. Test compaction against your real failure modes. Use WebSockets for sequential agent loops, not as a dumping ground for parallel work. Build reconnect and fallback paths before users hit them. Put cost ceilings around tool use, not just model tokens.

This update is not flashy, which is exactly why it matters. The next phase of agent competition will not be won only by models that score higher on coding benchmarks. It will be won by platforms that make long-running agent work economically and operationally sane. xAI just documented more of that layer. Now developers have to do the less fun part: wire it into systems that can survive their own ambition.

Sources: xAI Context Compaction docs, xAI WebSocket Mode docs, xAI Pricing, xAI Tool Usage Details, OpenAI Prompt Caching docs