OpenClaw’s CJK Token-Precheck Bug Shows Why Agent Cost Controls Need Real Token Accounting

Token budgets are becoming one of the least glamorous pieces of AI infrastructure, which is usually how you know they matter. OpenClaw PR #95327 looks, on first pass, like an internationalization fix: Chinese-heavy tool output was being counted too aggressively, causing a Qwen-backed agent to declare defeat before the provider actually ran out of room. The better read is sharper: coding agents are now expensive enough, stateful enough, and multilingual enough that rough prompt math is no longer acceptable engineering.

The reported failure is a useful stress test because the numbers are not theoretical. A real OpenClaw session asked an agent to summarize Chinese iCenter space pages and email the result. The model was Qwen3-235B-A22B with a 128k context window. The provider-side input landed at 85,643 tokens, comfortably below the model’s advertised ceiling. But OpenClaw’s built-in precheck estimated roughly 122,000 tokens against a configured 108,000-token budget and blocked the turn with an “Auto-compaction could not recover this turn” style failure.

That is downtime created by accounting, not by model capacity.

The bug is CJK; the lesson is FinOps

The root cause is exactly the kind of constant that starts life as “good enough” and ends up operating production. OpenClaw’s generic estimator used TOOL_RESULT_CHARS_PER_TOKEN=2, while its CJK path inflated character counts by 4x inside estimateStringChars(). Put together, the precheck effectively treated Chinese text as roughly two estimated tokens per Chinese character. For Qwen’s tokenizer, the real behavior is much closer to one token per Chinese character.

That mismatch made OpenClaw pessimistic by about 2.5x for this workload. The context engine, lossless-claw, had a DB estimate of 41,638 tokens for the relevant assembled context. The core precheck saw a much scarier number and took the safe-looking path: compact, then fail when compaction did not get the estimate under budget. Safe-looking is doing work here. A false overflow is not a security vulnerability, but it is still a production reliability bug. The agent did not complete the requested task even though the model could likely have handled it.

PR #95327 changes that boundary. If a context engine provides a positive estimatedTokens value from assemble(), OpenClaw can use that estimate for the preemptive overflow check instead of the generic string heuristic. The patch is small — 64 additions, 6 deletions, 2 files changed — but the design move is important. The component closest to the final prompt shape should have a say in budget policy.

That does not mean “trust every estimate.” Codex review made the right objection: the replacement estimate must account for prompt-local runtime context, not just the context-engine-managed message bundle. If the context engine undercounts system scaffolding, tool wrappers, reserved response budget, or runtime metadata, OpenClaw could trade false failures for provider rejections or surprise spend. Better accounting only helps when the accounting boundary matches what is actually sent to the model.

What operators should test before they trust token gates

The practical takeaway for agent operators is to stop treating token estimates as constants buried in code. They are runtime telemetry. If you run agents over multilingual docs, customer tickets, scraped pages, logs, PDFs, spreadsheets, code blocks, minified JSON, markdown tables, or generated tool output, you should expect estimator skew by content class. English prose is the easy case. Production is not made of easy cases.

A useful evaluation suite should include CJK-heavy pages, mixed English/Chinese technical docs, emoji-heavy chat exports, deeply nested JSON, minified JavaScript, SQL dumps, markdown tables, logs with timestamps and UUIDs, and source code in several languages. Run the same task across the models you actually use. Compare the precheck estimate, context-engine estimate, provider-reported input tokens, compaction decisions, and final outcome. If your platform cannot tell you which part of the prompt caused pressure — retrieval, tool output, history, system instructions, attachments, or runtime wrapper — then “prompt too large” is not observability. It is a shrug with stack traces.

This also matters for cost controls. The industry likes talking about agentic AI token usage as if the hard part is the monthly bill. The harder part is making budget policy precise enough that it does not break legitimate work. Over-blocking wastes human time and makes agents look flaky. Under-blocking creates provider errors, latency spikes, and unplanned spend. The healthiest systems log estimate-vs-actual deltas and then tune by model family and content type. A Qwen long-context workflow should not inherit the same assumptions as a short-context English-only chat model.

There is a second-order product lesson here. Context engines are no longer optional niceties for “memory.” They are becoming the place where prompt assembly, compression, losslessness guarantees, retrieval strategy, and token accounting meet. If the context engine can assemble a more faithful prompt than the core runtime, it should be able to expose a more faithful budget estimate. But that API needs contract tests. The runtime should know whether the estimate includes only retrieved messages, all tool results, the active user turn, system scaffolding, and reserved output. Ambiguity here becomes either downtime or spend.

For developers building on OpenClaw, the action item is not to wait for this PR and relax. Add regression cases for your own content. If your agent summarizes Chinese support tickets, Japanese incident notes, Korean customer emails, or multilingual product docs, test those inputs explicitly. If your agent shells out to tools that produce giant tables, test that too. Token policy is part of your workload contract now.

The broader editorial point is simple: this is not “Chinese token counting got fixed.” It is evidence that AI coding-agent FinOps has moved below dashboards and into prompt assembly. A budget gate is only as good as its accounting. False overflow is downtime. False safety is surprise spend. Both are engineering failures, and neither gets solved by buying a bigger context window and hoping the math goes away.

Sources: OpenClaw PR #95327, OpenClaw v2026.6.9-beta.1 release notes