Your Agent Calls the Wrong Tool, Gets the Wrong Result, and Retries Forever — ToolMisuseBench Quantifies the Damage

Agentic systems fail for operational reasons even when their language understanding is strong. The common failure modes are familiar to anyone who's run coding agents in production: invalid arguments, interface drift between training and deployment, weak error recovery, and inefficient retry loops that burn budget without making progress. What's been missing is a controlled, reproducible benchmark to measure these failures systematically. ToolMisuseBench addresses this gap with an offline deterministic benchmark covering CRUD, retrieval, file, and scheduling tool environments. It uses reproducible fault injection to track success rate, invalid call behavior, policy violations, recovery quality, and efficiency under hard budget constraints—including step, call, and retry limits. The 6,800-task public dataset enables fair comparison across agent architectures. The baseline results reveal important insights: schema-aware methods improve fault-specific recovery, but overall success under hard failure conditions remains limited across all tested approaches. The retry budget results are particularly concerning—agents that lack explicit retry policies consistently exhaust step budgets without recovering, creating scenarios where agents spin uselessly while accumulating costs. This matters because most production agentic coding pipelines have no systematic way to evaluate tool misuse and recovery behavior. Teams typically discover these failures in production rather than during evaluation, which leads to unexpected downtime and wasted resources. ToolMisuseBench's deterministic fault injection approach provides a way to test coding agents against their specific MCP tool configurations before deployment: inject known faults, measure recovery quality, and identify which tool interfaces need schema hardening. For teams building agent pipelines, the benchmark offers a way to stress-test their error handling mechanisms. Can your agent recover from invalid API responses? Does it handle rate limiting gracefully? Does it have sensible retry policies that prevent budget exhaustion? These operational concerns separate working systems from fragile ones, yet they're often overlooked in favor of language capability benchmarks. The findings also highlight a broader principle: building reliable agentic systems requires paying attention to failure modes, not just success scenarios. An agent that can generate perfect code but crashes when faced with unexpected tool responses isn't production-ready. ToolMisuseBench pushes the field toward more comprehensive evaluation where reliability is measured as rigorously as capability. Read more about ToolMisuseBench and its approach to quantifying tool misuse damage at the original source.