Agent Security Benchmarks Need to Measure Harm, Not Just Whether the Model Said the Bad Thing

The agent-security industry has spent too much time asking whether the model said the bad thing. SafeClawBench asks the more useful question: did the agent do the bad thing? That distinction sounds obvious until you inspect most safety evaluations, where a refusal in the final answer can make a system look safe even if a tool call, memory write, file mutation, or hidden state change already caused the damage.

SafeClawBench’s core contribution is not that it invents another prompt-injection leaderboard. It separates three things that are often collapsed into one vague “attack success rate”: semantic attack acceptance, audit-visible harm evidence, and sandbox-observed tool or state harm. For tool-using agents, those are different failure surfaces. A model can semantically comply without observable damage. It can refuse in text while still touching state. It can leave a trace that looks harmless while poisoning memory for later. Production systems care about state, not vibes.

The benchmark includes 600 controlled adversarial tasks, with 100 cases per reporting family. The six attack families are direct prompt injection, indirect prompt injection, tool-return injection, memory poisoning, memory extraction, and ambiguity-driven unsafe inference. The dataset card exposes a full benchmark_v5_600.json, executable fixtures for all 600 cases, a fast 60-case executable subset, and a 5-case smoke-test fixture. That packaging matters because agent security benchmarks should be runnable, inspectable, and boring enough to put in CI.

One safety score is not enough

SafeClawBench reports three endpoint families. CoreFail@600 measures semantic attack acceptance. HarmEvidence@600 looks for artifact-visible harm. ObservedHarm@Exec uses executable state oracles to check whether the sandbox actually changed in a harmful way. That layered design is the right abstraction for agents because the dangerous output is often not the chat message. It may be an email draft, a shell command, a database write, a memory insertion, or a tool result that should never have been trusted.

The no-defense results are uneven, as they should be. Under D0/no defense, CoreFail@600 is reported at 23.8% for GPT-5.5, 9.0% for Claude Opus 4.7, 35.5% for Qwen3.6-Plus, 27.5% for GLM-5.1, and 44.2% for Kimi K2.5. Layered prompt policy D3 reduces pooled CoreFail from 28.0% to 9.1%. That is useful. It is also not a security architecture.

The paper is careful about this, and builders should be too. Prompt policy can reduce semantic failure. It cannot replace runtime permissions, scoped credentials, memory provenance, tool mediation, sandboxing, or human approval for high-impact actions. If the model is the only thing standing between an injected webpage and a privileged tool call, the system is already badly designed.

Memory is the agent feature that doubles as an attack surface

The stress point in the benchmark is persistent memory. Under D0, memory poisoning and memory extraction each average 54.4% CoreFail@600, higher than direct prompt injection at 20.8%, indirect prompt injection at 16.0%, tool-return injection at 13.4%, and ambiguity-driven unsafe inference at 9.0%. That ordering should make every agent product team pause.

Memory is easy to sell because it makes agents feel personal, continuous, and useful. It is also a persistent database that the model can read from and write to under uncertain provenance. If an attacker can poison memory today, they may not need to win the current turn. They can shape the agent’s future assumptions. If an attacker can extract memory, the harm may not show up as an ugly final answer. It may show up as a protected preference, secret, contact detail, or internal instruction leaking through a tool path.

The engineering answer is not “turn off memory forever.” It is to stop treating memory like a cozy scrapbook. Memory needs ACLs, provenance, expiry, review, namespace separation, and audit logs. User-approved durable memory should not share the same trust level as a tool-return summary from a random webpage. Sensitive memory should not be available to every task. Writes should be attributable and reversible. If that sounds like database work, congratulations: that is because it is database work.

The scary number is the mismatch between semantics and state

The most important SafeClawBench number is not the model ranking. It is the matched analysis: in a 12,000-row comparison, 291 of 347 observed sandbox harms occurred in rows that passed the semantic check. Read that again. The evaluation that asks whether the model visibly accepted the attack would have missed most of the observed harms.

This is why final-answer moderation is insufficient for agents. A refusal can be theater. The model can say the right safety words while a prior tool call already changed state. Conversely, a semantically suspicious response may not produce harm if the runtime blocks the tool. The boundary that matters is the tool boundary. The sandbox, permission layer, and state oracle are not implementation details; they are the security model.

For practitioners, the checklist is clear. Treat external content and tool returns as tainted. Do not expose secrets to tools unless the current task requires them. Make memory writes scoped, reviewable, and revocable. Require explicit user authorization for high-impact actions: sending messages, spending money, deleting data, changing permissions, publishing content, or writing to production systems. Log tool calls and state transitions, not just chat transcripts. Run adversarial fixtures in CI, and score harm using executable oracles wherever possible.

There is also an evaluation lesson for vendors. If an agent platform claims it is safe because the model refuses prompt injection, ask how it measures state. What happens if the injected instruction arrives through a webpage, an email, a tool result, a file, or a stale memory entry? What tool permissions are available at that moment? What state changes are blocked by policy rather than politeness? A safe agent is not one that refuses nicely. It is one whose tools, memory, and sandbox remain safe under pressure.

SafeClawBench is useful because it moves the conversation from model morality plays to systems engineering. The model matters. The runtime matters more than most demos admit. If your agent can do work, your benchmark needs to measure the work it did — especially when the work was harmful.

Sources: arXiv, SafeClawBench dataset card, AgentDojo, ToolEmu, OWASP prompt injection guidance