Why Static Test Suites Aren't Enough for Red-Teaming
An agent that calls tools can be manipulated by nothing more than text embedded in a tool's response — indirect prompt injection. Existing security evaluations have leaned on hand-built environments and fixed injection points, which means the test suite has to be rebuilt by hand every time a new tool or environment is added. The ToolHazard paper removes that bottleneck: an Environment Simulator, an Attacker Agent, and a User Simulator together synthesize executable, stateful environments automatically, and generate the injection points and payloads along with them.
28 Environments, 512 Tools, and a 37% Attack Success Rate
The resulting ToolHazard-Bench spans 28 stateful environments, 512 tools, and 87 long-horizon tasks. Against an agent with no alignment training, the attack success rate (ASR) came in at 37.19%, and that rate shifted depending on injection timing and placement. The more notable finding is the side effect: attacks didn't just induce risky behavior — they consistently degraded the agent's ability to complete normal tasks across environments.
From Design to Operations: A Roadmap and Pitfalls for a Tool-Agent Red-Team Gate
Declare your pre-deployment bar as a number before you build anything else. Start from the pre-alignment baseline of 37.19% ASR, and set the gate's pass threshold at 27% ASR or below — a margin above the post-alignment observed value of 26.92%. Pair that with a second axis: normal task-completion rate must not drop by more than 5 percentage points. Lowering attack success while wrecking real task performance is a gate that passes and a product that fails.
Set coverage requirements as numbers too. Fix a minimum environment count, tool count, and task count in advance — say, 20 environments and 300+ tools — so the eval suite doesn't quietly thin out as new features ship faster than the suite grows. Declare this in code and every new tool triggers an automatic coverage check on deployment.
The failure pattern doesn't reduce to a single injection location. The paper's own observation is that a payload planted early in a tool response behaves differently from one planted late, and a single-shot task differs from a long-horizon one in how the agent responds. So the eval suite needs to treat timing and placement as axes and cover the combinations evenly — repeating one or two injection patterns systematically underestimates the real attack surface.
The recovery strategy is to feed that same diversity back into alignment data. Retraining on ToolHazard-generated alignment data drops ASR from 37.19% to 26.92%, and the data was effective even when built from just three attack strategies. But confirming the improvement against a single self-built benchmark risks overfitting to that benchmark specifically, so cross-benchmark validation against an external suite has to be part of the recovery procedure, not an afterthought.
Put that cross-validation on the operational checklist as its own line item. Confirm the same alignment data holds up on AgentDojo too — four domains (banking, Slack, travel, workspace), 97 tasks, 629 security test cases — while preserving benign task utility, not just on ToolHazard-Bench. Your log schema needs injection type, insertion timing, environment ID, tool ID, attack success/failure, and task-completion-rate delta as mandatory fields, so you can trace after the fact exactly which combination broke the defense.
Record ASR and the task-completion delta on the same table every release and track it as a trend. Don't let environment and tool coverage sit at 28 and 512 — expand the seed domains alongside every new tool that ships. If a release shows a widening gap between your own benchmark's defense and an external one like AgentDojo, read that as a sign the release's alignment data overfit to your own benchmark, and reprioritize retraining accordingly.
Takeaways at a Glance
Deploy a tool-using agent without alignment and it gets breached at a 37.19% attack success rate. The core of turning this into a working gate is defining the bar on two axes — ASR at or below 27%, task-completion drop within 5 points — building an eval suite that evenly covers injection timing and placement, and watching for overfitting by running your own benchmark alongside an external one like AgentDojo.