ASR 27%→5%, Utility Loss Just 1.6pp: Reviewing MAS-Guard's Multi-Agent Guardrails

Source Document

Mohammad Kachuee, Phu Mon Htut, Sheng Li, Nikolaos Pappas, Miguel Ballesteros Martinez, Yassine Benajiba, Sujith Ravi, Dan Roth (all affiliated with Oracle AI), "MAS-Guard: Unified Guardrails from Individual Agents to Multi-Agent LLM Systems", arXiv:2610.28397 [cs.CR], submitted 2026-10-20. This is a non-peer-reviewed preprint. The full text was cross-checked against a primary-source snapshot taken 2026-10-23T22:08:26Z (used in place of WebFetch because this session's egress was fully blocked).

All eight authors are affiliated with Oracle AI, and the cloud infrastructure used for evaluation (OCI Generative AI) belongs to the same company. The fact that the organization proposing the guardrail also ran its own evaluation on its own infrastructure is revisited in the credibility assessment below. No conference or journal review record was found.

Study Overview

The paper asks three questions: (RQ1) can a guardrail reduce attack success rate (ASR) without materially hurting benign-task utility; (RQ2) does this safety-utility trade-off hold across different agent backbones; and (RQ3) how much do local checks, deterministic system checks, and semantic system checks each contribute. MAS-Guard has two layers. The local A-Guard, attached to each agent's own execution loop, runs three experts — user prompt injection, tool-result prompt injection, and sensitive unintended tool calls — using a separate judge model (Gemma-4-26B-A4B-IT) distinct from the agent being evaluated. At the system level, developer-owned Agent Charters declare each agent's role, input/output boundaries, and capabilities, while a Control Plane applies deterministic checks to communication routes and message formats and semantic checks to message topics. The evaluation spans six benchmarks — ACIArena, TAMAS, AgentLeak, OmniLeak (multi-agent) and OverThink, ASB (single-agent) — using three agent backbones (GPT-5.6-Terra, Grok-4.3 as proprietary models, and the open-weight Muse-Glimmer-30B), compared against eight rival guardrails (Sandwich, Delimiter, ACI-Sentinel, LlamaFirewall, ToolSafe, AgentDoG, AgentDoG-1.5, FIDES) and a No-Guardrail baseline.

Key Results

Without any guardrail, overall ASR across the three models ranges from 27.1% to 30.9%. With MAS-Guard, it falls to 4.8% for GPT-5.6-Terra (-82.3%), 4.2% for Muse-Glimmer-30B (-86.3%), and 8.8% for Grok-4.3 (-71.5%), and all three backbones meet the utility criterion (at or above the lower bound of the No-Guardrail 95% confidence interval). Overall utility dropped by only 1.6 percentage points for GPT (50.0%→48.4%), 0.5pp for Muse (50.7%→50.2%), and 0.6pp for Grok (59.0%→58.4%).

Breaking it down by benchmark (for GPT-5.6-Terra), the largest reductions are on ASB (44.6%→13.2%), OverThink (34.5%→0.0%), and AgentLeak (23.1%→5.0%). However, the rival guardrail FIDES posts a lower ASR than MAS-Guard on ACIArena (24.0%→1.2% vs. MAS-Guard's 11.0%) and AgentLeak (23.1%→0.0% vs. 5.0%) — but FIDES's overall utility falls below the reference threshold for every one of the three backbones (flagged with a dagger in the original tables), so it does not satisfy both safety and utility at once.

The ablation for GPT-5.6-Terra (original Table 4) separates the contribution of local A-Guard checks from system-level checks. On one basis, A-Guard alone already lowers ASR from 27.1% to 7.8%; on a different basis, deterministic system checks alone barely move it, from 27.1% to 26.4%.

Source: original paper Table 3, 4, 7, 8

Credibility Assessment

This is a non-peer-reviewed arXiv preprint. All eight authors are affiliated with Oracle AI, and the cloud infrastructure used for evaluation (OCI Generative AI) belongs to the same company — the organization proposing the guardrail evaluated its own design on its own infrastructure, a clear conflict-of-interest limitation. Second, the authors state they will not release the exact guardrail-expert prompts to limit adversarial misuse, which hurts independent reproducibility. Third, the Agent Charter and Control Plane configurations were authored automatically by Codex with GPT-5.6-Sol at high reasoning effort; since the model authoring the configuration and the models being evaluated share an ecosystem, it is hard to rule out that the configuration was implicitly tuned to the benchmarks. Fourth, the authors themselves surface contrary evidence — FIDES posts a lower ASR on some benchmarks but fails the utility criterion, and conversely MAS-Guard does not achieve the lowest ASR on every benchmark either. Fifth, the authors acknowledge that ASR is used only as "an umbrella term for rate of successful attacks," not a severity-weighted measure, and that Utility was measured only on benign episodes rather than under active attack — both self-disclosed limitations.

Related Research

• Debenedetti et al. (2024). AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents — not used as a direct comparison by MAS-Guard, but the prior work that first established a dynamic environment for evaluating prompt-injection attacks and defenses (prior, NeurIPS 2024). Independently confirmed via WebSearch: it reported that then-current LLMs solved less than 66% of tasks even with no attack present — the same design principle MAS-Guard follows by tracking utility separately from safety.
• Chennabasappa et al. (2025). LlamaFirewall: An Open Source Guardrail System for Building Secure AI Agents — a rival guardrail MAS-Guard directly compares against (extension — MAS-Guard broadens the scope by combining local and system-level layers). Independently confirmed via WebSearch: released by Meta, combining a BERT-based PromptGuard 2 with a chain-of-thought auditor, AlignmentCheck. In this paper's own Table 3, LlamaFirewall's overall ASR for GPT-5.6-Terra is 22.1%, far higher than MAS-Guard's 4.8%.
• Zhang et al. (2024). Agent Security Bench (ASB): Formalizing and Benchmarking Attacks and Defenses in LLM-based Agents — the origin of one of the six benchmarks MAS-Guard actually uses (benchmark-origin, accepted at ICLR 2025). Independently confirmed via WebSearch: ASB spans 10 scenarios and over 400 tools, and the original paper reported a highest average ASR of 84.30%, concluding that existing defenses show limited effectiveness. In this paper, No-Guardrail ASR on the same ASB benchmark varies sharply by backbone — 44.6% (GPT), 70.8% (Grok), 43.4% (Muse) — showing that backbone vulnerability drives results more than benchmark difficulty alone.

All three anchor this paper's argument: AgentDojo is the root of the "track safety and utility together" design principle, LlamaFirewall is the rival guardrail MAS-Guard directly beats, and ASB is the benchmark origin actually used in the evaluation. The specific figures from AgentDojo and LlamaFirewall are each paper's own claims, not numbers this review verified against their primary sources directly, so they are kept out of the Key Results section above.

Reviewer's Take

First, I judge the most important practical signal here is the asymmetry the ablation reveals: local checks beat system checks. Deterministic system checks alone barely moved ASR (27.1%→26.4%), likely because most attacks in these benchmarks are malicious content traveling through an already-permitted route. Investing first in local, in-execution content inspection looks like the higher-ROI move compared with structural defenses like route or format checks.

Second, I think the comparison with FIDES is the most important implication this paper doesn't foreground. FIDES posts a lower ASR than MAS-Guard on ACIArena and AgentLeak but fails the overall utility criterion — meaning a "lower ASR" guardrail is not automatically a deployable one. Picking a guardrail on a single safety metric risks choosing something that wins on the benchmark and gets rolled back in production.

Third, I judge that Grok-4.3's residual ASR (8.8%) being noticeably higher than the other two models (4.2-4.8%) is also a number the authors don't foreground. It implies the same guardrail doesn't transfer evenly across backbones, so validating a guardrail on one model and porting it unchanged to another carries real risk.

Practical Applications

• Deploy local execution-boundary checks first — A-Guard alone already cut ASR from 27.1% to 7.8%. Treat system-level route/format checks as the next priority, not the first.
• Always read safety metrics alongside the utility criterion — as the FIDES case shows, optimizing for ASR alone can select a guardrail that hurts utility. If a vendor only reports ASR, ask for the utility number too.
• Re-validate a guardrail across multiple backbones — Grok-4.3's residual ASR was nearly twice the other models'. Don't port a guardrail validated on one model unchanged to another.
• Cover the OWASP Top 10 for LLM's prompt-injection and excessive-agency categories first — A-Guard's three experts target exactly these two categories, and they also produced the largest observed reductions.
• Treat a vendor's self-reported numbers conservatively until independently reproduced — this paper's own evaluation, run on the proposing organization's infrastructure with unreleased prompts, is itself an example of why.

Conclusion

MAS-Guard's contribution is a design that unifies local agent-level checks and system-level policy under one shared intervention space — allow, reflect, confirm, block — and it demonstrates a 71.5-86.3% ASR reduction across six benchmarks and three LLMs while keeping utility loss under 1.6 percentage points. But it is a pre-peer-review preprint, evaluated by the proposing organization on its own infrastructure, and the FIDES comparison shows that a "lower ASR" does not always mean a "better guardrail." Teams adopting multi-agent guardrails should prioritize local checks, track safety and utility as separate metrics, and re-validate across multiple backbones. Using attack success rate as a pre-deployment gate is the same approach taken in One in Three Gets Through: Turning Red-Team Evaluation Into a Deployment Gate for Tool-Using Agents.

References

• MAS-Guard: Unified Guardrails from Individual Agents to Multi-Agent LLM Systems — arXiv abstract (primary source)
• Full HTML text of the same paper — used for table/figure verification
• AgentDojo — arXiv:2406.13352
• LlamaFirewall — arXiv:2505.03574
• Agent Security Bench (ASB) — arXiv:2410.02644
• One in Three Gets Through: Turning Red-Team Evaluation Into a Deployment Gate for Tool-Using Agents — sunny34.com blog