Local Attack Success 90%, Consequence Conversion 7-100%: Reviewing Agent-Canary-Mas

Source Document

Xiuming Liu, Le Gai, Zhiyuan Chang, Peiyang Li, Bin Zhu, Shuo Wang, Chao Feng, "Agent-Canary-Mas: End-to-End Evaluation of Protocol Risk in Long-Horizon Multi-Agent Systems", arXiv:2610.15529 [cs.AI], submitted 2026-10-11 (v2, revised 2026-10-14), DOI 10.48550/arXiv.2610.15529. Affiliations: Shanghai Jiao Tong University, Tsinghua University, Ant Group.

This is a preprint that has not been peer reviewed. Several of the seven co-authors are affiliated with Ant Group, and among the three evaluated protocols (A2A, ANP, BCS), BCS (Bot Coordination Service) is the coordination service Ant Group publishes as the core of its own distributed agent platform, Avernet. In other words, the authors' employer's own protocol is one of the three being graded. That said, the results are not especially favorable to BCS — its routing-authorization class (T3) actually shows the highest consequence-conversion rate of the three protocols. The paper does not include a separate conflict-of-interest or funding-disclosure statement.

Study Overview

The paper asks three questions. First, how often does a locally successful protocol attack survive later agent behavior to change the actual task outcome (RQ1, consequence conversion)? Second, can an incident be traced to the responsible agent using only the logs a protocol keeps by default (RQ2, attribution)? Third, what happens to risk and task completion when a protocol-context Guard checks authority at handoff points (RQ3, defense)?

The dataset is 975 matched malicious-benign task pairs spanning three protocols (A2A, ANP, BCS) x five business domains (Healthcare, Finance, Logistics, Research, Travel) x five risk classes (13 subclasses). Each pair shares the same goal, roles, tools and initial state, varying only one participant-controlled protocol property. The authors ran this suite with three models — MiniMax-M3, GLM-5.2, and GPT-5.6 Terra — for 2,925 model-conditioned pairs and 5,850 execution instances.

The key methodological choice is separating "local attack success" (ASR) from "downstream task-outcome change" (PTCR) within the same execution. All runs use a shared OpenClaw agent framework, swapping only the underlying model while holding roles, tools and topology fixed.

Key Findings

Task Completion Rate (TCR) stayed between 99.2% and 100.0% across all conditions, so the analysis centers on ASR (local attack success among completed pairs) and PTCR (consequence conversion among locally successful attacks). The per-protocol averages (across all five risk classes) for each model are:

At the protocol-average level, ASR and PTCR look like they move together, but once you break the data down by risk class (T1-T5), the two diverge sharply in opposite directions. Treating these as interchangeable measures of "risk" is the paper's central warning.

The paper attributes the ANP-T4 gap to the fact that ANP's DID-based authentication verifies who sent a result, not whether that result should be trusted as the basis for a business decision. Routing (T3), by contrast, is structurally hard to walk back once a request is misdirected — once it is in, the consequence usually follows.

RQ2 (attribution) results are equally stark. Using only the records a protocol retains by default (Native View), macro-averaged Audit-F1 across 15 protocol/risk-class combinations was 29.7%. Adding cross-node evidence that links selection, delivery, consumption and effect (the Agent-Canary-Mas View) raised it to 95.5% (+65.8 points). By protocol: A2A 20.0% to 100.0%, ANP 0.0% to 93.9%, BCS 69.0% to 92.6%. ANP's native logs, built around decentralized identity authentication, contain no link at all between a participant, its action and the outcome — hence 0% native attribution.

RQ3 (defense) is scoped to the Finance domain with MiniMax-M3 only (195 task pairs), adding a "protocol-context Guard" that re-checks authority at every handoff. The Guard suppresses attacks, but task completion (TCR) also drops from roughly 99.8% to 95.1% on average.

Under the Guard, BCS-T3 (routing) goes from 66.7% ASR to 0% — the attack is blocked outright. ANP-T4 (content authority) stays nearly as exploitable locally (92.9% ASR) but its PTCR drops to 0%. One defense stops entry; the other denies authority after entry — different control points, both counted as "the Guard."

Credibility Assessment

Three things support the results. First, the matched-pair design holds model, topology, tools and initial state fixed, isolating the treated protocol property. Second, the central finding — that ASR and PTCR measure genuinely different things — repeats consistently across three architecturally distinct protocols (A2A's peer-to-peer design, ANP's decentralized design, BCS's centralized design). Third, this is not a press-release summary: 975 task pairs, 5,850 executions, and a published 13-subclass risk taxonomy back it up.

The caveats are just as clear. This is a pre-peer-review preprint, and as noted above, BCS is a product of a co-author's own employer (Ant Group) — a real conflict of interest. The authors themselves flag that the semantic Judge scoring outcomes runs on the same model condition as the agents being evaluated, which can correlate errors. The defense experiment (RQ3) covers only one domain-model combination (Finance, MiniMax-M3, 195 pairs) and doesn't generalize to other combinations, and the paper does not measure the latency or token-cost overhead of deploying the Guard. The tasks themselves are controlled sandbox scenarios; "harm" is a proxy — task-local state transitions, not real financial or clinical damage — and the paper says so explicitly.

Related Work

• Li et al. (2026). A2ASecBench: A Protocol-Aware Security Benchmark for Agent-to-Agent Multi-Agent Systems (ICLR 2026) — an A2A-specific security benchmark that found attacks frequently bypass default safeguards in official A2A demos (travel, healthcare, finance). Its evaluation stays at bounded, local attack success and does not separate local success from downstream consequence the way Agent-Canary-Mas does. It is one of the benchmarks Agent-Canary-Mas directly compares against in its own Table 1.
• Du et al. (2026). ProtocolBench: Which LLM Multi-Agent Protocol to Choose? (ICML 2026) — a prior comparison of A2A, ACP, ANP and Agora on task success, latency, communication overhead and failure resilience, reportedly finding protocol choice can shift completion time by up to 36% (from search snippets; original not read). It does not cover security or consequence conversion, making it complementary rather than overlapping.
• An et al. (2026). ACIArena: Toward Unified Evaluation for Agent Cascading Injection (ACL 2026) — evaluated cascading injection across 6 MAS implementations and 1,356 test cases at the level of general multi-agent topology, concluding that topology alone is insufficient to judge MAS robustness (from search snippets; original not read). Agent-Canary-Mas can be read as re-confirming that same limitation at a finer grain — down to individual protocol objects such as identity, routing and content.

All three are benchmarks Agent-Canary-Mas directly cites and compares against in its Related Work. They agree on direction — agent-to-agent protocols are exploitable — but Agent-Canary-Mas is the only one to separate local success from task outcome and quantify the size of that gap, which runs to tens of percentage points.

Reviewer's Take

First, I think treating "attack block rate" as a single security metric needs to be reconsidered. Ranking ANP-T4 as top priority just because its ASR is in the 90s would be overreaction, given its PTCR sits in the teens; ranking ANP/BCS-T3 as low priority because its ASR looks like 50-67% would leave the actually dangerous class unmonitored, given its PTCR reaches 93-100%. A security dashboard that only shows ASR is, in itself, a warning sign.

Second, I think Guard adoption should not be read as pure risk reduction. Average completion drops from 99.8% to 95.1%, and specific items like BCS-T2 fall as low as 76.7%. If that completion cost isn't written into an operational SLA up front, field teams who can't tolerate the friction are likely to quietly turn the Guard off.

Third, I find it notable — and, paradoxically, reassuring — that BCS, a product of a co-author's own employer, comes out worst on a specific risk class (T3) among the three protocols. That's a signal the authors did not cherry-pick favorable results for their own product. That said, it does not cancel out the conflict of interest itself, so I'd treat BCS-specific numbers as provisional until independently reproduced.

Practical Takeaways

• Track ASR and PTCR separately — if you operate agent-to-agent protocols (A2A, MCP, etc.), log "was the attack blocked" and "did it actually affect the task when not blocked" as two distinct metrics. Merging them produces both ANP-T4-style overreaction and T3-style underreaction.
• Prioritize routing and authorization handoffs — every case in this data where ASR is low but PTCR spikes is a routing/delegation (T3) class. Put authority checks at the point where the next recipient or hop is decided.
• Budget the completion cost of a Guard into an SLA — a 4.7-point average drop, and up to 23 points on specific items, needs to be disclosed and thresholded in advance, or the Guard risks being disabled in the field.
• Don't investigate incidents with native logs alone — a 29.7% Audit-F1 means the logs usually can't tell you who was responsible. Design in a cross-node trace linking selection, delivery, consumption and effect from the start.
• Demand independent reproduction of vendor self-evaluations — when a protocol or platform provider publishes a benchmark that includes its own product, don't adopt the results as-is without external reproduction or at least public-data verification.

Conclusion

Agent-Canary-Mas's contribution isn't a new attack technique — it's a framework that separates "local attack success" from "actual task consequence" and measures both. Across 975 matched task pairs, ASR is high (mostly around 90%), but the share that actually converts into a task-level consequence (PTCR) swings from 7.1% to 100% depending on risk class and protocol. Cross-node evidence lifts incident-attribution accuracy from 29.7% to 95.5%, and authority checks at handoff points reduce risk at the cost of completion rate. Given this is a pre-peer-review preprint with a real conflict of interest around the BCS evaluation, treat the numbers as directional signals, and start by instrumenting your own environment to track ASR and PTCR separately rather than importing these figures wholesale. If your organization exposes agent-to-agent communication over a standard protocol, pairing this with our review of red-teaming the MCP transport layer helps map risk across both the handoff and transport layers.

References

• Agent-Canary-Mas: End-to-End Evaluation of Protocol Risk in Long-Horizon Multi-Agent Systems — arXiv abstract (source)
• Full HTML text of the same paper — used for table/number verification (session egress was blocked; cross-checked against a GitHub Actions snapshot copy, see note above)
• A2ASecBench (ICLR 2026) — related work
• ProtocolBench (ICML 2026) — related work
• Red-teaming the MCP agent transport layer — sunny34.com blog