Capability Gap Closes, Security Gap Doesn't: Reviewing GatewayBench on MCP Gateways

Source Document

Yitian Zheng (Tsinghua University), Zhaorun Chen (University of Chicago), Haoyu Li, Bo Li (University of Illinois Urbana-Champaign), "GatewayBench: Unified Capability and Security Evaluation Platform for MCP Gateways", arXiv:2610.23449 [cs.AI], submitted 2026-10-16, DOI 10.48550/arXiv.2610.23449.

This is a preprint without peer review. All authors are academic researchers at Tsinghua, the University of Chicago, and UIUC, and none is affiliated with any of the seven gateways under test (AWS AgentCore, Agentgateway, LiteLLM, Lunar MCPX, Enkrypt, Strata, Bifrost) — so no vendor conflict of interest is evident. The paper discloses no funding source, and the main experiment (450 DTap-based tasks) is explicitly run as "a single run due to time and cost constraints," so there is no repeated-trial variance to check against.

Study Overview

The paper asks two questions: how much does MCP gateway design (flattened vs. hierarchical) affect an agent's task success, cost, and security, and how does that effect shift with the underlying model and agent harness? Flattened gateways expose the full tool list from every connected MCP server on every turn; hierarchical gateways expose only tool-discovery meta-tools, letting the agent query for what it needs on demand.

The evaluation dataset combines 450 tasks from the DecodingTrust-Agent Platform (DTap) across CRM, finance, and travel (150 benign, 150 direct-attack, 150 indirect-attack) with 74 execution tasks from MCP-Universe covering repository management, browser automation, and location navigation. To emulate real deployments, the authors inject 0, 500, 1,000, or 1,500 task-irrelevant "noisy tools" sampled from a pool of 11,118 tools collected from 251 authentic MCP servers. Noisy tools run on a mock server that returns a standard error on any call, avoiding real side effects, while the actual task tools are verified against real execution-based checkpoints from DTap and MCP-Universe.

Key Results

With no noisy tools injected (Table 2), hierarchical gateways post a markedly lower indirect attack success rate (ASRind) than flattened gateways across all three model/harness combinations (mean 19.6–30.9% vs. 42.5–60.5%). But moving the base model from GPT-5.1 to GPT-5.4 changes the picture.

Under GPT-5.4, task success rate (TSR) converges — 87.2% flattened vs. 85.1% hierarchical — while the indirect attack success gap holds: 47.2% vs. 30.9%. Closing the capability gap by upgrading the model does not close the security gap.

Indirect attacks split into two paths: tool injection (malicious content planted in a tool itself) and environment injection (planted in data a tool returns). Almost the entire architecture gap comes from tool injection (under GPT-5.4: flattened runs 53.9% down to 45.1% as noisy tools scale, hierarchical stays near 31% regardless). On environment injection, the two architecture means stay within about 3 points of each other at every noisy-tool count — gateway design barely matters there.

Scaling the noisy-tool pool also splits the cost curves. Flattened gateways keep the full tool list in context every turn, so tokens and latency grow roughly linearly; hierarchical gateways stay nearly flat, since only retrieved tools enter context.

Task success tracks the same crossover: under GPT-5.4 with no noisy tools, flattened leads by 2.1 points (Table 3), but at 1,500 noisy tools hierarchical leads by 2.3 points. The appendix experiment on real MCP-server tools (158–358 tools) reproduces the same direction, but the authors themselves flag it as an "early, exploratory stage" covering only three gateways in a single run.

Credibility Assessment

Two points support trust here. First, the flattened/hierarchical direction holds consistently across three model/harness combinations and two independent task datasets (DTap and MCP-Universe). Second, the scale itself — 524 tasks, 11,118 real tools, 7 production gateways under controlled conditions — is the first of its kind, and outcomes are scored by rule-based checks on environment state rather than self-reported success.

The caveats are just as clear. The authors state the main 450-task experiment is "a single run due to time and cost constraints," so there is no way to tell how much of the reported gap reflects run-to-run variance. The real-MCP-server replication in the appendix covers only three gateways and 158–358 tools, a narrower range than the main experiment. Noisy tools run on a mock server returning standard errors, which does not capture real-world latency or failure patterns. And for Enkrypt, which posts the lowest ASR of any gateway, the authors explicitly warn this reflects "capability collapse" (23.3% TSR) rather than genuine robustness.

Related Work

• Debenedetti et al. (2024). AgentDojo: A Dynamic Environment to Evaluate Prompt Injection Attacks and Defenses for LLM Agents — Prior work. The leading pre-MCP benchmark for prompt injection delivered through tool outputs; GatewayBench relocates the same threat model to the MCP gateway layer.
• Wang et al. (2025). MCP-Bench: Benchmarking Tool-Using LLM Agents with Complex Real-World Tasks via MCP Servers — Benchmark origin. Evaluates agent tool-use competency over 28 real MCP servers and 250 tools but holds gateway design fixed; GatewayBench turns that fixed exposure scheme into the variable it studies.
• Wang et al. (2025). MCPTox: A Benchmark for Tool Poisoning Attack on Real-World MCP Servers — Method extension. Targets a different attack surface — malicious instructions planted in tool metadata at registration rather than in tool content — but reports attack success above 60% for o1-mini and DeepSeek-R1, putting the risk at a comparable scale.

Read together, GatewayBench's findings do not contradict these predecessors. Both AgentDojo and MCPTox establish that injection through the tool surface is a real threat; GatewayBench is the first to show that threat varies systematically with gateway architecture, refining rather than overturning the earlier picture.

Reviewer's Take

First, I'd argue the most operationally important finding here isn't the 80-plus percent task success rates but the asymmetry itself: the capability gap closes while the security gap does not. It's tempting to assume that upgrading the model to match task performance also neutralizes the gateway's effect on security — this data says the opposite. The stronger the model, the more cleanly gateway choice becomes a pure security variable.

Second, I don't think the environment-injection result should be read as a minor footnote. Switching to a hierarchical gateway does not protect against malicious instructions planted in data a tool returns. Gateway architecture is a defense against tool injection specifically; environment injection needs its own content-filtering or sanitization layer regardless of which gateway you pick.

Third, the spread among hierarchical gateways themselves (tool-injection ASR from 20% to 41%) is nearly as wide as the gap between architecture types, which means "go hierarchical" is not a safe shortcut if the specific implementation is weak. Vendor selection needs to run on per-product numbers, not the architecture label.

Practical Takeaways

• Evaluate model, harness, and gateway as one system — don't reason about gateway security in isolation; TSR for the same gateway can swing by up to 37.6 points across model/harness combinations.
• Separate your defenses by attack path — gateway architecture only moves tool-injection risk; add a dedicated content filter for environment injection.
• Report TSR alongside ASR — a low ASR alone isn't proof of safety; if TSR is also low, you may be looking at capability collapse, not robustness (the Enkrypt case).
• Budget tokens and latency against tool count — if you run a flattened gateway, plan for roughly linear token growth once the exposed tool set passes a few hundred.
• Compare vendors by measured numbers, not architecture category — implementation choices (layered discovery, semantic search, virtual filesystem) can swing ASR by more than 2x within the same "hierarchical" label.

Conclusion

GatewayBench elevates the MCP gateway from a hidden implementation detail to a measurable design choice in the agent stack. Its central finding is that closing the model-capability gap closes the task-success gap but not the indirect-attack-success gap (47.2% vs. 30.9%), and that remaining gap comes almost entirely from attacks planted in the tools themselves. Given the authors' own disclosed limits — a single run for the main experiment and a narrower real-server replication — treat the numbers as directional and validate against your own tool environment before committing to a gateway. For the operational side of narrowing tool exposure, see our guide to search-based tool calling for MCP catalogs.

References

• GatewayBench: Unified Capability and Security Evaluation Platform for MCP Gateways — arXiv abstract (source)
• Full HTML text of the same paper — used to verify tables, tokens, and latency figures
• AgentDojo (Debenedetti et al., 2024) — arXiv abstract
• MCP-Bench (Wang et al., 2025) — arXiv abstract
• MCPTox (Wang et al., 2025) — arXiv abstract
• The 30-Tool Threshold: Search-Based Tool Calling for MCP Catalogs — sunny34.com blog