Trained Only on Normal Workflows, 0.89% False Positives: Reviewing Skynet

Source Document

Chaoyu Zhang, Hexuan Yu, Heng Jin, Shanghao Shi, Ning Zhang, Yi Shi, Yulia R. Gel, Y. Thomas Hou, Wenjing Lou, "Skynet: Workflow-Level Anomaly Detection for Agentic AI via Semantic and Structural Modeling", arXiv:2609.06835 [cs.CR], submitted 2026-09-06, DOI 10.48550/arXiv.2609.06835. Accepted at ACM MobiHoc 2026 (Nov 23–26, 2026, Tokyo). Affiliation: Virginia Tech, Washington University in St. Louis (WashU).

ACM MobiHoc is a leading venue in mobile networking and computing, and this paper is an accepted, peer-reviewed submission (the formal presentation is scheduled for November 2026). Funding is disclosed in the acknowledgements: two Office of Naval Research grants, six National Science Foundation grants, and the Virginia Commonwealth Cyber Initiative (CCI). All nine authors are academics (Virginia Tech, WashU); no corporate sponsor or industry affiliation is disclosed, so no clear conflict of interest was found. This review checks the full text against a snapshot captured 2026-09-09T22:02:39Z (a copy of the primary-source HTML), because this session's outbound network access was blocked entirely — including the unrelated control target example.com, confirmed across three retries 10 minutes apart — making direct WebFetch impossible.

Study Overview

The research question is: at what granularity should anomalies in agentic AI workflows be detected? Existing defenses inspect individual prompts, actions, or trace fragments in isolation; the authors argue this misses how a single compromised step propagates through the dependency structure of a workflow. Skynet turns planning steps, tool calls (including MCP), inter-agent messages, and system events into a directed workflow graph, then trains dual semantic and structural encoders via contrastive learning using only benign workflows. Never seeing an attack or failure during training, it flags any execution that falls off the benign manifold under one decision rule — a zero-day-oriented design — and represents the diversity of benign behavior with a K=150-mode k-means bank.

Evaluation draws on three benchmarks independently released by different groups: Agent-SafetyBench (111 adversarial trajectories), AgentErrorBench (200 intrinsic-failure trajectories across AlfWorld, GAIA, and WebShop), and ATBench (497 broad safety-in-the-wild trajectories across 8 risk sources and 14 failure modes) — combined into 808 anomaly cases (111+200+497=808, arithmetic checks out). Benign training data comes from ETO trajectories on AlfWorld, SciWorld, and WebShop, split 4,494/562/562 for train/validation/test. Six baselines are compared: text-only monitoring (GuardAgent), provenance-level defenses (TraceAegis, Guardian), and graph anomaly detectors (F-GAE, CoLA, TopoGCL).

Key Results

On the Combined scenario (808 anomalies plus 562 benign test cases), Skynet delivers the most balanced performance among the six detectors at a single fixed threshold. Precision, recall, and false-positive rate (FPR) trade off differently by method, so reading any one number alone is misleading.

Rankings flip on any single benchmark. Guardian hits 100% recall at 0.18% FPR on Agent-SafetyBench and AgentErrorBench — a lower FPR than Skynet's 0.89% — but collapses to 18.31% recall (F1 30.90) on ATBench and falls to 49.75% on Combined: a detector that works only when the threat distribution is narrow. Skynet's own subclass recall is uneven too: within ATBench, indirect prompt injection sits at 74.03% recall and inherent agent failures at 75.56% — precisely the categories that most resemble benign execution.

The component ablation uses the same Combined data but is a separate run, so its numbers are not directly comparable to the table above — the Full Skynet baseline here reads 96.64% Acc / 0.71% FPR, not the 93.36% / 0.89% in the Combined row above.

The biggest loss comes from collapsing the K=150 benign modes into a single reference (recall drops roughly a third, FPR barely moves). Removing counterfactual views does the opposite: recall loss is small, but FPR jumps roughly 1.5×, exceeding the 1% budget. Recall, in other words, depends on multi-mode scoring; FPR control depends on counterfactual training and workflow-aware augmentation. Latency stays workable even for the largest workflows (30–121 nodes): mean 0.139s, P95 0.311s. Online per-step scoring runs 4.8ms mean at step 1, climbs to a P95 of 0.573s around step 20, then falls back to a 48.8ms mean by step 80 as graph-reconstruction cost amortizes.

Credibility Assessment

Three things earn trust. The three anomaly benchmarks were released independently by different groups, limiting co-design bias between training and evaluation. A single fixed threshold is applied across all four scenarios rather than tuned per benchmark. And the ablation isolates which component drives recall versus which controls false positives.

The caveats are just as real. The six baselines each tune their own threshold via a "balance-oriented validation rule," while Skynet uses a fixed quantile — so the head-to-head in Table 1 is not perfectly apples-to-apples (the paper itself acknowledges this by also reporting threshold-independent ROC curves). Benign training data comes from ETO-generated trajectories rather than real production traffic, so distribution shift at deployment is untested. All three anomaly benchmarks are static red-team data; none evaluate an adaptive adversary who knows Skynet's detection scheme and tries to evade it. In the broader prompt-injection detection literature, attacks specifically engineered to evade a detector have driven detection rates to 0% (the DataFlip line of work) — useful background evidence that workflow-level detection is not a complete answer.

Reviewer's Judgement

I judge the most operationally important result here to be Guardian's collapse pattern, not Skynet's top F1 score. Precision and recall tuned to a narrow benchmark can fall apart the moment the threat distribution widens, so adopting an anomaly detector based on single-benchmark performance alone is a risky call.

I also read the ablation's most damaging finding — collapsing multi-mode scoring to a single reference — as evidence that this approach's core value lies not in the detection algorithm but in accurately modeling the diversity of normal behavior. For an organization with a uniform pipeline, K=150 may be overkill; for a highly heterogeneous production environment, it may be too small. K likely needs re-tuning per domain.

Putting It to Work

• Log workflows in a reconstructible form — capture MCP tool calls, inter-agent messages, and system events so they can be rebuilt as a directed graph.
• Validate detectors on combined scenarios — a detector tuned to one benchmark, like Guardian here, can collapse once the threat distribution broadens.
• Tune the mode count (K) per domain — start low for a uniform pipeline, higher for a heterogeneous one, and watch how recall responds.
• Budget for mid-step latency — online scoring's worst-case P95 (~0.57s) lands in the middle of a workflow's steps, not at the start or end; account for that in SLAs.
• Pair with content-level defenses — indirect prompt injection and intrinsic failures are exactly where workflow-level detection recalls least; cover them with separate content checks.

Conclusion

Trained only on benign workflows, never on an attack or a failure, Skynet reached 89.36% recall at 0.89% FPR under one fixed threshold across 808 combined anomaly cases from three independently released benchmarks. That result holds on static red-team data; it says nothing yet about adaptive evasion or real production distributions. Recall drops precisely where threats resemble benign execution, such as indirect prompt injection, so re-measuring FPR on your own traffic before adoption — and pairing it with content-level defenses — is the safer path. For how to diagnose and attribute agent failures operationally once they occur, see Agent Failure Attribution Pipeline: Operating Standards.

References

• Skynet: Workflow-Level Anomaly Detection for Agentic AI via Semantic and Structural Modeling — arXiv abstract (source)
• Full HTML text of the same paper — used to verify Tables 1–4
• Agent Failure Attribution Pipeline: Operating Standards — sunny34.com blog