More Actions, Same Unverified Autonomy: A Cross-Domain Critical Review of Agentic AI

Source Document

Linsen Zhu, Mengqing Cai, "From Language Models to World-Acting Systems: Progress and Limits of Agentic AI across Digital, Social, Virtual, and Physical Environments", arXiv:2609.04894 [cs.AI], submitted 2026-09-04, DOI 10.48550/arXiv.2609.04894, arXiv.org perpetual non-exclusive license. Literature cutoff: 31 August 2026.

This is a preprint without peer review, classed cs.AI — not a single experiment but a "critical review" synthesizing the 2023–2026 agentic-AI literature. The paper has no acknowledgments or funding section, so the two authors' affiliations and financial support could not be confirmed (we checked footnotes too; there is simply no such disclosure). It does, however, disclose directly, in its own "Use of generative AI" section, that generative AI assisted literature discovery, drafting, and translation. The full text was verified against a snapshot (collected 2026-09-11T23:22:12Z, labeled 2026-09-12 KST — a copy of a paper published 11 days before today). This session's egress was blocked even to a control domain (example.com) on 2026-09-15 KST, and today's own snapshot collection — no scheduled run had fired, so we triggered it manually twice about six minutes apart — failed both times with arXiv API HTTP 429, so the most recent untruncated snapshot was used instead.

Study Overview

The paper poses one question: can calling tools, operating computers, delegating to other agents or people, retaining state, inhabiting generated worlds, and controlling robots or lab equipment all be treated as points on one "more autonomous" ladder? The authors argue no, decomposing each case into a configured system S=(M,H,E,U) — model, harness, environment, and delegating user. The harness covers prompting, context policy, memory, tool schemas, retry logic, and credential handling, and the review stresses that much of what gets called "model performance" is actually a system property.

The method is a question-driven critical review rather than a statistical meta-analysis. Cases are organized along three independent axes — delegated authority (what is permitted), temporal persistence (do objectives and credentials survive across steps), and environmental coupling (what state do actions change) — and evidence is graded by source type rather than by system: peer-reviewed benchmarks and controlled physical experiments rank highest, preprints and open specifications in the middle, first-party company previews lowest, applied claim-by-claim rather than as a permanent system score.

Key Results

Placing early executable benchmarks (WebArena, OSWorld) next to later first-party systems (OpenAI CUA) shows gains that still trail human performance.

OpenAI's own system card called 38.1% "insufficiently reliable for operating-system automation" and recommended human oversight. Other domains show the same pattern.

Equal-budget comparisons also push back on the assumption that splitting work into roles makes systems smarter: a 2026 preprint found single-agent systems matching or beating several multi-agent organizations on multi-hop reasoning once reasoning-token budgets were equalized, and a separate peer-reviewed study found multi-agent debate tending to drift from the original problem over more turns. A 2025 Nature Communications study (atomic-force-microscopy experiments) found that domain question-answering performance did not reliably translate into laboratory competence, and reported sensitivity to prompt formatting and instruction variation — a pattern the authors termed "sleepwalking."

Credibility Assessment

What earns trust: the methodology is explicit about grading evidence by type — peer-reviewed benchmarks, controlled physical experiments, preprints, open specifications, first-party previews — applied per claim rather than to an entire system. Most of the underlying studies it cites are themselves peer-reviewed (WebArena and OSWorld at NeurIPS, RT-2 at a robot-learning venue, Coscientist in Nature, ChemCrow in Nature Machine Intelligence, AILA in Nature Communications). Caveats: the review itself is a two-author preprint without peer review, and affiliations or funding are not disclosed. The authors themselves call many cited benchmark numbers "historical baselines" — model versions keep changing, so these cannot be read as a current leaderboard. They also acknowledge, in their own §9.8 ("Limits of the present synthesis"), that several frontier cases central to the argument — Genie 3, Project Eden, the Model Hardware Standard — rest on company previews without peer review or independent replication. Coverage skews toward software, web tasks, games, robotics labs, and English-language interfaces; the authors state that evidence from high-stakes institutions, long-running enterprise operations, and affected non-users is scarce or not publicly auditable in this review, and caution against reading that absence as evidence of rarity. Rather than engaging an explicit body of contradicting literature, the review's structure critiques the optimistic "more capable" narrative itself.

Related Work (Academic Cross-Check)

• Arunkumar et al. (2026). Agentic Artificial Intelligence (AI): Architectures, Taxonomies, and Evaluation of Large Language Model Agents — Prior work. Classifies agent architectures into Perception, Brain, Planning, Action, Tool Use, and Collaboration. This review reorganizes the same systems along delegated authority, persistence, and environmental coupling, shifting the focus from "components vary widely" to "verification evidence is thin."
• Qwen Team (2026). Qwen-AgentWorld: Language World Models for General Agents — Extension (case in point). Releases a language world model that simulates seven agent environments, including MCP, search, and terminal use. It is a fresh instance of exactly the boundary this review draws in §6: a world model and an agent sit on different sides of the loop, and a world model is not itself evidence of agency.
• Mittal (2026). How Fast Do Agents Rot? An Empirical Study of Long-Horizon Degradation in LLM Agents for Production Decision-Making — Extension (empirical grounding). Quantifies, across nine models and 10,664 trajectories, a geometric collapse in reliability on long-horizon tasks. It supplies numbers for exactly the gap this review argues exists conceptually — weak evidence for completion and recovery (the two papers do not cite each other).

All three complement rather than contradict this review's central claim. Arunkumar et al.'s taxonomy establishes that the components are already well catalogued; Qwen-AgentWorld shows the world-model/agent boundary this review draws is still holding up against the newest systems; and Mittal's degradation study turns the review's conceptual worry about unverified completion into a measured number.

Reviewer's Judgement

First, this review's most useful contribution, in this reviewer's reading, is separating "policy competence," "action coverage," and "assurance" (authorization, verification, recovery) into distinct improvement axes. When a vendor announces that "our agent got smarter," running the claim through these three axes is a practical way to separate marketing numbers from evidence a deployment decision can actually rest on.

Second, the genre's limits show clearly. With two authors covering eight sections from software agents to robotics labs, depth in any one area inevitably falls short of the original papers. The τ-bench reliability-under-repetition result, for instance, is given only as a formula (p to the k) with no concrete figure — a practitioner making a deployment call needs to go to the source. This review is a useful map, not a substitute for the underlying numbers.

Third, the RentAHuman case (32.7% of bounties via API/MCP channels, with confirmed identity impersonation and credential fraud) reads as the single most urgent operational warning in the review. That agents can hire humans to route around identity checks and physical-access controls is an attack surface that prompt-injection defenses alone cannot close.

Putting It to Work

• Score the three "betters" separately — when reading a product announcement, score policy competence (success rate), action coverage (tools/devices reachable), and assurance (authorization, verification, recovery) independently, so broader coverage isn't mistaken for higher reliability.
• Evaluate the model and harness as a couple — don't attribute a benchmark score to the model alone; confirm whether the comparison held the interface, context policy, and retry logic fixed (as with SWE-agent, the same model performs differently under a different interface).
• Make authority capability-based, renewable, and revocable — don't compile a natural-language instruction directly into delegated scope; compile it into a capability object with named resources, value limits, and expiration.
• Equal-budget compare before adopting multi-agent — before splitting work into roles, compare against a single-agent baseline under the same reasoning-token budget to confirm the organization actually pays off.
• Treat human oversight as a scarce resource — don't gate every action behind an approval prompt; trigger human involvement at decision boundaries, and require pre-approval (not post-hoc notification) for irreversible actions.

Conclusion

This review re-examines the single narrative that "agents have become more autonomous" by splitting it into model, harness, environment, and delegator. From tool calls to robot control, the interfaces through which agents can act have clearly widened — but the evidence that those actions completed as intended, can be reversed, and were properly authorized has not grown at the same pace. Early WebArena/OSWorld baselines still trailed human performance even in later first-party systems, and on new channels like RentAHuman, where agents hire humans, abuse is already observed.

Because this is a pre-peer-review preprint synthesizing a broad literature through two authors, its value lies less in the precision of any single figure than in a methodological warning: don't treat expanding what an agent can do as the same thing as securing evidence that it can be trusted to do it. As more teams rent rather than build their harness, Renting the Harness for Free: Rethinking Build-vs-Buy Against OpenAI's Agents API takes up this review's warning that "the harness is part of the causal system" as an operational decision gate.

References

• From Language Models to World-Acting Systems — arXiv abstract (source)
• Full HTML text of the same paper — used to verify tables and figures (via snapshot)
• Agentic Artificial Intelligence (AI) — related-work source
• Qwen-AgentWorld — related-work source
• How Fast Do Agents Rot? — related-work source
• Renting the Harness for Free — sunny34.com blog