Source Document

Zheng Jiang, Fang Niu, Bo Peng, Houde Qian, Wenyi Li, Deyao Hong, Mingju Gao, Yizhe Chi, Youjie Zheng, Kaisen Yang, Calvin Xiao, Lifeng Sun, Huan-ang Gao, "WorkflowArena: Benchmarking Executable Workflow Orchestration in LLM-Based Multi-Agent Systems", arXiv:2611.12856 [cs.CL], submitted 2026-11-09, DOI 10.48550/arXiv.2611.12856. Affiliations: Navers Lab, Einsia.AI, Tsinghua University.

This is an arXiv preprint that has not been peer reviewed. The Acknowledgements state the project "is funded and supported by Navers Lab, Einsia.AI," and all 13 authors are affiliated with that same lab or Tsinghua University — no independent external funding source is disclosed. That said, none of the nine compared models (Claude Opus 5.5, Sonnet 5.5, GPT-6 Astra, Sol, Luna, Kimi K3.1 Pro, GLM-5.3 Flash, DeepSeek V4.1 Flash, Doubao-Seed-Evolving) was built by the authors' own lab, so the design doesn't appear built to flatter a particular product. The full text was checked against a session snapshot captured 2026-11-10T22:08:19Z (UTC) — direct WebFetch access stayed blocked by a session egress proxy fault across three retries spaced ten minutes apart.

Study Overview

The question is direct: recent agent harnesses let the model decide, at run time, who does what and in what order — the workflow structure itself. Can this "orchestration skill" be measured separately from the ability to actually finish the task? The authors formalize this with a benchmark that never prescribes structure in advance and scores only the integrated final deliverable through a hidden verifier.

WorkflowArena consists of 145 long-horizon tasks across 8 professional domains (software engineering, operations, science, mathematics, finance, law, content, and 3D). Each task pairs an initial workspace with an interdependent integrated deliverable and a hidden scoring rule. Nine models were evaluated across 25 model-effort configurations, with the Claude family run in Claude Code and the GPT family in Codex. A separate "workflow transfer" experiment holds the executor fixed while swapping the model that authors the workflow, isolating orchestration quality from raw task-completion capability.

Key Results

Even the best configuration (Claude Opus 5.5, Claude Code, max effort) resolved only 36.95% of tasks on average across the 8 domains (Table 1 in the source). The spread across domains is wide: MATH nearly saturated at a best of 95.83%, while LAW and FIN never exceeded 9.09% and 18.75% respectively. No single configuration led in every domain.

ModelHarnessEffortOverall resolved
Claude Opus 5.5Claude Codemax36.95%
Claude Opus 5.5Claude Codehigh34.11%
Claude Sonnet 5.5Claude Codemax33.21%
GPT-6 AstraCodexxhigh30.80%
GPT-6.1 SolCodexxhigh29.50%
Kimi K3.1 ProClaude Codemax16.48%
GLM-5.3 FlashClaude Codemax4.43%

Source: Table 1 in the paper (selected high and low rows from 25 configurations; Overall is the 8-domain average).

Raising effort helped different model families unevenly. Opus ranged from 22.02% to 36.95% and Sonnet from 16.88% to 33.21% across effort settings, while Astra moved only between 27.34% and 30.80% and Sol between 25.63% and 29.50% — the same effort lever was a much bigger gain for the Claude family.

Switching to resource and cost reorders the comparison again. Opus (high) reached 34.11% at a mean of 7.36M tokens and $6.66 per task, while Astra (xhigh) reached a lower 30.80% at $29.26 — 4.4x Opus's cost for 3.31 points less. Conversely, Sol (low) resolved 27.59% for just $3.61.

ConfigurationOverall resolvedAvg. cost (USD/task)
GPT-6.1 Sol (Codex, low)27.59%$3.61
Claude Opus 5.5 (Claude Code, high)34.11%$6.66
GPT-6 Astra (Codex, xhigh)30.80%$29.26

Source: §4.3 (Figure 4) in the paper; cost estimated from main-agent and worker token usage.

Delegation patterns also diverged sharply by harness (§4.4 in the paper). At high effort, Claude Code's Opus and Sonnet average only 0.86 and 0.76 workflow calls per task but recruit 5.55 and 6.14 workers per call when they do delegate. Codex's Astra and Sol call more often — 2.09 and 1.94 times — but recruit only 1.72 and 1.69 workers. Among tasks with at least one call, a single call accounts for 70% of Opus's and 68% of Sonnet's executions versus about 32% for Astra and Sol; conversely, large delegations of nine or more workers make up 24% of Opus's and 30% of Sonnet's invocations, while roughly 99% of Astra's and Sol's workflows use four workers or fewer.

The most counterintuitive result is the workflow transfer experiment (Table 2 in the paper, a 32-task subset, continuous scores). Handed Opus's authored workflows, all four execution models gained an average of +6.41 points in static mode and +6.27 in dynamic mode — Kimi climbed from 35.75 to 46.61, DeepSeek from 18.37 to 25.24. Yet Astra, which on its own (max effort, 27.34%) is far stronger than the best of the four executors (Kimi, at 16.48%), produced workflows that averaged −2.53 points in static mode and only +1.34 in dynamic mode — actually lowering Kimi's and DeepSeek's scores in both modes.

Execution modelSelf (dynamic)Astra workflow (static)Astra workflow (dynamic)Opus workflow (static)Opus workflow (dynamic)
Kimi K3.1 Pro35.7529.6933.8045.0446.61
GPT-6 Luna25.0824.8729.2230.3725.35
Doubao-Seed-Evolving20.3922.0724.2528.0427.46
DeepSeek V4.1 Flash18.3712.8517.6921.7925.24

Source: Table 2 in the paper. Values are continuous scores (credit partial progress), a different basis from the resolved-rate percentages above.

Credibility Assessment

Grounds for trust: all 145 tasks were applied identically across 25 configurations, and the transfer experiment separates source and execution models to isolate orchestration quality from raw task capability. Most domains (SWE, MATH, and others) use programmatic verifiers, leaving little room for grader subjectivity.

Caveats: this is a pre-peer-review preprint with a disclosed funding interest (Navers Lab, Einsia.AI). Each configuration was evaluated once per task, so no variance estimate is available, and the Claude and GPT families were each tested in only one harness (Claude Code and Codex respectively) — model effects and harness effects are confounded, a limitation the authors themselves acknowledge. Content and 3D domains were graded by an LLM judge (Gemini-3.8-Flash) with no measured agreement against human graders. Because the source model in the transfer experiment also writes the instructions given to workers, the gains can't be fully separated from task knowledge conveyed in those instructions versus pure organizational skill. No prior work directly contradicting WorkflowArena's core claim was found.

Related Research

  • Yao et al. (2024, ICLR 2025). τ-bench: A Benchmark for Tool-Agent-User Interaction in Real-World Domains — explicitly named in WorkflowArena's own text as representative of long-horizon, policy-constrained benchmarks that "score the agent as a single unit" without observing delegation structure. WorkflowArena keeps the same kind of executable environment and independent verification but records every delegation call, making the structure itself observable. Prior-work relation.
  • Zhang et al. (2024, ICLR 2025). AFlow: Automating Agentic Workflow Generation — a prior search-based approach that applies Monte Carlo Tree Search to code-represented workflow candidates and shows the resulting designs transfer across models. It addresses the same underlying question as WorkflowArena's transfer experiment (do workflows transfer between models?), but AFlow finds a fixed workflow via offline search, while WorkflowArena has the model decide its own organization live, during execution, scored on real professional tasks. Prior-work relation.
  • Cemri et al. (2025). Why Do Multi-Agent LLM Systems Fail? — a prior failure taxonomy (MAST) built from analyzing over 150 multi-agent execution traces into 14 failure modes. WorkflowArena's finding that Astra is strong solo but authors harmful workflows for other models reads as a quantitative re-confirmation of the same category of system-design failure that MAST catalogs qualitatively. Prior-work relation.

All three are directly cited and discussed in WorkflowArena's own text; their bibliographic details (authors, venue, arXiv ID) were cross-checked via WebSearch. No prior work was found that reports results directly contradicting WorkflowArena's core claim.

Reviewer's Judgement

First, the number worth remembering here isn't 36.95% but the sign flip in the transfer experiment. Astra is dramatically stronger than all four execution models when working solo, yet the workflows it authors actively hurt two of them (Kimi, DeepSeek). That's a concrete counterexample to the assumption that "the model that solves tasks best also organizes work best," and a reason to reconsider defaulting the orchestrator role to whichever model scores highest solo.

Second, it's premature to read the Claude Code vs. Codex delegation gap (fewer, larger delegations vs. more frequent, smaller ones) as a trait of the models themselves. As the authors acknowledge, each model family was tested in only one harness. Whether this difference comes from the model or from the delegation interface the harness exposes can't be separated from this paper alone, and translating it into "Claude delegates better" without a same-model, cross-harness comparison overgeneralizes.

Putting It to Work

  • Evaluate orchestrator and executor roles separately — the single strongest model isn't automatically the best "planner." Separate workflow authorship from execution and test, on your own tasks, whether a cheap executor still performs well under an expensive model's plan.
  • Tune effort settings per model, not uniformly — in this data, raising effort was a much bigger lever for the Claude family. Map cost-performance curves per model instead of applying one effort setting everywhere.
  • Track cost-adjusted resolved rate, not resolved rate alone — Astra (xhigh) cost 4.4x Opus (high) for a lower resolved rate. Picking by resolved rate alone misses this kind of reversal.
  • Instrument delegation stats in your own harness — calls per task and workers per call are diagnostic signals that vary by domain (§4.4 in the paper), and can be used for anomaly detection.
  • Isolate the harness variable in model comparisons — an observation like "this model delegates well" may be entangled with the interface design of the harness it ran in. Hold the harness fixed when swapping models.

Conclusion

WorkflowArena's contribution is putting a number on the question "can today's agents organize their own work?" The answer is: they can, but not reliably — the best configuration averaged only 36.95%, and none led in every domain. The more important finding is the separation the transfer experiment reveals: orchestration skill is a different axis from task-completion skill, a practical basis for deciding which role — "brain" or "hands" — to give a model. This is still a pre-peer-review preprint with model and harness effects confounded across the Claude and GPT families. The choice between declaring workflow structure up front versus letting the model organize it live is explored further in our piece on operating standards for MS Agent Framework's declarative workflows.

References

Ask AI about this review

The assistant has read this review and the numbers it verified. Ask anything — it answers from the text and says so when something isn't in it.

Loading the chat…