Source Document
Dipin Khati, Shubham Mehrotra, Bin Bi, Zhou Yu, Denys Poshyvanyk, James Zhu, Sitaram Asur, Phil L. Mui, "AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSL", arXiv:2611.13183 [cs.SE], submitted 2026-11-09, DOI 10.48550/arXiv.2611.13183. 2nd Workshop on Agentic AI Benchmarks and Applications for Enterprise Tasks (AABA4ET), a NeurIPS 2026 workshop paper. Affiliations: Salesforce (most authors), William & Mary (2 authors including the first, who did this work during a Salesforce internship).
This is not a reviewed main-track paper but a workshop paper submitted to a NeurIPS 2026 workshop (AABA4ET), and an arXiv preprint that has not been through peer review. Workshop submissions typically go through lighter review than main-track papers. There's a clear conflict of interest: the DSL under evaluation (AgentScript) and the platform it belongs to (Agentforce) are both Salesforce's own products, and most authors — including the first, who did the work during a Salesforce internship — are affiliated with that company. That said, the results don't flatter the company: none of the five evaluated models was built by Salesforce, and the best score was only 25.0% pass@3, which if anything highlights how hard its own DSL is for current LLMs. No separate funding disclosure (Acknowledgements) was found in the text. The full text was checked against a primary-source HTML copy in docs/research-authoring/snapshots/2026-11-13 (collected 2026-11-12T22:07:46Z).
Study Overview
The question splits two ways: does the near-saturation of LLM code generation on public benchmarks like HumanEval and SWE-bench carry over to an enterprise-only DSL that is essentially absent from pretraining data? And if it doesn't, what can recover it?
The authors build AgentScript-Eval around AgentScript, the declarative DSL Salesforce's Agentforce uses to define conversational agents. It draws 100 tasks from real agent-development work across three categories: authoring an agent from a specification (30), debugging a broken definition (40), and optimizing a working one (30). Grading is execution-based rather than text-similarity: a generated program is compiled, and if it compiles, it's run through multi-turn dialogue scenarios that must satisfy every expected action, state, and response (all-or-nothing per task). Five models — three frontier hosted models, one smaller hosted model, one open-weight model — were evaluated across seven generation configurations: direct generation, an agentic compile-and-edit loop, three versions of a domain skill pack, and an alternative platform CLI.
Key Results
In direct generation (zero-shot, pass@3), model scale and price did not predict the ranking.
| Model | pass@3 | Compile rate | Pass rate | Tokens/task |
|---|---|---|---|---|
| Claude Opus 4.8 | 25.0% | 38.0% | 31.3% | 9.9k |
| GPT-5.5 | 20.0% | 38.0% | 29.1% | 9.5k |
| Gemma-4 31B IT (open-weight) | 15.0% | 48.0% | 28.1% | 7.0k |
| GPT-5 mini | 14.0% | 32.0% | 20.9% | 9.2k |
| Claude Sonnet 5 | 14.0% | 21.0% | 17.4% | 8.2k |
Source: Table 1 in the paper (direct generation, zero-shot, k=3). pass@3 counts a task as solved if any of 3 executions of the same generated program clears every check; pass rate is the mean per-task fraction of checks passed (partial credit).
The 31B open-weight Gemma-4 posted a higher compile rate (48.0%) than any frontier model at the lowest token cost, and its pass rate (28.1%) sits within 3 points of the best (31.3%). Conversely, Sonnet 5 trails the smaller GPT-5 mini on both compile rate and pass rate among hosted models — defaulting to the most expensive model has weak support here.
In agentic configurations (k=1, pass@1) — where the model can compile and iterate — what mattered for the skill pack was not how it was delivered but what it contained.
| Configuration (Claude Opus 4.8) | pass@1 | Compile rate | Pass rate | Tokens/task |
|---|---|---|---|---|
| Baseline (no skills) | 33.0% | 64.0% | 46.5% | 835k |
| Skills v1 (inert docs) | 28.0% | 89.0% | 51.5% | 480k |
| Skills v2 (live plugin) | 14.0% | 86.0% | 43.6% | 389k |
| Skills v3 (v2 + rewritten authoring skill) | 37.0% | 98.0% | 61.1% | 440k |
| afcode (minimal-tool CLI) | 21.0% | 89.0% | 45.3% | 293k |
Source: bottom of Table 1 in the paper (agentic, Claude Opus 4.8, k=1). k differs from the zero-shot table's pass@3, so the two can't be compared directly (Appendix D in the paper) — only compare the "adding an agentic loop" effect within the same k=1 basis.
Moving from v1 (inert docs) to v2 (a real, executable plugin) changes delivery format and skill-set composition together, and performance actually dropped from 28.0% to 14.0%. v3 keeps v2's plugin, its 40-skill catalog, and its tool policy fixed, and rewrites only the authoring skill's guidance — shrinking its entry point from 830 to 461 lines as a progressive-disclosure restructure. That single change lifted the score to 37.0% (+23 points, the paper's best result) with a 98.0% compile rate. The authors conclude the cause was guidance density, not packaging.
But that conclusion is specific to Opus 4.8. Appendix E's Sonnet 5 results reorder the ranking.
| Configuration (Claude Sonnet 5) | pass@1 | Compile rate | Pass rate |
|---|---|---|---|
| Baseline (no skills) | 20.0% | 65.0% | 40.4% |
| Skills v1 (inert docs) | 27.0% | 79.0% | 43.7% |
| Skills v2 (live plugin) | 16.0% | 62.0% | 29.4% |
| Skills v3 (rewritten guidance) | 23.0% | 91.0% | 49.5% |
| afcode (minimal-tool CLI) | 28.0% | 79.0% | 52.7% |
Source: Table 4 (Appendix E) in the paper. The Opus row matches the table above.
By task category (Table 5 in the paper, Opus only), the sharpest gap tracks task type: authoring scored 0.0% for every model zero-shot and never exceeded 16.7% under any agentic configuration, while optimization was already 53.3% zero-shot. The likely structural cause: debugging and optimization supply an existing program as scaffold, while authoring does not.
Credibility Assessment
Grounds for trust: grading by compile-and-execute rather than text similarity avoids miscounting superficial matches as successes. The same 100 tasks and checks were applied identically across 7 configurations and 5 models, and the v2-vs-v3 comparison holds the plugin, skill catalog, and tool policy fixed while changing only one skill's wording — narrowing the causal claim to content rather than packaging. Debugging tasks were built by injecting defects of a recorded type, location, and intended fix.
Caveats: this is a workshop paper, pre-peer-review, with a conflict of interest since the DSL under test is a product of the company most authors work for. Direct generation uses pass@3 to capture some variance, but agentic configurations are evaluated once per task (pass@1) with no variance estimate, confidence interval, or significance test reported. The skill-pack experiment is designed and narrated around Opus 4.8, with Sonnet 5 relegated to a supporting appendix table — and as shown above, the two models point in different directions, a signal against generalizing the conclusion. The benchmark is small (100 tasks: 30/40/30) and limited to a single proprietary DSL, so generalization to other enterprise DSLs is unverified. No prior work was found that directly contradicts "LLMs handle public languages well" — if anything, this paper shows that premise doesn't carry over to a proprietary DSL.
Related Research
- Yu et al. (2018, EMNLP 2018). Spider: A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task — cited directly in AgentScript-Eval's own text as representative of a "public DSL well represented in pretraining data" (text-to-SQL). Spider targets a language models have some familiarity with (SQL), while AgentScript-Eval targets a proprietary DSL entirely absent from pretraining data — a contrast the paper draws explicitly. Prior-work relation.
- Jimenez et al. (2023, ICLR 2024). SWE-bench: Can Language Models Resolve Real-World GitHub Issues? — cited directly in AgentScript-Eval's own text to support the claim that frontier models "resolve a substantial share of real GitHub issues." It serves as the contrast case showing that success on public languages (Python, etc.) doesn't transfer to a proprietary enterprise DSL. Prior-work relation.
- Chen et al. (2026). SkillJuror: Measuring How Agent Skill Organization Changes Runtime Behavior — not cited in AgentScript-Eval; independently identified and confirmed via WebSearch (direct WebFetch access to the full text stayed blocked by the session egress fault, so the figures below are used only to describe the relationship, not folded into the Key Results above). Across 82 tasks, it reports that progressive disclosure of skill structure (a short root file, with detail pulled in on demand) helps on implementation, inspection, and repair tasks but is weaker where exact output conventions or numerical thresholds decide success. That reads as consistent with AgentScript-Eval's own pattern — v3's progressive-disclosure rewrite scored best overall on Opus, yet its gain was comparatively smaller on the debugging and optimization tasks where numeric and state precision matter most. Extension relation.
Reviewer's Judgement
First, I'd judge the paper's headline number (37.0% from a guidance rewrite, +23 points) to be an Opus 4.8-specific result, not a general finding. Applying the same change to Sonnet 5 yields 23.0% — lower than v1 (27.0%), which only changed packaging. Even afcode (28.0%, no skill pack at all) outperforms every skill-pack version on Sonnet 5 (Table 4 in the paper). The claim that "how guidance is written matters more than how it's packaged" is itself well-supported, but turning the specific rewrite recipe into a model-agnostic prescription overgeneralizes.
Second, from a token-cost-versus-payoff view, v2 is the most painful case in the data. It cut tokens nearly in half relative to baseline (835k → 389k), and pass@1 fell right along with it, from 33.0% to 14.0%. We already knew cost and performance don't always move together; this shows a configuration where both get worse at once. Judging a skill pack by compile rate alone (v2's is a respectable 86.0%) would miss this regression entirely.
Third, authoring never exceeding 16.7% under any configuration suggests that the reputation "LLMs write code well" rests substantially on the ability to fix existing code rather than create it from nothing. For enterprise DSL automation, having a person sketch the skeleton and letting the agent refine it looks like the more realistic division of labor at the current capability level.
Putting It to Work
- Trust authoring and editing differently — given the structural gap (authoring near 0%, optimization at 40-60%), design new-DSL automation so a person provides the skeleton and the agent handles revision and optimization.
- A/B test skill packs before shipping — the same skill set regressed when only delivery format changed (v1→v2), then recovered after a content rewrite (v3). Run the comparison on your own tasks first.
- Tune per model, not once for all — the configuration that was best for Opus was worse for Sonnet. Validate skill-pack and harness designs independently per model.
- Track compile rate and behavioral accuracy separately — v2 looks healthy on compile rate (86.0%) but its pass rate falls below baseline. Don't use compile success as a quality proxy.
- Shortlist low-cost open-weight models too — Gemma-4 31B had the highest compile rate at the lowest cost; measure cost-adjusted performance before defaulting to the priciest model.
Conclusion
AgentScript-Eval's contribution is showing, with execution-based grading, that the "LLM code generation is near-saturated" narrative doesn't carry over to a proprietary enterprise DSL absent from pretraining data. The best model reached only 25.0% pass@3, every model scored 0% on authoring, and an agentic loop plus a rewritten skill delivered the largest gains — but those gains worked differently across models, with the configuration that was best for Opus actually regressing for Sonnet. The workshop status, pre-peer-review state, and in-house DSL evaluation all remain, so the numbers are best read as directional signals that call for your own comparison before adoption. Catching code that compiles but behaves wrong before it merges is explored further in our piece on designing a full-pass-rate merge gate for coding agents.
References
- AgentScript-Eval: Benchmarking LLMs and Agents for Code Generation in an Enterprise DSL — arXiv abstract (source)
- Full HTML text of the same paper — used for table/figure verification
- Designing a Full-Pass-Rate Merge Gate for Coding Agents — sunny34.com blog
Ask AI about this review
The assistant has read this review and the numbers it verified. Ask anything — it answers from the text and says so when something isn't in it.
Loading the chat…