Source Document
Christopher Kevin, Narendran Raghavan, Jean-Francois Puget, Roshni Malani, Meghana Puvvadi, Moshe Abramovitch, Mohit Gupta, Rama Akkiraju, Subodh Prabhu, Yogesh Dangi, Wei Luo, Seong Hee Lee, "Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills", arXiv:2608.20614 [cs.AI], submitted 2026-08-20, DOI 10.48550/arXiv.2608.20614. Affiliation: NVIDIA. The methodology's open-source implementation, NVIDIA SkillEvaluator, is released under Apache-2.0.
This is a preprint without peer review. All twelve authors work at NVIDIA, and the evaluation framework itself is an open-source tool built and released by the same company. The paper is, in effect, a company's own methodology applied through its own tool to a corpus of 145 skills it selected itself, and no separate funding source is disclosed. The code is Apache-2.0, leaving a replication path open, but the raw execution trajectories and logs used for scoring are not released — they contain internal hostnames and repository paths. Live access to the source was blocked for this session, so the full text was cross-checked against an arXiv HTML snapshot the site's automated pipeline fetched at 2026-08-24T21:48:56Z.
Study Overview
The paper asks two questions: can static, document-only checks judge whether an agent skill (a SKILL.md package of procedural know-how plus optional scripts) is worth deploying, and if not, what else is needed? The authors first baseline the two static layers already common in industry — a structural score (roughly 50 rules across four weighted dimensions covering frontmatter and documentation conventions) and an LLM-as-Judge rubric (9-10 criteria scoring clarity and trigger fit) — against 145 real skills drawn from internal repositories and public catalogs. They then run ACES (Agentic Continuous Evaluation of Skills): paired trials that hold the question, agent, model, task, and grading policy fixed while only the target skill's presence changes, across four harnesses — Claude Code, Codex, OpenCode, and Terminus-2 — with trajectories normalized into a shared format called ATIF. The mean score difference between paired conditions is defined as Skill Lift. Of 64 production skills, 58 yielded 947 scored paired cases.
Key Results
First, how well the two static layers agree with each other. 94.5% of skills cleared the 70-point structural gate, but only 48.9% reached 80 points; 86.2% passed the LLM-judge gate. Yet the two static scores correlate at only Spearman ρ=0.14 (Pearson r=0.08) — the static methods do not converge on each other, let alone on anything else.
| Metric | Value |
|---|---|
| Structural 70-point gate pass rate | 94.5% |
| Structural score ≥80 rate | 48.9% |
| LLM-judge 70-point gate pass rate | 86.2% |
| Structural vs. LLM-judge correlation (Spearman / Pearson) | ρ=0.14 / r=0.08 |
| Structural score vs. measured Skill Lift correlation | ρ=-0.0181 (95% CI [-0.2667, 0.2327]) |
| LLM-judge score vs. measured Skill Lift correlation | ρ=-0.0266 (95% CI [-0.2745, 0.2247]) |
The live-trial results matter more. Across the 947 paired cases, mean composite score with the skill present was 0.7460 versus 0.5326 without — a Skill Lift of 0.2134 (95% CI [0.1967, 0.2301]) — while outcome-only lift (the mean of accuracy and goal-accuracy deltas) was 0.1799. By case, 689 (72.8%) showed positive lift, 171 were zero, and 87 were negative. Breaking lift down by metric, final-answer accuracy improved by 0.1431, while process metrics moved further: skill_execution by 0.3263, behavior_check by 0.2983, and skill_efficiency by 0.2758 (though efficiency was positive in only 41.7% of cases, a high-variance tradeoff). The pivotal finding is how little the static scores track this live lift: structural score correlates at ρ=-0.0181, LLM-judge score at ρ=-0.0266 — both statistically indistinguishable from zero. Mean lift varied by harness — OpenCode 0.3611, Claude Code 0.2904, Codex 0.1264, Terminus-2 0.0896 — with uneven case counts (211, 251, 259, and 226 respectively), but all four were positive. A separate routing-stress test found mean lift stable at 0.133-0.149 while visible skills ranged from 1 to 20, but at 50 visible skills the with-skill pass rate fell to 0.55 and mean wall time jumped from 258 to 1,290 seconds.
Credibility Assessment
Three things earn trust. The paired design holds the question, agent, model, task, and grading policy fixed, separating the skill's contribution from the agent's own capability. The direction is consistently positive across four harnesses of quite different character. And two cluster-robustness checks — bootstrapping over skills ([0.1898, 0.2350]) and over skill-harness cells ([0.1880, 0.2385]) — both overlap the headline interval, showing the 947 cases were not treated as if independently sampled when they are not.
The caveats are just as clear. This is a preprint without peer review, and every author and the evaluation tool itself belong to NVIDIA — an unambiguous conflict of interest. The corpus skews heavily toward System Access (49), Deployment (41), Platform (29), and Data Infra (21) skills, with only 3 Troubleshooting and 1 each of Dev-Tooling and Other, limiting generalization to under-represented categories. Raw trajectories and logs are withheld to avoid exposing internal information, so external verification depends on reproducing the released code rather than inspecting the original evidence. There is also a contrasting prior view worth noting: tools the paper compares against, such as SkillTester, focus on static utility and security checks without requiring a live agent run at all — this paper's central claim is that such static approaches carry essentially no measured predictive power. Because that cuts against common industry practice, replication on other organizations' skill corpora is the next test this result needs to pass.
Reviewer's Judgement
First, the most operationally weighty result here is not the 0.2134 lift figure but the near-zero correlation between static scores and measured lift. The common assumption that "passing structural and LLM-judge review means it is safe to ship" has no support in this corpus, and the warning likely extends beyond skills to any agent artifact reviewed through a similar static gate.
Second, the fact that most of the lift comes from process metrics — skill execution and behavior compliance — rather than final-answer accuracy has a direct implication for evaluation design: outcome-only scoring likely misses much of what a skill actually changes.
Third, I would not yet treat the routing-stress finding (stable under 20 visible skills, pass rate collapsing to 0.55 at 50) as a settled basis for catalog policy. It is a single 25-variant side study, so the sample is small and this paper alone cannot pin down the exact threshold where routing burden starts to bite. Still, the direction — that visible-skill count introduces risk independent of any individual skill's content quality — seems worth measuring separately for any organization whose catalog is growing.
Putting It to Work
- Never gate deployment on static checks alone — since structural and LLM-judge scores correlate with real effect at ρ≈0, run at least one live with/without comparison on sample tasks first.
- Track outcome and process metrics separately — score skill activation, script invocation, workflow order, and error recovery alongside final-answer accuracy, not instead of it.
- Judge adoption by paired lift, not absolute score — the delta between with-skill and baseline separates the skill's contribution from the agent's own strength.
- Cap visible-skill count per workspace — run your own routing-stress test before crossing roughly 20 visible skills to catch pass-rate and latency degradation early.
- Schedule re-evaluation after model updates — a stronger baseline model can shrink measured lift even as absolute performance rises, so re-measure skill value on a cadence.
Conclusion
ACES's real contribution is not a new scoring algorithm but evidence that the static scans widely used in industry are not a proxy for real deployment value. Both the 94.5%-passing structural check and the 86.2%-passing LLM-judge rubric were statistically unrelated to Skill Lift measured through live execution — a concrete number for the gap between reading a file and running it. That said, the shared NVIDIA affiliation of every author and the evaluation tool, the withheld raw trajectories, and the skewed category distribution all warrant caution. The risk of treating static checks as a sufficient deployment gate is explored from another angle in Putting Tool-Agent Red-Teaming Into the Deployment Gate.