Why Static Gates Drift From Real Impact
An agent skill is a reusable procedural package — a SKILL.md file plus scripts — that harnesses like Claude Code, Codex, and OpenCode load on demand mid-task. Before a skill ships to a catalog, industry practice has leaned on two static checks: a structural score against roughly 50 frontmatter and documentation rules, and an LLM-judge rubric scoring clarity and trigger fit. NVIDIA's ACES study, posted to arXiv, ran both checks against 145 production skills: 94.5% cleared the structural 70-point gate, and 86.2% cleared the LLM-judge gate.
The two checks don't even agree with each other. Correlation between the structural and LLM-judge scores was a weak Spearman ρ=0.14, and the share clearing an 80-point structural bar dropped to 48.9%. Swap which static check you lean on and pass/fail can flip — which means treating either one as a release gate rests on an already shaky foundation.
What 87 Cases Hide Behind an Average of 0.2134
ACES held the question, agent, model, task, and scoring policy fixed and toggled only whether the skill was loaded, running 947 paired trials to measure Skill Lift — the score delta between loaded and unloaded. The average lift was 0.2134, but only 689 cases (72.8%) actually improved, 171 showed no change, and 87 (9.2%) got worse with the skill loaded. The number that matters most: correlation between the static scores and this measured lift was ρ=-0.0181 for structural and ρ=-0.0266 for LLM-judge — both statistically indistinguishable from zero.
Breaking lift down by metric sharpens the point further. Final-answer accuracy lift was 0.1431, while skill-execution adherence hit 0.3263 and instruction-behavior adherence hit 0.2983 — process metrics moved far more than the outcome metric did. A grading setup that only scores the final answer misses most of what a skill actually changed.
A Release-Gate Checklist for Agent Skill Catalogs
Before promoting a skill from draft to stable, declare numeric thresholds in code first. Require at least 10-15 paired trials (loaded vs. unloaded), hold back promotion if the Skill Lift confidence interval includes zero, and auto-route to review whenever the negative-lift share exceeds 10%. Score outcome metrics (final accuracy) and process metrics (skill-execution and behavior adherence) as separate fields, not one blended number.
The most common failure is assuming a passed static check means it's safe to ship. In this corpus that assumption had essentially no backing — the correlation sat at -0.02. The recovery path is straightforward: route new skills through a pilot workspace for a paired comparison before they land in the default-exposed catalog, and when negative lift shows up, quarantine the skill automatically and roll it back to the prior version or an unloaded state.
Treat catalog size itself as a separate operational risk. Average lift held steady at 0.133-0.149 with 1-20 exposed skills, but pass rate fell to 0.55 and average latency jumped from 258 to 1,290 seconds once the count reached 50. That routing experiment was a small supplementary run, so pinning down the exact tipping point from it alone is premature — but the direction is clear enough to cap default-exposed skills per workspace and route anything beyond that cap through explicit search. Lift also varied sharply by harness for the same skill: 0.3611 on OpenCode and 0.2904 on Claude Code versus 0.1264 on Codex and 0.0896 on Terminus-2 — reason enough not to carry a lift number validated on one harness over to another without re-checking it.
Schedule re-evaluation around model swaps. As the base model gets stronger, the unloaded baseline score rises with it, which can shrink the same skill's measured lift — so re-run a representative skill sample against every major model update. Track lift trend at the catalog level and flag skills drifting toward zero as retirement candidates, and keep catalog-size changes and individual skill-content changes in separate changelogs so a pass-rate drop can be traced to the right cause.
Where to Start
Passing a structural or LLM-judge check is not release approval — in this corpus, correlation with real impact was effectively zero. Measure Skill Lift directly through paired comparisons, track outcome and process metrics separately, and manage exposed-skill count and per-harness variance as their own risks, and you catch the 9.2% of regressions a static gate alone would let through.
References
Evaluating Skills, Not Just Agents: Agentic Continuous Evaluation of Skills — arXiv