A harness is a deploy unit, not a prompt string

"Harness" doesn't mean the system prompt alone. It's four parts bundled together — prompt text, tool schemas, permission and sandbox policy, and the pinned model version — that jointly decide what an agent does. Change any one of them and the order in which tools get called, how arguments get filled, and how failures get handled can all shift at once.

Yet even teams that put application code through PR review and CI often ship harness definition files with no approval step at all. A few edited prompt lines or one changed schema field can introduce a regression with nothing in place to catch it.

Regressions show up in the trajectory before they show up in the output

Anthropic's guidance on agent evals draws a line between the final outcome and the transcript — the full record of tool calls, arguments, and intermediate results. Two runs can land on the same answer while taking completely different tool-call paths to get there.

Harness regressions behave the same way. Compare output text alone and a change looks fine, while underneath it the agent has started calling a tool it never called before, or reordered arguments in a way that breaks a downstream system. Without diffing the trajectory, these regressions surface only after they've already shipped.

From Design to Operations: A Rollout Roadmap for the Harness Regression Gate

Start by building a golden trajectory set. Pull 40 or more representative scenarios from production traces and freeze their tool-call order, arguments, and final state as snapshots. Below 40 cases, the regression rate itself becomes statistically too noisy to serve as a deploy gate.

Pin the block threshold to numbers so it doesn't drift in practice: if the tool-call sequence match rate against the golden set drops below 95%, or new failure cases exceed 3, block the automated deploy and route it to human approval.

Three failure patterns show up repeatedly. First, adding a single instruction line to the system prompt flips the call order so a previously secondary tool now fires first. Second, adding an enum value to a tool schema breaks parsing because a parser cached the old schema version. Third, unpinning the model version and auto-promoting to the latest release changes how the same prompt fills tool arguments.

Recovery branches by failure type. A sequence reversal gets an immediate rollback to the prior harness version; a schema parsing error gets a cache invalidation followed by revalidation; a model-promotion regression gets the pin restored to the previous version, with the promotion itself deferred to the next release cycle.

On the operations side, treat harness changes like application code: route them through PR review, and mark the diff with four standard fields — prompt, tool schema, permissions, model pin. Keep a harness version field in trace logs so you can trace exactly which deploy introduced a regression, and mask user utterances and PII in traces before they're saved into the golden set.

Add a rehearsal step before deploy: replay the full golden set in staging, confirm the match rate and new-failure count, and only then let the change through the real deploy pipeline. Tag any deploy that skipped rehearsal separately so it's flagged for a post-hoc audit.

The improvement loop starts with logging regression rate and new failure cases as a separate changelog for every deploy. Tracked over time, this makes clear which kinds of changes trigger regressions most often. OpenAI's own guidance on detecting prompt regressions recommends rerunning the same eval whenever a prompt, model, or parameter changes and comparing results — and that repeated rerun is exactly what keeps a golden set current.

Key Takeaways

Treat the harness as a deploy unit made of four parts — prompt, tool schema, permissions, model pin — not just prompt text, and pin a golden trajectory set of 40+ cases with a 95% sequence match floor into the deploy gate as code. That's how you catch trajectory regressions before shipping instead of after, which output-only comparison misses.

References

Demystifying evals for AI agents — Anthropic

Evals API use-case: Detecting prompt regressions — OpenAI Cookbook

The verification-tax review: commits up 180%, releases up only 30% — sunny34.com Research