Failure Prediction and Collaboration-Value Prediction Are Different Problems
A cost-aware protocol routing study out of Argonne National Laboratory ran 4,181 competition-level math problems (Omni-MATH 2) through the same gpt-oss-120b solver under four protocols — direct solving (Baseline), iterative self-correction (Single), planner-executor-reviewer collaboration (PER), and multi-agent deliberation (Broadcast) — building a matched evaluation where every problem hits every protocol. Point the same confidence score at different targets and the picture splits: predicting Baseline failure scores AUROC 0.8847 and AUPRC 0.8950, both strong, but predicting whether PER delivers the first success collapses to an AUPRC of 0.1674. Confidence is a solid signal for "this is probably wrong," but a much weaker one for "and it's worth escalating this far to fix."
Ignore that split and you end up designing an escalation gate around a single threshold. Teams routinely try to use one confidence score to decide both whether to re-check an answer and how expensive a collaboration protocol to invoke — without noticing that the two predictions already differ in difficulty by more than fivefold. Gate design has to start from admitting that gap.
Four Protocol Tiers, Four Different Cost Curves
On the held-out set of 423 problems, Baseline solved 56.3% at 18.2K tokens, the difficulty-based Tier-majority heuristic solved 65.0% at 28.9K, and a frozen LLM router solved 73.8% at 71.3K. A self-confidence gate, by contrast, hit 78.0% at just 45.0K tokens — nearly a third cheaper than the frozen router while solving more problems. The post-hoc oracle (the ceiling) reached 92.4% at 101.1K tokens, leaving a 14.4pp gap between the gate and that ceiling.
That gap didn't close in any of six held-out router evaluation setups, holding steady at 18.5–28.9pp. Current escalation judgment isn't yet reliable enough to hand PER- and Broadcast-tier collaboration fully over to automatic promotion.
From Design to Operations: A Collaboration Escalation Gate Checklist
Separate the binary gate from the multi-tier gate at the planning stage, and give each its own target numbers. Set the tier-1 (Baseline↔Single) gate around a 45K token budget with solve-rate degradation capped at 15pp below oracle, and route tier-2 promotion (into PER or Broadcast) through sampled A/B tests or human review instead of an automatic threshold. Collapsing both decisions into one threshold hands high-cost calls to a signal whose AUPRC sits in the 0.10s.
The failure pattern shows up as directional bias. The conservative Tier-majority policy under-escalates 27.4% of the time and over-escalates 12.5%; the frozen LLM router cuts under-escalation to 18.0% but pushes over-escalation up to 33.3%. The higher-solving Llama/Gemma-family frozen routers drop under-escalation to 6–11%, but their over-escalation climbs to 63–71% — proof that a smarter router isn't automatically a cheaper one.
Recovery starts with measuring the two error rates separately. Rising under-escalation means solvable problems are getting stuck at Baseline, so lower the threshold; rising over-escalation means expensive collaboration is leaking out unnecessarily, so raise the threshold or force human review before tier-2 promotion.
Build pre-deployment scenario tests around three problem types mixed together — ones Baseline actually gets right, ones that need PER, and ones that need Broadcast — and check gate pass rates against each. Log standard fields: confidence score, the gate tier reached (Baseline/Single/PER/Broadcast), whether the answer was actually correct, tokens consumed, and excess tokens over the oracle. That split is what lets you track failure-prediction AUROC and collaboration-value AUPRC as separate metrics. Mask any sensitive problem context before storage and restrict access to the minimum staff needed for audit.
Track the oracle gap (currently 18.5–28.9pp) weekly and use it to trigger threshold recalibration. A sudden swing in under- or over-escalation rates in a given week is a signal the input distribution shifted; log any recalibration work as a separate change entry, and put token cost on the same dashboard as latency and API cost so the numbers actually inform deployment decisions.
Takeaways at a Glance
Confidence scores are strong at catching failure (AUROC 0.88) but weak at choosing how far to escalate (AUPRC in the 0.10s). Design around that gap by separating a tier-1 binary gate (45.0K tokens, 78.0% solve rate) from tier-2 high-cost promotion (human review or A/B testing) — that split is what safely covers the judgment gap that still runs 14.4pp or more behind the oracle.
References
LLMs Can Predict Failure Risk, But Struggle to Predict Which Collaboration Protocol Pays Off — arXiv