Right Answer, Wrong Pick: What OrchSLM Reveals About Small-Model Routing

Source Document

Chengxi Zhang, Yu Yao, "OrchSLM: Probing the Dynamics of Small Language Model Orchestration", arXiv:2609.13470 [cs.AI], submitted 2026-09-11, DOI 10.48550/arXiv.2609.13470. Affiliations: Harvard University, MIT. Licensed CC BY 4.0.

This is a preprint, not yet peer-reviewed. The two authors note equal contribution with author order decided by a coin flip, so it reads as a genuine co-first-author paper. The text has no funding or conflict-of-interest section (checked around the references and for Acknowledgments/Funding/Conflict keywords — none found); the only disclosure present is a "Generative AI usage" note stating the authors used LLMs to help with writing clarity and experimental code. We cross-checked the full text against a GitHub Actions snapshot of the arXiv HTML full text taken 2026-09-15T23:33:28Z — this session's egress was blocked even for the control domain example.com, and a manual re-trigger of the snapshot workflow also failed with arXiv API HTTP 406, so we used the most recent snapshot available (collected two days prior).

Study Overview

The question is simple: when several small language models (SLMs) are combined without letting them talk to each other — only their cached answers are pooled — what actually determines routing quality? The authors split this into three axes: whom to trust (expert weighting vs. equal weighting), how to combine evidence (sum vs. max), and when agreement counts as a signal (self-consistency vs. mutual support). Their claim is that existing methods — majority voting, self-consistency routing, cross-support, expert-weighted routing, single-best routing — are all just specific parameter settings of one generalized routing family, OrchSLM.

The experiments cache answers from seven open-source SLMs (Qwen2.5-7B-Instruct, Llama-3.1-8B-Instruct, Llama-3.1-Tulu-3-8B-DPO, DeepSeek-Math-7B-Instruct, Gemma-2-9B-IT, Phi-3.5-mini-instruct, Ministral-8B-Instruct-2410) and compare routing accuracy across four benchmarks: MATH-500 (competition math, one dominant expert), GSM8K (saturated arithmetic), GPQA (hard, balanced science), and MMLU-Pro (broad, multi-topic science). Results use 10 subsampling replicates (the K20R10 protocol), with standard deviations reported.

Key Results

The single biggest lever was expert weighting. Comparing equal-weight routing against expert-weighted routing shows a consistent improvement across all four benchmarks.

By contrast, how evidence is aggregated (sum vs. max) mattered far less — only GSM8K favored max by a clear +0.8pp, the other three tasks differed by under 1pp. The more consequential finding is the gap between oracle coverage and actual routing accuracy. Oracle coverage asks "did any of the seven models get it right"; routing accuracy asks whether the router actually picked that answer. The paper reports SLM-MUX (Wang et al., 2025) — the prior cache-based router it builds on — under the same protocol (K20R10), so we include it alongside.

Two different bases are at play here, so keep them separate — "gap to oracle" is an absolute difference in percentage points, while how much better the best generalized router is than SLM-MUX is a separate relative-improvement calculation (MATH-500 +8.7%, GSM8K +1.3%, GPQA +16.1%, MMLU-Pro +4.4%, each computed directly against the SLM-MUX figure). On the math tasks the oracle gap stays around 6pp, but on GPQA it exceeds 35pp — meaning the correct answer is present somewhere in the seven-model pool roughly 79% of the time, yet the router fails to surface it more than half the time. The paper also reports, via a figure (Figure 6) rather than a table, that oracle coverage keeps rising as the pool grows, while routed accuracy typically peaks at three or four models and then declines — we report the direction only, since no precise figures accompany that chart.

Credibility Assessment

Three things support trust here. First, the direction of the expert-weighting effect never reverses across four benchmarks with quite different characters (competition math, saturated arithmetic, hard balanced science, broad science). Second, the authors report standard deviations from 10 subsampling replicates, giving readers a way to judge whether a table difference is signal or noise (e.g., GPQA's best generalized routing at 0.4328 sits outside the range implied by SLM-MUX's 0.3727 ± 0.0087). Third, having an explicit oracle-ceiling reference line separates "routing got better" from "the pool happened to contain the right answer."

There are real caveats too. The detailed answer-extraction and normalization rules live only in the appendix, so the main text alone doesn't fully support reproduction. The expert-weighting prior (κ) that drives most of the gains requires, in the authors' own words, "validation data or an estimate of model quality" — meaning this lever is unavailable without a labeled validation set in production. The study is also limited to benchmarks with extractable, gradeable answers; extending it to open-ended generation is left as future work. As for conflicting evidence: prior work (Li et al., 2024, "More Agents Is All You Need") argues that adding more agents to a majority vote keeps improving accuracy. This paper qualifies rather than refutes that claim — oracle coverage does keep climbing with pool size, but routed accuracy peaks at three or four models, adding a caveat that a bigger pool makes picking the right answer harder even as it makes the right answer more available.

Related Research (Literature Cross-Check)

• Wang, Wan, Kang, Chen, Xie, Krishna, Janapa Reddi, Du (2025). SLM-MUX: Orchestrating Small Language Models for Reasoning — Direct comparison (prior work / extension). A cache-based router using independent sampling, confidence-based selection, and model-subset search, benchmarked under the identical K20R10 protocol in this paper. OrchSLM claims to subsume this method as a specific parameter setting of its generalized family.
• Li, Zhang, Yu, Fu, Ye (2024). More Agents Is All You Need — Contradiction / qualification. Argues that adding more agents to a majority vote improves performance. This paper's observation — oracle coverage keeps rising with pool size while routed accuracy peaks at three or four models — adds a condition to that claim rather than overturning it outright.
• Wang, Wang, Athiwaratkun, Zhang, Zou (2024). Mixture-of-Agents Enhances Large Language Model Capabilities — Contrast (different paradigm). An interactive design where models iteratively refine answers by seeing each other's outputs. OrchSLM's non-interactive design — post-generation aggregation of cached answers only — represents a deliberately different point on that design space.

Bibliographic details for these three works were taken from this paper's own reference list. We could not independently re-verify their primary text because this session's egress was blocked (we do not cite any numeric claims from those three papers themselves — only the SLM-MUX figures reported directly in this paper's comparison table are cited). The three sit in distinct relationships to this paper: SLM-MUX is the prior router this work directly generalizes and outperforms under the same protocol, "More Agents" is a claim shown to hold only conditionally, and Mixture-of-Agents represents the interactive alternative this paper deliberately avoids.

Reviewer's Assessment

First, the most practically important number in this paper isn't 79.38% — it's GPQA's 35.25pp gap. The correct answer sits somewhere in the seven-model pool 78.54% of the time (oracle), yet the router only surfaces it 43.28% of the time. The common instinct to "grow the pool to raise coverage" doesn't solve the routing problem on hard tasks; if anything, it can mask it.

Second, it's notable that the improvement over SLM-MUX scales with task difficulty (+8.7% on MATH-500 vs. +16.1% on GPQA). A single, comparatively simple lever — expert weighting — pays off more on harder tasks where consensus is shakier than on easier ones where it isn't. That suggests routing engineering effort is better spent where agreement is unstable, not where it's already easy.

Third, while the paper is honest that expert weighting (κ) consistently beats free signals like self-consistency and majority voting, its admission that setting κ requires validation data is a real deployment obstacle. Any pipeline shipped without a labeled validation set won't be able to reproduce this paper's best-case numbers as-is.

Practical Takeaways

• Tune expert weighting first — Deciding "how much to trust which model" mattered far more than choosing between sum and max aggregation. It was the single largest lever in this study.
• Secure a small validation set early — Setting the expert-weighting prior (κ) requires data to estimate model quality. Without a validation set, this paper's best-case gains are not reproducible.
• Don't scale the pool blindly — Adding models raises oracle coverage but routed accuracy typically peaks at three to four models. Verify the router can actually distinguish new candidates before adding them.
• Match strategy to task difficulty — Simple strategies suffice on saturated tasks (e.g. basic arithmetic); expert weighting pays off more on hard, balanced tasks. Avoid locking in one fixed router for every task type.
• Monitor "heard" vs. "helpful" separately — The paper reports that a model can heavily influence the router's final pick (attribution) without actually improving accuracy (contribution). Periodically checking leave-one-model-out accuracy changes is a cheap safeguard.

Conclusion

OrchSLM's contribution isn't another router to add to the pile — it's separating three decisions that existing routers bundle together (trust, aggregation, agreement) and measuring each one's effect size independently. The result: expert weighting is by far the dominant lever, and there is a substantial — especially on hard tasks — distance between "the pool contains the right answer" and "the system actually selects it." Because this is a pre-peer-review preprint and its main lever requires validation data to configure, treat the numbers as directional and reproduce them on your own benchmark before relying on them. For a concrete design that routes to small models specifically to cut cost, see Cutting Costs 10x with Small-Model Task Routing on this site's blog.

References

• OrchSLM: Probing the Dynamics of Small Language Model Orchestration — arXiv abstract (source)
• Same paper, full HTML text — used for table/figure cross-checking (via snapshot)
• Cutting Costs 10x with Small-Model Task Routing — sunny34.com blog
• SLM-MUX (Wang et al., 2025) — related work source
• More Agents Is All You Need (Li et al., 2024) — related work source
• Mixture-of-Agents (Wang et al., 2024) — related work source