Why Passing the Full History Makes Multi-Agent Systems Expensive
Orchestrating a multi-agent system means picking the next role and backbone somewhere between two extremes. Decide purely from the query up front and you can't react to progress or errors, so accuracy suffers. Pass the entire execution history on every step and each decision has to churn through irrelevant intermediate content, so cost balloons. Gated-Memory Routing, proposed by a University of Central Florida team in arXiv 2609.00237, inserts a single learned memory gate between those extremes: a retrieval gate pulls only the relevant slice, and a write gate decides whether a new reasoning step gets kept at all.
Accuracy and Cost Are Owned by Different Parts
In a same-system experiment that froze the routing logic and changed only memory handling, GSM-Hard accuracy landed at 70.27 versus 70.55 — essentially tied — while cost dropped 40% (0.985→0.587). HumanEval accuracy actually edged up, from 89.06 to 89.84, while cost fell 52.9%. But the leave-one-out ablation tells a different story about where the accuracy comes from: removing the LLM Router alone drops GSM-Hard accuracy by 14.0 points, while removing both memory gates costs only 2.56 to 6.25 points. The cost savings aren't from the gate either — they're from Adaptive Halting. Strip out halting and accuracy barely moves, but cost jumps 43% on GSM-Hard and spikes from 0.032 to 0.086 on HumanEval — a 168.8% increase. Put another way, the stopping rule alone is responsible for up to 63% of the savings.
Two Caveats: Synthetic Cost and Task Bias
These figures come from a synthetic, parameter-proportional unit price (input 0.003N, output 0.010N per billion parameters), not real API billing. They're valid as directional signals, but a number like 43.9% or 31.9% shouldn't be copied straight onto an invoice-savings line. There's also a task-level exception: on MBPP, a fixed pipeline (MacNet-Chain, 82.66) outperformed the gated approach outright. Short, uniform tasks leave little room for orchestration to pay off, so stacking a gate on top can shrink net gains instead of growing them.
From Design to Operations: A Memory-Gating and Halting Rollout Plan
Separate your goals before writing code. If cost is the target, check the halting threshold and gate recall first; if accuracy is the target, audit the backbone router's selection accuracy instead. A reasonable bar: accuracy delta within ±2 points, cost per step down at least 30%, and standard deviation under 1 point across three training seeds.
Set the halting threshold too aggressively and multi-step reasoning tasks get cut off before the answer is complete, wrecking accuracy. Keep separate thresholds per benchmark, and wire a canary stage that automatically reverts the threshold the moment the accuracy delta crosses its baseline.
Applying gated routing uniformly to short, uniform tasks leaves you with pure overhead. Put a task-difficulty classifier in front, and route anything below the complexity bar to a fixed pipeline instead.
When the write gate discards intermediate reasoning a later step actually needs, the system ends up recomputing the same sub-task repeatedly. Log every write-gate rejection, and feed sessions with recurring rework back into your weekly failure review to retrain the gate.
Log role assignment, backbone selection, gate pass/fail, the halting point, and tokens per step on every run. Before shipping, confirm the ranking holds across at least three training seeds, and only shrink the role catalog once you've verified the accuracy loss stays under 1 point — the source experiment cut 26 roles to 13 with less than a point of loss.
Reconcile the synthetic cost metric against your actual invoice every quarter. Re-classify each new task type between fixed pipeline and gated routing as it arrives, and manage the halting threshold from a single dashboard that tracks accuracy delta and cost savings side by side.
Takeaways at a Glance
Ship the halting rule before the memory gate — it's the higher-leverage change for the implementation effort involved. Treat the ablation result as your priority order: the backbone router owns accuracy, halting owns most of the cost, and only fold the synthetic cost metric into a budget once you've re-validated it against your actual pricing plan.
References
sunny34.com Research: gated-memory routing cuts multi-agent cost 32%