Explore More, Drift Less: Reviewing CANOPY's Agentic RL

Source Document

Liming Pu, Xiaoxia Li, Yifu Liu, Teng Cao, Bin Yang, "Explore More, Drift Less: Outcome-Only Reinforcement Learning Can Suffice for Long-Horizon Interactive Agents", arXiv:2609.01245 [cs.LG], submitted 2026-09-01, DOI 10.48550/arXiv.2609.01245. Training code is planned for release (github.com/AlibabaResearch/SignalCoverageRL, the repository was empty as of submission).

This is a preprint without peer review, classified under cs.LG. The paper text does not state author affiliation. The GitHub organization name (AlibabaResearch) suggests a possible Alibaba affiliation, but this cannot be confirmed, and there is no section disclosing funding or conflicts of interest. Both backbones used in the experiments, Qwen3-14B and Qwen3.5-9B, are Alibaba's own Qwen family — if the affiliation guess is correct, that would be a conflict-of-interest factor, but for now it remains speculation. Because of a session egress failure, the full text was checked against a snapshot (docs/research-authoring/snapshots/2026-09-05/2609.01245.txt) taken at 2026-09-04T22:03:39Z rather than via live WebFetch.

Study Overview

There is a common belief that outcome-only RL — rewarding an agent only on final task success — soon hits a ceiling on small open models when training long-horizon agents. Recent work has tried to work around this by adding denser step-level rewards, SFT priors, skill libraries, or multi-agent orchestration around the training loop. The authors argue the ceiling is an artifact of how training is conducted, not a limit of outcome-only RL itself, and name two specific failures. Signal starvation: under group-relative reward, a training signal only appears when a rollout group for a task mixes successes and failures; with too little exploration, exactly the hardest — and most instructive — tasks go silent. Policy drift: repeatedly training on a bounded task pool lets an unanchored objective collapse the sampling distribution, breaking the policy right when informative groups are already becoming scarce.

The proposed method, CANOPY (Coverage-ANchored On-PolicY RL), answers both failures directly. "Explore more" scales the rollout group size to 32 and removes the per-turn generation cap, keeping signal alive even on the hardest tasks. "Drift less" adds a light KL anchor, strictly on-policy updates (each batch of rollouts is used only at the step that collected it), and a token-level loss confined to the agent's own action tokens. On top of this, "budget transfer" switches to a larger interaction budget (more turns, more context) at test time than during training. Evaluation covers AppWorld, a long-horizon coding benchmark with an official leaderboard, and SWE-bench Verified (real-repository bug fixing) as a transfer check.

Key Results

On the AppWorld leaderboard, a Qwen3-14B policy trained with CANOPY reached Test-Normal TGC 86.9 and Test-Challenge TGC 67.6, the highest score among trained-policy entries. That leads the next-best trained policy, ESAT (also Qwen3-14B, reported at mean@8), by 11.7 points on Test-Normal and 9.1 on Test-Challenge. Training-free systems that wrap a frontier closed model with skill libraries, retrieval, or test-time debugging (HCL-GP, ASSAY, and others) post higher numbers, but the authors are explicit that HCL-GP uses a non-standard joint-scenario protocol not comparable to the other rows, and the rest lean on much larger backbones plus external machinery — so none of them belong in a like-for-like comparison with trained policies.

Numbers on a different basis (training budget vs. enlarged budget) are mixed in here too and need separating. Evaluating the same step-90 checkpoint at the training budget (50 turns / 32k tokens) and the enlarged budget (100 turns / 61k tokens), CANOPY's Test-Normal mean@4 rises from 79.5 to 83.2, while the base model reaches only 32.4 even at the enlarged budget (22.8 at the training budget, per the text) — more than 47 points behind CANOPY. The authors read this as evidence that training, not the enlarged budget, is what buys the capability.

The ablations — restoring one changed component at a time — line up with the diagnosis. From a Test-Normal mean@4 of 79.5 at the 90-step retraining baseline, shrinking the group size from 32 to 8 costs 16.4 points, and halving the gradient mini-batch relative to the rollout batch (so the second update trains on data drawn by an earlier version of the policy) costs 17.4 points — the two largest drops. Removing the KL anchor costs 7.0, removing the token-level loss costs 5.4, and dropping the hardest tier costs 6.0, while switching to a denser reward moves the score the least, by only 1.8. The authors estimate the standard error of a 4-run mean at about 0.5 points, so the 16.4- and 17.4-point drops are clearly outside noise while the 1.8-point change sits near the noise floor and should be read as suggestive only, as the paper itself notes. The same direction reproduced on SWE-bench Verified: the Qwen3.5-9B resolve rate rose from 31.3% to 47.9% (mean@4), and to 50.2% at the enlarged budget.

Credibility Assessment

Three things earn trust. The headline result is externally verifiable through AppWorld's official leaderboard submission rather than a private in-house metric. The ordering of the ablation results matches the authors' own diagnosis (signal starvation, policy drift) exactly, and they supply a noise estimate that lets a reader judge which drops to trust. And the principles validated on AppWorld — larger groups, KL anchor, on-policy updates, token-level loss — transferred to SWE-bench Verified with only the hyperparameters retuned, reproducing the same direction of improvement.

The caveats are just as clear. The training code is only "planned for release" as of submission, with an empty repository, so independent reproduction is not yet possible. Author affiliation and funding are not stated in the text and cannot be confirmed; if the guessed affiliation is correct, using that company's own model as the backbone would be a conflict of interest. The AppWorld training pool has only 90 tasks, a scale limitation the authors themselves flag as future work ("environment scaling"). Each ablation variant was trained only once (no repeated seeds), so anything short of the largest drops should be read cautiously. The paper itself scopes its claims to tasks with automatic, program-based verification — code and terminal environments — and does not claim the result for browser or GUI control or any open-ended task without an automatic verifier.

Reviewer's Judgement

First, this paper's most valuable contribution is not the 86.9 leaderboard number itself but how closely the ablation results track the diagnosis. The two factors the authors call the root causes — group size and mini-batch composition — produced the two largest drops (16.4, 17.4 points), while the factor they call secondary, reward densification, produced the smallest change (1.8 points, near the noise floor). It is a rare case where the claim and the evidence point the same way.

Second, the framing that "training, not budget transfer, buys the capability" is persuasive but arguably overstated. The base model also improved by more than 40% at the enlarged budget (22.8 to 32.4), which means budget expansion on its own is not a lever to dismiss. Since CANOPY's advantage is the sum of a budget-transfer effect and a training effect, the training contribution alone is probably narrower than the raw table gap suggests.

Third, the self-imposed scope — automatically verifiable domains like code execution and terminals (Appendix J) — is both what makes this paper trustworthy and the key filter for adoption decisions. For open-ended tasks without a verifier, or GUI control that needs a human to judge the outcome, the signal-starvation diagnosis may still hold, but the fix of "manufacturing more signal" simply is not available.

Putting It to Work

• Check your rollout group size first — going from a group of 8 to 32 alone produced more than a 16-point difference. If your RL pipeline's group size is small by this paper's standard, it is the first lever to try.
• Prioritize staying on-policy over the KL anchor — splitting the gradient mini-batch smaller than the rollout batch, which introduces some off-policy behavior, cost more (17.4) than removing the KL anchor (7.0). Checking your batch composition first has a better cost-to-benefit ratio.
• Don't over-invest in reward design — densifying the reward produced the smallest improvement in this paper (1.8 points, which the authors themselves treat as within noise). Invest there only if it remains a bottleneck after fixing exploration and drift.
• Confirm the domain is verifiable before adopting — these principles are validated only on code and terminal tasks with automatic verifiers (unit tests, state comparison). Don't carry them over unchanged to tasks that need human judgment.
• Cross-check results on a public leaderboard — publishing to a reproducible, external leaderboard like AppWorld or SWE-bench Verified, rather than a private benchmark, is what let this paper's numbers earn trust here.

Conclusion

CANOPY's contribution is not a new algorithm but a re-diagnosis of why outcome-only RL was thought not to work. Addressing signal starvation and policy drift head-on with a larger group and on-policy, KL-anchored updates took a 14B open model to the top of the trained-policy field, ahead of training-free systems wrapped around much larger closed backbones, and the same principles carried over to real-repository software repair. That said, the code is not yet public, author affiliation and conflicts of interest cannot be confirmed, and the authors themselves scope the result to automatically verifiable domains. For the operational question of how much interaction budget to give an agent, see the Memory-Based Agentic RAG Rollout Checklist.

References

• Explore More, Drift Less: Outcome-Only Reinforcement Learning Can Suffice for Long-Horizon Interactive Agents — arXiv abstract (source)
• Full HTML text of the same paper — used to verify the results tables and ablation figures
• The Memory-Based Agentic RAG Rollout Checklist — sunny34.com blog