Medal Rate 37.9%→72.7%: Reviewing MLExpert, an Autonomous ML Engineering Agent

Source Details

Shangheng Du, Jinxin Shi, Shuaiyu Zhang, Yiming Wu, Junpeng Zhao, Shiyang Feng, Xin Li, Liang He, Xiangchao Yan, Bo Zhang, "MLExpert: Towards a Generalist Agentic Model for Autonomous Machine Learning Engineering", arXiv:2610.22344 [cs.AI], submitted 2026-10-15, DOI 10.48550/arXiv.2610.22344. Affiliation: Shanghai Artificial Intelligence Laboratory · East China Normal University. Code and model weights released (github.com/InternScience/MLExpert, huggingface.co/InternScience/MLExpert).

This is a preprint, not yet peer-reviewed. Most co-authors are affiliated with Shanghai AI Laboratory. Note that the Kimi-K3 and GPT-5.6 Sol comparison numbers were not measured by this paper's own team — they are cited secondhand from a competing paper (Frontis-MA1), which matters when weighing those specific comparisons. The full text was cross-checked against a GitHub Actions snapshot collected at 2026-10-18T22:08:40Z (direct WebFetch access was blocked by this session's egress proxy).

Study Overview

The question is simple: how far can a model get at long-horizon machine learning engineering if it organizes its own experiments instead of following a human-designed workflow? Prior MLE automation systems mostly let fixed search rules decide experiment order, with the LLM only generating or debugging code at designated steps. The authors flip this: the model itself decides when to analyze, revise, or change direction.

The method has three parts: Kaggle-MLE Factory, which converts past Kaggle competitions into executable tasks; an interaction harness that preserves code, execution logs, and candidate submissions as a tree; and post-training in three stages (general agent SFT → MLE-domain SFT → Windowed Interaction RL, WIRL). WIRL's core idea addresses delayed reward: since one experiment's outcome reflects several accumulated decisions, it groups consecutive decisions into an "execution window" and assigns the whole window a shared, verifiable reward based on medal thresholds. Evaluation covers MLE-Bench Lite (22 Kaggle tasks), NatureBench Lite (10 tasks), and AIME scaffold optimization.

Key Results

The main comparison is a controlled setup: the same 35B base model (Qwen3.6-35B-A3B) versus MLExpert, both under the same harness. Any Medal (share of tasks winning a medal) is the headline metric, computed over 22 tasks.

We checked that the medal-tier sum (Gold+Silver+Bronze) matches Any Medal (MLExpert: 50.0+16.7+6.1=72.8 ≈ 72.7, within rounding). The gap holds under a general-purpose harness (Claude Code) too — MLExpert 69.7% vs. base 42.4% (+27.3pp). Appendix Table 9 reports mean±SEM over three runs — MLExpert's Any Medal is 72.73%±2.62pp, and with only 22 tasks the error bars are not negligible.

The stage breakdown (chart above, paper's Table 3) shows each phase contributing independently: general agent SFT 37.9%→47.0%, MLE-domain SFT 47.0%→62.1%, WIRL 62.1%→72.7%. Starting from the same MLE-SFT checkpoint, WIRL (72.7%) clearly beats GRPO (56.1%) and PPO-style optimization (60.6%) (paper's Table 4); removing either the trend reward or policy correction alone drops it to 66.7% and 65.2% respectively, showing both design choices matter. On NatureBench Lite, Surpass-SOTA rose from 0.0% to 50.0%.

Credibility Assessment

Three things support trust here. First, the core comparison is controlled — same base model, same harness — so training and model effects aren't conflated. Second, the authors explicitly excluded training tasks overlapping with MLE-Bench/NatureBench, controlling for contamination. Third, they report mean±SEM over three runs (Table 9), being transparent about variance, and the gap holds under a different, general-purpose harness too.

Caveats are equally clear. This is a pre-peer-review preprint, and the Kimi-K3/GPT-5.6 Sol numbers are secondhand citations, so evaluation conditions (harness, number of attempts) may differ. The ablations (Tables 3–4) report only the single aggregate Any Medal metric, without tier-level breakdowns. With only 22 tasks, SEMs across models run 1.5–7pp wide. The compute budget (one GPU, 12 hours per task) is limited, so whether the gap holds at larger scale isn't established by this paper alone.

Related Work (Academic Cross-Check)

• Chan et al. (2024). MLE-bench: Evaluating Machine Learning Agents on Machine Learning Engineering — benchmark-origin. OpenAI's 75-task Kaggle-based benchmark; its 22-task "Lite" subset is what MLExpert uses. At release in 2024, the best setup (o1-preview + AIDE) reached only a 16.9% bronze-level rate.
• Liu et al. (2025). ML-Agent: Reinforcing LLM Agents for Autonomous Machine Learning Engineering — prior. A related effort from the same Shanghai AI Laboratory lineage, reportedly training a 7B model with RL on just 9 tasks to outperform much larger models (independently confirmed via WebSearch; original text not re-read under egress block — figures used only to describe the relationship). MLExpert reads as a follow-up that scales up model size and task breadth.
• Yang et al. (2026). Frontis-MA1: Training an AI4AI Model towards Recursive Self-Improvement in Machine Learning Engineering — contradiction/parallel. Using evolutionary search (operator learning) instead of RL, it reportedly raised medal rate to 60.6%/71.2% from a 39.4% base (independently confirmed via WebSearch; figures used only to describe the relationship). MLExpert's own paper cites this work for its Kimi-K3/GPT-5.6 Sol comparison numbers.

Taken together, these three works show MLE-Bench Lite's practical difficulty ceiling shifting fast — from 16.9% (bronze-level) in 2024 to multiple methods reaching the 60–77% range by 2026. That two entirely different approaches (RL vs. evolutionary search) converge on a similar final range suggests MLExpert's result is better read as part of a generational shift than as a uniquely superior method.

Reviewer's Take

First, I'd argue the paper underplays something more important than the 72.7% headline: the medal composition within it. Table 9 shows 50.0pp of MLExpert's 72.7pp medal share is Gold, while the base model's 37.9pp medal share included only 22.7pp Gold. The core gap looks less like "winning more medals" and more like "winning higher-tier medals when it does win."

Second, WIRL's margin over GRPO/PPO (72.7% vs. 56.1%/60.6%) should be discounted somewhat, since the proposed method is compared against baselines the same authors designed and tuned. The paper doesn't state whether tuning effort was matched across methods, so I'd read a double-digit-point gap conservatively.

Practical Takeaways

• Execution-window reward design — for long tasks where feedback accumulates across many decisions, bundle rewards at "outcome boundaries" rather than per full trajectory to ease delayed feedback.
• Separate SFT stages by role — train general tool-use and domain-specific judgment separately so each stage's contribution can be measured independently.
• Make contamination control a default — add explicit train/eval task-overlap exclusion to your pre-deployment checklist.
• Publish tier breakdowns, not just aggregates — report tier-level breakdowns alongside any single aggregate metric to show whether gains are "more wins" or "higher-quality wins."
• Report repeated runs with SEM — the fewer the tasks, the more a single run misleads; report mean and standard error across repeated evaluations.

Conclusion

MLExpert's contribution lies less in a new model than in reframing long-horizon MLE as a problem of state management plus verifiable reward design. The staged SFT→RL breakdown and WIRL's edge over GRPO/PPO show each design choice contributing independently, but this is a preprint, some comparison figures are secondhand, and ablations report only a single aggregate metric. Given that a very different method (Frontis-MA1) reached a similar final range around the same time, it's safer to read these numbers as a directional signal rather than proof of one method's absolute superiority. Before adopting a verifiable-reward design in your own pipeline, it's worth first checking the reward-hacking safeguards covered in our RLVR verifier design post.

References

• MLExpert: Towards a Generalist Agentic Model for Autonomous Machine Learning Engineering — arXiv abstract (primary source)
• Full HTML text of the same paper — used for table/stage-number cross-checks (via a 2026-10-18T22:08:40Z snapshot, session egress was blocked)
• MLE-bench (Chan et al., 2024) — benchmark origin
• Frontis-MA1 (Yang et al., 2026) — comparison prior work
• RLVR Verifier Design and Reward Hacking — sunny34.com blog