16% to 76% Compilation Success: Reviewing Chat2Scenic's RAG for Autonomous-Driving Scenario Generation

Source Information

Yuan Gao, Wenting Miao, Mattia Piccinini, Haoyu Wang, Qunying Song, Johannes Betz, "Chat2Scenic: An Iterative RAG-Based Framework for Scenario Generation in Autonomous Driving," arXiv:2607.14387 [cs.RO], submitted 2026-07-15. Accepted to IEEE/RSJ IROS (Intelligent Robots and Systems) 2026.

IROS is a leading peer-reviewed robotics conference, and this arXiv preprint is the version accepted there. The authors are affiliated with the Technical University of Munich's autonomous-driving research group (TUM-AVS), and both code and benchmark are public on GitHub, which supports reproducibility. The submission date, 2026-07-15, is well before this review's publish date — it was selected because it directly matches the requested topic (ontology/standard construction for autonomous-driving scenarios, specifically DSL grounding) and offers verifiable numbers. This session's direct access to arxiv.org is blocked by network policy, so the figures here are cross-checked against excerpts (including the results table) retrieved through a search tool.

Study Overview

Autonomous-driving simulation validation leans heavily on converting natural-language documents — regulations, incident reports — into executable scenario scripts. The authors argue this conversion produces low quality as a one-shot generation task, and propose Chat2Scenic: an interactive framework that refines scenarios iteratively through dialogue while retrieving DSL (domain-specific language) grammar and regulatory knowledge together via RAG. The authors describe it as "the first iterative retrieval-augmented framework to generate scenario scripts in DSL form."

For evaluation, they built a benchmark of 123 scenarios drawn from regulations including NHTSA and UN Vehicle Regulations. Two metrics: Compilation Success Rate (CSR) — whether the generated script actually compiles — and Framework Accuracy (FA) — how faithfully a compiled script implements the original regulation's intent.

Key Results

Chat2Scenic reached 76.42% CSR and 58.17% FA. The two retrieval-based baselines — Retrieval Assemble, which stitches retrieved results together directly, and Retrieval full script generation, which generates the entire script from retrieved context in one pass — scored 30.08% CSR / 11.03% FA and 16.26% CSR / 10.86% FA respectively.

The gap pattern between the two metrics stands out. Retrieval Assemble nearly doubles CSR over full generation (30.08% vs 16.26%), but FA stays almost flat between them (11.03% vs 10.86%). In other words, simply assembling retrieved results produces more "code that compiles" without producing more "code that correctly implements the regulation's intent" — compiling and being semantically correct are genuinely different axes, and the numbers show it. Chat2Scenic beat both baselines on both metrics, and on FA specifically it pulled more than 5x ahead of the baselines (roughly 11% to 58.17%).

Credibility Assessment

The grounds for trust are solid: it passed IROS 2026 peer review, and both the 123-scenario benchmark and the code are public on GitHub, so a third party can reproduce it under identical conditions. The two baselines represent genuinely different designs (assembly vs. generation), so the comparison doesn't rest on one easy reference point.

Three caveats apply. First, no ablation comparing Chat2Scenic against an LLM-only generation without RAG turned up in the retrievable excerpts, making it hard to separate RAG's contribution from that of the iterative-refinement loop itself. Second, the 123-scenario benchmark was built by the authors themselves, so whether its difficulty distribution is as neutral as an independent public benchmark needs separate verification. Third, while 58.17% FA is a large improvement over the baselines, in absolute terms it's just over half — what kind of failures make up the remaining ~40% (grammar errors vs. semantic errors) needs checking against the full paper.

Related Work

• dialogllmscenic authors (2023). Dialogue-based generation of self-driving simulation scenarios using Large Language Models — Prior work. Introduced the dialogue-based generation approach first, but relies on LLM generation alone without RAG. Chat2Scenic extends the same iterative, dialogue-driven refinement structure by adding regulatory and DSL-grammar retrieval, which is what drives the compilation-success improvement.
• Domain Knowledge Distillation from Large Language Model: An Empirical Study in the Autonomous Driving Domain (2023) — Complementary approach. This prior work focuses on extracting domain knowledge (an ontology-like concept structure) from an LLM, the opposite direction from Chat2Scenic's "retrieve knowledge to feed generation." Read together, the two show that handling autonomous-driving domain knowledge splits into an extraction-based direction and a retrieval-based direction.

Both papers share the direction that structuring autonomous-driving domain knowledge beats LLM-only generation, but neither directly tests Chat2Scenic's specific design — retrieving that structured knowledge live into an iterative, dialogue-driven generation loop.

Reviewer's Take

The number worth dwelling on here isn't the final 76.42% CSR, but the fact that CSR and FA move independently at baseline. Simply assembling retrieved results raises compilation success easily (16.26%→30.08%) while semantic accuracy barely moves (10.86%→11.03%). If a scenario-generation pipeline reports only compilation success rate, it's easy to miss that the results include scenarios that are "grammatically fine but semantically wrong."

The absence of an ablation isolating RAG's own contribution is a real gap. Iterative dialogue-based refinement alone could plausibly drive a substantial improvement; when the two factors (retrieval + iteration) are reported together, it's hard to judge where additional investment pays off most.

Passing IROS peer review with a public benchmark and code is good grounds to bring this result into practice. But a 123-scenario benchmark is small relative to the full span of regulatory clauses, so replicating on domain-specific scenarios of your own is a necessary step to confirm the FA gap holds.

Practical Takeaways

• Track compilation success and semantic accuracy separately — don't collapse a scenario-generation pipeline's quality into CSR alone. As this result shows, the two can move independently, and CSR alone will miss the real problem.
• Separate DSL grammar into retrievable knowledge — instead of re-explaining grammar rules in a long prompt every time, keep them as a separate, retrievable knowledge source and pull only what's needed per generation loop.
• Manage the regulation-to-scenario mapping explicitly — grouping scenarios by regulatory clause, as the benchmark does, lets you regenerate only the affected scenarios when a regulation is amended.
• Measure the iteration-count-to-improvement curve yourself — the paper doesn't report per-iteration contribution; measure it on your own data before adoption to find where returns diminish.

Conclusion

Chat2Scenic backs the claim that combining retrieval with iterative refinement improves the conversion from regulatory text to executable scenarios, in a form reproducible via public benchmark and code. The most notable result isn't the final performance number but the fact that CSR and FA moved differently at baseline — compiling and behaving as intended are different problems, and failing to measure them separately risks misjudging a scenario-generation pipeline's actual quality. The missing ablation separating RAG's contribution from iterative refinement's is a gap that needs follow-up verification.

References

• Chat2Scenic: An Iterative RAG-Based Framework for Scenario Generation in Autonomous Driving — arXiv abstract (source)
• Chat2Scenic — GitHub (code and benchmark)
• Dialogue-based generation of self-driving simulation scenarios (2023) — related work
• Domain Knowledge Distillation from LLM: Autonomous Driving (2023) — related work
• Designing Ontology-Grounded Autonomous-Driving Scenarios — sunny34.com blog