Source Document

Nova-Silicon Team, "AnalogLayoutWorld: An Open Benchmark for Agentic Analog Layout Design", arXiv:2611.16991v1 [cs.AI], submitted 2026-11-12. Affiliation: Nova-Silicon (an analog-EDA company; corresponding email [email protected]). The benchmark repository is only described as "to be open-sourced soon" (footnote 1 in the paper).

This is an un-peer-reviewed preprint, and every author is on the company team that built the benchmark. That matters, because it shows up directly in the results: the same team sets the task suite and the scoring weights, and also evaluates its own proprietary agent, Novara, on that same benchmark and reports it as the top performer. The acknowledgments credit researchers at Shanghai AI Laboratory and Fudan University for advice only, with no external funding source disclosed.

Study Overview

The question: can coding agents extend beyond software into long-horizon engineering tasks governed by physical constraints? The authors test this through analog integrated-circuit (IC) layout — turning a fixed, transistor-level netlist into a manufacturable layout (a GDS file). They built 40 expert-reviewed tasks (amplifiers, references, power-management circuits, sampling circuits, oscillators) on the SKY130 process, giving agents only open-source EDA tools and documentation (gdsfactory, KLayout, Magic, ngspice — no benchmark-specific tools or MCP interface) and a three-hour wall-clock budget to design, verify, and revise on their own. After the agent stops, grading runs independently in a fresh environment: DRC (design-rule checking), LVS (layout-vs-schematic matching), PEX-based electrical simulation, area efficiency, and engineering quality. Each task runs three independent repetitions, averaged.

Key Results

Across 13 model–harness configurations (excluding Novara, which the authors evaluate separately), 91.1% of attempts produced a GDS layout, but only 21.3% satisfied the benchmark's full acceptance criteria (passing DRC, LVS, and every prescribed simulation case). This figure appears twice in the paper's own abstract and introduction, and we independently reconstructed it by averaging the Acc.(%) column across the 13 configurations in Table 1 (3.3, 8.3, 9.2, 0.0, 35.0, 27.5, 14.2, 12.5, 0.0, 20.0, 14.2, 70.8, 62.5) — the mean comes to 21.3%, matching the text.

Model / SystemHarnessDRC(%)LVS(%)SIM(%)Acc.(%)Avg. Score
Novara (in-house, closed)—97.5100.075.072.50.8154
GPT-6 AstraCodex99.298.382.570.80.7483
GPT-6.1 SolCodex92.591.780.062.50.7023
GLM-5.3OpenCode80.069.242.535.00.4779
Kimi-K3Kimi Code50.040.030.020.00.2900
DeepSeek V4.1 FlashDSH37.530.017.59.20.2246
GLM-5.2 FP8OpenCode10.010.05.00.00.1075

Source: paper's Table 1 (13 model-harness evaluations, averaged over three repetitions per task). Novara is a closed, in-house system the benchmark's creator runs on its own cluster, and its row carries no comparable harness or resource data. GPT-series agents led on physical and electrical verification (DRC, LVS, SIM), while the rest struggled even to produce a circuit-equivalent layout. The case studies spell out why. In the current-starved ring VCO task (Task 9), all three GPT-6 Astra attempts passed DRC, LVS, and simulation, but their aspect ratios ranged from 22.63 to 26.31 against a bound of 4, so they were disqualified on geometry. On the same task, Novara's first attempt hit a ratio of 1.24 and scored 0.9015. In a 30 mA PMOS-pass LDO task (Task 17), both GPT-6 Astra and DeepSeek V4.1 Flash passed electrical verification, yet their power-delivery sub-scores diverged sharply — 0.748 versus 0.199 (per Table 2 and the surrounding text) — because the DeepSeek layout's output routing used only 10 of the roughly 63 via cuts the SkyWater process requires for that current. Both layouts were "electrically fine," but their engineering quality was not.

Credibility Assessment

Two things support trust here. First, grading is not self-reported — it reruns independently in a fresh environment after the agent stops. Second, part of the circuit data comes from external open datasets (Analog Design Bench, LOADBench), so the benchmark's creator did not design every task in isolation.

The caveats are larger. The biggest is a conflict of interest: the benchmark's creator sets the scoring weights and quality definitions, then evaluates its own closed agent, Novara, on that same benchmark and reports it as the top performer. Novara runs entirely on the creator's own cluster with no external API calls, so there is no outside way to confirm it faced the same constraints (the three-hour budget, harness limitations) as the other systems, and the repository isn't public yet. Reproducibility is the next concern — the 13 public model-harness configurations each ran at most three repetitions, with no reported variance or confidence intervals. The task pool (40 tasks) is also narrow, concentrated on analog amplifier-style circuits, so the conclusions don't obviously generalize to digital IC or mixed-signal design. This remains a pre-peer-review preprint. We found no prior work directly contradicting the conclusions within this specific domain — though as the related work below shows, the pattern of "agents produce an artifact but miss the quality bar" recurs across software-focused agent benchmarks too.

Related Work

Read together, analog layout automation looks like it has its circuit-data and simulation infrastructure (LOADBench, Analog Design Bench) in place first, with agent evaluation (AnalogLayoutWorld) building on top of it, while follow-on efforts like TapeoutBench are pushing evaluation scope beyond layout into later steps (routing, tapeout). We did not cite any numbers from the second and third items directly, since egress limits kept us from independently re-verifying them.

Reviewer's Take

First, the number that matters most here for practitioners isn't 0.8154 (Novara's top spot) — it's the gap between 91.1% and 21.3%. A nearly 70-point gap between "the agent produced a GDS file" and "that layout is actually usable" is a warning against judging success on artifact existence alone when deploying agents on physically constrained engineering tasks. GPT-6 Astra's ring-VCO case — electrically perfect, geometrically disqualified every time — shows this gap isn't about agents failing to produce anything; it's about repeatedly tripping on a different acceptance axis.

Second, we think it's premature to read Novara's top ranking as independently verified best-in-class performance. When a benchmark's creator sets the scoring rules and then evaluates its own closed system on that same benchmark, reporting it as the winner, that result should be treated as a vendor claim until the repository is actually public and a third party reproduces it. Notably, Novara's row is missing the resource fields (harness, token counts, tool-call counts — all "—") that let readers compare whether it fought under the same conditions as everyone else.

Third, a number the paper doesn't headline is budget exhaustion: GLM-5.3 Flash hit the three-hour time limit in 51 of 120 attempts (42.5%), versus zero for GPT-6 Astra. That operational predictability gap doesn't show up in the score table, but we'd check it before average score when picking a model for a real pipeline.

Practical Takeaways

  • Report production rate and acceptance rate separately — when evaluating long-horizon agents, always report "did it produce an artifact" and "did it meet the quality bar" as separate numbers. Gaps near 70 points, as seen here, are not unusual.
  • Treat vendor self-benchmark wins as claims, not facts — when a benchmark's own creator sets the scoring rules and reports its own system as the winner on that benchmark, treat it as a reference point only until the repository is public and a third party reproduces it.
  • Weigh budget-exhaustion rate in model selection — similar average scores can hide very different timeout rates (0% to 42.5% here). For operational stability, check this before average score.
  • Budget for a second verification round, not just the first pass — layouts that clear electrical verification still repeatedly fail secondary criteria like geometry and current capacity, so build extra verification rounds into the budget after the first pass.
  • Use open verification tooling for self-checking loops — agents here self-verified using only the open EDA chain (DRC, LVS, simulation), with no benchmark-specific tooling — a workable pattern for cutting human involvement while keeping a quality-assurance loop running.

Conclusion

AnalogLayoutWorld tests whether coding agents can extend into long-horizon engineering work under physical constraints — analog IC layout — using an independent verification pipeline. The results show a clear gap: across 13 public model-harness configurations, 91.1% produce a layout but only 21.3% meet the acceptance bar, and layouts that pass electrical verification still frequently fail secondary criteria like geometry and current capacity. The benchmark creator's own closed system topping that same benchmark, however, is not an independently settled fact until the repository is public and reproduced by a third party — it should be read as a vendor claim for now. The lesson that production and acceptance need to be reported separately converges with Gating Full Pass Rates for Coding Agents: A Design Checklist, which reaches the same conclusion on the software side.

References

Ask AI about this review

The assistant has read this review and the numbers it verified. Ask anything — it answers from the text and says so when something isn't in it.

Loading the chat…