Source Document
Stefan Hut, Lorenzo Masoero, "Can AI Agents Simulate A/B Test Outcomes? A Validation Framework for Agentic Experimentation", arXiv:2608.02345 [cs.CL; cs.AI; stat.AP], submitted 2026-08-03, DOI 10.48550/arXiv.2608.02345. The arXiv comments field records acceptance as a workshop paper.
This is a workshop-accepted preprint rather than a journal-refereed article. The conflict of interest is visible in the document itself: both authors use amazon.com addresses, the validation data is the historical experiment log of "a large e-commerce service," and the simulator is described only as an "internal simulation platform." The foundation model's name, version and vendor are not disclosed. That is workable for a methods contribution, but it rules the results out as vendor-neutral evidence of what agent simulation can do.
Study Overview
The question is framed the way practitioners actually ask it: if you dress an LLM agent in a customer profile and simulate an A/B test before it runs, can you trust the prediction enough to save traffic and calendar time? The authors formalise this as a Simulated Randomized Controlled Trial (S-RCT) and decompose prediction error into two terms.
The decomposition reads (estimated ATE − true ATE) = (the simulator's population-level prediction − true ATE) + (the finite-sample estimate − the simulator's population prediction). The first term is approximation error: structural bias from the behavioural model, the persona rendering and the conditioning. The second is subsampling error: variance from drawing a finite number of agents. Separating them is the paper's real contribution.
Validation runs on 67 historical marketing A/B treatment pairs from that e-commerce service, with click-through rate on a high-traffic product surface as the outcome. Each simulation uses Nagt = 1,000 agents, and every agent is paired one-to-one with a real customer who took part in the original experiment (an "agentic twin," sampled on trigger). What the simulator sees is only a persona built from observable profile features. The design is within-subject: each agent is exposed to both arms, so composition differences between arms disappear.
Calibration proceeds in two phases. Phase 1 withholds the pre-period outcome from the feature set and renders the control context for both arms — an A/A simulation with zero treatment effect by construction — then fits a calibration function mapping raw simulator output to realised pre-period outcomes via Platt scaling under log loss. Phase 2 runs the simulator with full features and real assignment and passes the output through that fitted function. The train/test split is temporal: pre-period versus treatment period.
Key Results
Uncalibrated baseline accuracy (Table 3, within-subject, 1,000 agents) is below. The random floor differs by metric, so the two must be read together.
| Metric | Value (SE) | Random floor |
|---|---|---|
| Sign overlap | 0.70 (0.03) | 0.50 |
| Sign overlap, bias-corrected | 0.80 (0.03) | 0.50 |
| Sign accuracy | 0.70 (0.06) | 0.50 |
| Launch alignment | 0.41 (0.06) | 0.33 |
| Corrected MSE | 0.0222 (0.0108) | — |
| Mean absolute error | 0.0893 (0.0178) | — |
Launch alignment discretises the posterior into harmful, inconclusive and launch, which is why its floor is 0.33 rather than 0.50. It is not comparable side by side with the sign metrics.
The two headline improvements attack different error terms. The claim that Platt-scaling calibration compresses squared prediction error by roughly 77× was measured on 16 experiments from the benchmark, not all 67, with the calibration function fitted separately per experiment on that experiment's own pre-period. The within-subject design contributes a different gain: standard errors shrink by about 2.4× on average, and sign accuracy rises from 65% under a between-subject split to 0.70.
The authors' own limitations are stated plainly — magnitude overshoot persists even when the sign is right; the benchmark is single-domain (marketing CTR); correlated responses from a shared model may understate variance; and the evaluation is retrospective, scored against outcomes already known. They describe predicting which customers will trigger into an experiment as "the main missing piece for deployment."
Credibility Assessment
What earns trust here is the honesty of the scoring frame. Error is split into approximation and subsampling so you can see which fix touches which term; every accuracy metric carries its random floor; every estimate carries a standard error. Binding each agent to a real participating customer avoids the familiar trap of inventing a population and then scoring predictions against it. The limitations section is not defensive.
Three layers of caution apply. Sample: 67 pairs is not generous, and the marquee 77× figure rests on 16 of them; the 0.06 standard error on launch alignment is a consequence of the same scale. Reproducibility: with the model unnamed, the platform internal and the data proprietary — and both authors employed by the company that holds it — no external party can replicate this. Generalisation: marketing creative tests are a domain of mostly small effects, and nothing here shows the same accuracy for pricing, delivery promise or search ranking, where the behavioural model differs.
Reviewer's Judgement
First, the number that touches decisions is not 77× but 0.41. Launch alignment — the metric that separates ship, hold and kill — sits at 0.41 against a 0.33 floor with a standard error of 0.06, roughly 1.3 standard errors above chance. That is a signal worth noting, not a substitute for a launch decision. Sign overlap at 0.70 against a 0.50 floor is genuinely useful, which points to the right deployment: use the simulator to rank and screen candidates, not to decide them.
Second, the 77× should be read as an artefact of procedure rather than a property of the model. If the calibration function is fitted per experiment on that experiment's own pre-period, this is not a general-purpose simulator; it is a pipeline that requires a usable pre-period for every test. New surfaces and new products — precisely where teams most want a cheap preview — have no pre-period to fit on, and a 16-experiment evaluation is thin ground for carrying the multiplier into another domain.
Third, the decomposition quietly refutes a common expectation: that running more agents makes the simulation more accurate. Adding agents and moving to a within-subject design both reduce only the subsampling term (the observed gain being about 2.4× on standard errors), while approximation error — how well a persona of observable features stands in for a real customer — is untouched. The authors' note that magnitude overshoot survives calibration is consistent with exactly that split.
Fourth, the retrospective framing deserves more weight than it usually gets. The authors call triggering-population prediction the missing piece for deployment; in practice it is half the job. No number in this paper speaks to accuracy when you do not yet know who will enter the experiment.
Putting It to Work
- Deploy at the screening stage only — let simulation rank candidate variants and filter obvious losers; leave ship/no-ship to live traffic. The 0.70-versus-0.41 gap is the reason.
- Require pre-period A/A calibration — treat uncalibrated effect sizes as overstated, and decline to quote simulated numbers for tests with no usable pre-period.
- Report the metrics separately — never collapse sign, magnitude and decision accuracy into one score; publish each against its own floor (0.50 / — / 0.33).
- Re-validate on your own history — instead of citing the paper's multiplier, replay a few dozen of your own completed experiments through the same protocol and build a domain-specific scorecard.
- Know what more agents buy — scaling agent count shrinks standard errors, nothing else. If bias is the suspect, spend the budget on the persona feature set.
Conclusion
The value of this framework lies less in its predictive performance than in how it grades itself. By splitting error into approximation and subsampling and attaching a random floor to every metric, it lets us describe the state of agent simulation without inflation: it gets the direction right and the decision wrong — sign overlap 0.70, launch alignment 0.41. For the experiment-design side of the same problem, see AI Landing Page Experiment Design.