Photo By: Shubham Dhage
For decades, companies have relied on A/B testing to answer a deceptively simple question: what happens if we change something? Change the price. Alter the promotion. Rewrite the message. Move the placement. Then expose real customers to both versions and measure the difference.
The method works because it observes actual behavior. But it also has a cost. Experiments consume traffic, engineering resources, time and, in some cases, revenue that could have been generated under the existing strategy.
A new line of research raises an intriguing possibility: what if an AI system could simulate an experiment before a company runs it?
Researchers at Amazon Science recently tested that idea using a framework they call a Simulated Randomized Controlled Trial. The study evaluated AI-generated simulations against 67 historical marketing A/B tests to determine whether agents could predict how real participants would respond to different treatments.
The results suggest both the potential and the problem. A foundation-model simulator showed promise, but it systematically overestimated the magnitude of experimental effects. After researchers calibrated the simulation using pre-experiment behavioral data, its squared prediction error fell by roughly 77 times. A within-subject design, in which simulated agents experienced both sides of the experiment, further reduced standard errors.
The implication is not that companies can simply replace A/B tests with chatbots. It is that simulation may become a valuable screening layer before companies commit to an experiment in the real world.
The Problem With Asking AI to Play a Customer
It is easy to imagine how a synthetic consumer simulation might work. Give an AI model a description of a customer, a product and a proposed intervention. Ask it what the customer would do.
For example: “Act as a price-sensitive consumer. Would you purchase this product if the price increased by 10%?”
The response may sound convincing. It may even resemble what a real customer might say. But plausibility is not the same as behavioral validity.
A language model is trained to generate likely sequences of language. That makes it remarkably capable at producing coherent explanations of what a hypothetical customer might think. It does not, by itself, establish that the simulated customer will behave like the population being modeled when an economic variable actually changes.
Real consumers operate under constraints that a persona prompt does not automatically capture: budgets, competing products, previous purchases, price sensitivity, promotions, availability and changing market conditions.
The Amazon study illustrates the distinction. The researchers found that a baseline simulation could reproduce some directional patterns, but it systematically overshot effect magnitudes. In other words, an AI could produce a plausible simulation without necessarily producing a well-calibrated estimate of what would happen in the real experiment.
That is the central challenge for synthetic consumer simulation.
Calibration Matters More Than Personality
The most interesting part of the emerging research may therefore have less to do with making synthetic customers sound more human and more to do with making their behavior empirically grounded.
The Amazon researchers used a two-phase pre-period calibration protocol before simulating the historical experiments. That calibration substantially reduced prediction error, suggesting that the value of an AI simulation depends not only on the underlying model but also on how it is conditioned against observed behavior.
That distinction matters. A synthetic consumer does not become useful simply because its persona is detailed. It becomes useful when the assumptions governing its behavior are connected to evidence about how comparable consumers have actually behaved.
For businesses, that could change the role of simulation. Instead of asking an AI to invent a likely customer response, companies could use simulation to pressure-test a decision against historical patterns, observed market conditions and quantitative relationships before deciding whether an expensive live experiment is justified.
The live experiment does not necessarily disappear. Its role changes.
From A/B Testing to Experiment Triage
Traditional experimentation creates a binary structure: run the test or don’t run it. Calibrated simulation introduces another possibility. Simulate first. Test selectively.
A consumer company considering a major pricing change, for example, could use a simulation to evaluate a range of potential scenarios before allocating live traffic to an experiment.
The same principle could apply to marketing spend, promotions, product launches or other commercial decisions.
The goal would not be to produce a synthetic replacement for the customer. It would be to narrow the decision space.
That distinction is important because simulation has its own uncertainty. A model can be calibrated against historical behavior and still fail when conditions move outside the range represented in its data.
The Amazon researchers explicitly identify limitations in the current approach and stop short of treating the simulation results as a substitute for experimentation.
For now, that makes the most defensible future for AI simulation a complementary one: use models to identify promising interventions, estimate likely outcomes and determine which questions are worth testing in the real world.
Where Causal Modeling Enters
Moving beyond simple persona prompts requires models that account for the relationships among variables, observed outcomes, and alternative decisions.
Evaluating an intervention requires answering counterfactual questions: Would raising a price reduce unit demand enough to hurt contribution margin? Would additional advertising generate incremental sales or simply reach customers who would have purchased anyway?
Answering these questions requires combining several quantitative disciplines. AI agents can process unstructured market signals, while specialized models evaluate the underlying business question using causal inference, elasticity modeling, forecasting, or stochastic simulation.
This dual-layer framework is beginning to appear in enterprise software. For example, platforms like Kapnova, co-founded by CEO James Sun and CTO Shenbo Xu, whose academic research at MIT focused on causal inference in observational data, are building architectures that separate market signal ingestion from quantitative evaluation. Rather than relying on text generation to predict business outcomes, these systems use autonomous agents to monitor market signals while passing underlying decisions to econometric and Monte Carlo models before capital is committed.
The A/B Test Moves Downstream
A/B testing retains an advantage synthetic simulation cannot easily replicate: real customers provide real outcomes. Eliminating live experimentation remains premature.
However, AI simulation could change what companies ask experiments to do. Instead of testing every plausible idea against live traffic, businesses can use calibrated simulations to eliminate weak scenarios and reserve real-world experiments for decisions where remaining uncertainty matters most.
The future of enterprise AI experimentation may not be a choice between synthetic customers and real ones. It will likely be a system where AI helps identify which questions deserve to be asked in the real world, and quantitative methods evaluate the choices.