Synthetic users aren't users. If you work in research, you will have heard that this year. You may have said it.
It is the sceptics' answer to a pitch that keeps arriving: research without the people.
Chris Chapman, who co-chairs the Quant UX conference, put it as bluntly as anyone. Synthetic data is not data, he wrote, and using it is not research.
Most of the profession agrees. When User Interviews surveyed 150 researchers this spring, four in five used AI regularly in their work. Eight per cent used synthetic participants.
I agree too. Synthetic users aren't users. That's the whole point.
The sceptics treat that line as the end of the argument. The vendors treat it as a flaw to fix, with accuracy scores and digital twins. Both are asking how close the copy can get to a person. It is the wrong question.
Below the screen
We are building a conversational interface to guidance on using AI in government. You ask a question, it finds the guidance that covers it, and it answers from that. Then you can ask a follow-up.
Before any person used it, I asked a model to test it. What we ended up with is not quite what the vendors sell. It is a synthetic usability test: a model that uses the service the way an expert reviewer would, and plays made-up people as it goes. The made-up people are the synthetic users. The test around them is an instrument.
The first run found thirteen problems. Two were serious, and neither would show on a good day.
One was a wait of up to thirty seconds, with nothing on screen to say anything was happening.
The other was worse. The question you typed was copied into the tab title, and the page's analytics recorded tab titles. This was on a page that told you not to type anything personal.
No participant would ever have reported that. People don't read their tab titles, and nobody reads the traffic leaving their browser. The model could, because it can look below the screen: at the code, and at the network.
Then I asked which method it had used. "I didn't use a skill," it said. It had worked from memory: one reviewer, one pass. Jakob Nielsen found that a single reviewer finds only about a third of the problems in an interface.
Less like a person
So we gave it a method. It played people with jobs to do: a supplier starting on a new project next week, and someone at an arm's length body asking whether the guidance covered them at all. A second reviewer worked from the same evidence without seeing the first list. Microsoft's guidelines for AI products sat beside Nielsen's heuristics.
It found twenty-nine problems, and four of the five worst were new. Two came from the personas. Asked whether a team could use Claude Code, it treated "Claude" as a blank to fill in. Asked as the arm's length body, it sent them to the wrong organisation's incident process.
No automated test finds those. They needed someone who wanted something.
Then we made it less like a person again. It now records the network on every run. Every question gets asked twice, since the service doesn't give the same answer each time. And before each task it writes down the right answer for that persona, because the second reviewer had praised an answer that was wrong for the person asking.
Each change made it a better test, and a worse impression of a user.
What the people found
A week later, six people tried it for real: developers, a data scientist, a delivery manager and two business analysts. Three work on it, and three are closer to the people it is for. Six is enough to find problems, not to measure them. Each asked their own questions.
They found a different kind of problem. One asked a follow-up, and it had forgotten the first question. One asked for a longer answer and was refused, because the request had been read as harmful.
One business analyst asked whether AI could get them into sensitive data they weren't allowed to see. The answer was an error: "Sorry, there is a problem with the service." The question that most needed a clear no got a broken page.
Only one problem turned up in both rounds: lists came out as one long paragraph. Everything else the people found came from something they wanted: to keep talking, to get more, or to ask the awkward question.
The personas never did. When we told the synthetic test to push back, it found the service defending a wrong answer. But someone had to think of telling it.
The personas stay polite for a reason. Models are tuned on what people prefer, and people favour familiar text, so the model drifts towards the typical answer. Jiayi Zhang and colleagues traced this mode collapse in 2025. A synthetic persona is the typical user: it asks the typical question, politely, and moves on. In May, Yu Lu Liu and colleagues found simulated users gave chatbots far more praise than real people do.
Real people are not typical, not even of themselves. When Aaron Fisher and colleagues measured the same people again and again, each person varied two to four times more than the groups did. They called it a lack of group-to-individual generalisability: what is true of a group may not be true of anyone in it. The person who asked for a longer answer was one person, on one day, with a reason of their own.
Keep them apart
So whose reading counts? Each one counts for something different.
A synthetic usability test can clear the obvious before anyone's time is spent: the blank screen, the leak, the broken error. It can't tell you what people need. It can't stand in for them at an assessment, or for the people who have no other way to get the service.
The more a synthetic user is sold as a person, the less it is worth.
The easy answer is to do both: run the model, then ask real people to check its work. That wastes the people.
Keep them apart instead. Let the model be a model. Give it a method, the code and the network, and have it ask everything twice. Let people be people, and give them no script.
I nearly gave them one. The first draft of our survey, written by a model, told people what to ask first. I cut it. People don't arrive with our question. So the survey asks them to bring their own, and to ask a few.
The models will keep getting better at sounding like us. A defence of research that needs them to stay bad will not last. Defend people for what a script can't do: they bring their own question, and they push back when the answer is wrong.
The memory and the refusal are both fixed now. It remembers what you asked, and you can ask it for more. It took a person to ask.