Synthetic Sample Augmentation: The Subgroup Decides

Synthetic sample augmentation fills the cuts a panel cannot deliver. What two published measurements show about subgroup error, and the holdout that settles...

A study is designed with seventeen market areas. Feasibility comes back and three of them will not fill. The options on the table are to drop those three markets, to extend field and pay for the extra weeks, or to read them anyway on a thin base with a footnote nobody opens in the steering committee. Synthetic sample augmentation is the fourth option that has arrived in the last two years, and the argument for it is exactly those three markets.

Key Takeaways

• Synthetic data and synthetic respondents are different jobs with the same name, and the confusion is where most bad decisions on this topic start.
• One agency reported augmenting seventeen market areas after validating ten niche audiences against a holdout of real respondents.
• One panel company measured imputation error above 10 percentage points at the subgroup level, which is the exact level augmentation is sold for.
• Those two results do not contradict each other. They measured different material with different controls.
• The question that settles it is not which model was used. It is what was kept out of the model.

What synthetic sample augmentation actually is

Synthetic data is data produced by an algorithm rather than collected from a person, built by learning the statistical patterns in a real dataset and using them to estimate how similar people would answer. Kantar frames the family as three jobs: augmenting existing data, creating new data, and simulating future scenarios.

An augmented sample is a dataset where some records came from respondents and some were modelled. The modelled records exist to make a specific cut readable, usually a geography, an age band inside a small segment, or a professional profile that the panel cannot deliver in volume.

A holdout sample is the part of the real data that is deliberately kept out of the model, so the model's output can later be compared against answers it never saw. It is the only instrument in this article that produces a number you can defend, and it is the one most often missing.

Those three definitions are worth writing down because the field uses the first term for all three jobs, and the buyer ends up evaluating a persona simulation with the criteria of a statistical augmentation, or the reverse.

How a low incidence cut is filled today

This is the sequence that runs inside enterprise insight teams and agencies right now, before any of this technology enters the conversation.

1. The brief arrives with the cuts the business wants to read: by market, by segment, by profile.
2. The research team turns those cuts into quota targets and sends them to field for feasibility.
3. Feasibility comes back and some cells do not fill. The usual failures are low incidence profiles, such as a medical specialty, a small geography, or a specific B2B buying role. STRAT7 says this plainly even while holding a panel network of over 47 million respondents across 70 markets: some audiences are hard to reach regardless of network size.
4. Someone makes the call in a feasibility meeting. Drop the cut, extend field, raid another budget, or accept a thin base.
5. If the thin base is accepted, the cut is read with a caveat in the methodology slide.
6. Six months later the cut appears in a decision deck with the same visual weight as every other bar on the chart.

Step four is made on judgment. There is no measurement on the table at the moment the decision is taken, which is what makes the arrival of a fourth option so consequential.

A persona is not a sample, and the market uses one word for both

UNIQUE INSIGHT The single most useful thing a research team can do with this topic is to separate two jobs that share a vocabulary.

A synthetic respondent answers a stimulus. You show it a concept, a package route, a claim, and it reacts. The validation question is whether its reaction ranks the routes the way humans rank them. We wrote about that job, and about the fidelity numbers behind it, in Synthetic Respondents Concept Testing.

An augmented sample completes a cut. There is no stimulus. The model is estimating how an underrepresented group would have answered a question that other people in the same study already answered. The validation question is whether the estimated distribution matches the real one in that specific cut.

The two jobs fail differently, so they need different tests. A persona that ranks eight concepts correctly tells you nothing about whether an augmented geography cut is safe to read. A vendor who answers a question about one with evidence about the other is either confused or counting on you being confused.

Two jobs sold under one word

The phrase synthetic data points to two different jobs. On the left, a synthetic respondent answers a stimulus shown to it, such as a concept, a package route or a claim. It is validated by whether it ranks the options the way people rank them, and it fails by putting the routes in the wrong order. On the right, an augmented sample completes a cut using answers that other respondents already gave in the same study. It is validated by its error against a holdout of real answers, measured in that same cut, and it fails by producing a thin cut that reads as stable and is not. A closing line states that evidence about one of these jobs is not evidence about the other.

Same vocabulary, different jobs, different tests. Most bad decisions on this topic start with the confusion between the two columns.

What one house measured when it augmented a geography

Big Village published a working account of augmentation in July 2025, and it is worth reading because it describes the control, not only the result.

The client was a telecommunications brand with a scope covering seventeen Designated Market Areas. A national base was achievable. Readable, statistically stable base within each individual market was not, and in some markets the available panel could not deliver the numbers required for comparisons between markets at all.

Before offering augmentation as a formal service, the agency ran an internal proof of concept. It modelled synthetic data for ten niche audiences and compared the output against a holdout sample of real respondent data, with cuts by age and gender to check consistency across subgroups. It reports that the synthetic data aligned closely with the real sample, and that in several cases the mean absolute error was lower than that of the holdout group itself.

The account also states the boundary out loud: the agency does not use synthetic data to replace real sample, it uses it to close feasibility gaps.

What another house measured when it imputed a survey

Verasight published the opposite-looking result in September 2025, in the second report of a series by G. Elliott Morris, Benjamin Leff and Peter K. Enns.

The base was 1,500 interviews with a nationally representative sample of American adults, run on the company's own online panel. From each respondent's demographic attributes the team assembled a written persona, fed those personas to large language models through their APIs, and asked the models to answer the same survey questions as that person.

In the first report, the models approximated population proportions on questions that are asked constantly and are heavily polarized. Approval of the president landed around 4 percentage points off. On less polarized and newer questions, the models were badly wrong. At the subgroup level, even on the question the models handled best, mean absolute error passed 10 percentage points against the real crosstabs.

The second report tried to fix that. The team took the best performing model and moved to a state of the art one. They invoked chain of thought reasoning in the prompt. They added administrative records of each respondent's actual turnout in the 2024 primary and general elections, plus a real attitudinal answer on tariffs. Under that best case, the models still did not consistently reproduce the real survey results. More than that, the additional information sometimes reduced accuracy.

The report contrasts its findings with a different design. Park and colleagues built agents for 1,052 Americans from two-hour interviews, surveys, or both. On held-out General Social Survey items, those agents reached 83, 82 and 86 percent of the participants' own two-week test-retest consistency, against 74 percent for agents built from demographics alone. Same family of technology, very different material, and a very different result.

Why those two results do not contradict each other

UNIQUE INSIGHT Put the two accounts side by side and the disagreement dissolves into a difference of material and control.

Big Village modelled from respondent level data collected inside the same study, for audiences adjacent to audiences it had actually measured, and it kept real respondents out of the model to check against. Verasight modelled from a written persona assembled out of demographic attributes. In the second report it added administrative and attitudinal fields. Then it asked a general purpose model for an individual's opinion on political questions the model had no study specific evidence about.

One is interpolation inside a measured space. The other is extrapolation into an unmeasured one. Both are called synthetic data.

Three published measurements, three owners

Three panels, each with its own unit. Panel one, personas built from demographic attributes, measured by Verasight in 2025 by Morris, Leff and Enns on 1,500 interviews: error against real results was about 4 percentage points on the most polarized national question, and mean absolute error against the crosstabs passed 10 percentage points at the subgroup level. Lower is better. Panel two, agents built per person by Park and colleagues for 1,052 Americans, scored on held-out General Social Survey items as a share of the participants’ own two-week test-retest consistency: 74 percent for demographics only, 82 percent for a structured survey, 83 percent for a two-hour interview, and 86 percent for interview and survey combined. Higher is better. Panel three, Big Village in 2025 modelled ten niche audiences from study data and compared them with a holdout of real respondents, with cuts by age and gender, reporting error in several cases lower than the holdout group’s own, without publishing a figure. A closing line names what separates the results: the material the model learned from, and what was kept out of it. Every number belongs to the source named beside it and none of them is a Cassi.ai measurement.

The results look contradictory until you read what each model learned from. Interpolation inside measured material and extrapolation outside it are both sold as synthetic data.

The useful conclusion is not that one house is right. It is that the phrase "synthetic sample augmentation" covers practices whose error varies by an order of magnitude. What separates the good end from the bad end is how much real, study specific material the model learned from. The Park result points the same way: interviews beat demographics by a wide margin. And the other half is whether anything was kept back to test it.

The check that decides, and who is supposed to run it

The buyer does not need to adjudicate the methodology debate. The buyer needs one procedure, and it is the same one both accounts above point at from opposite directions.

1. Split before you model. The holdout is chosen and set aside before the model sees anything, not carved out afterwards from what is convenient. For example, hold back a random fifth of the real respondents in each market before any modelling starts.
2. Choose the cut you will actually read. If the decision deck will show market by market, the holdout comparison has to be market by market. A match on the national total says nothing about the cell that will drive the investment. The same logic sits behind AI agent evaluation without an answer key: check the output where the decision lands, not on average.
3. Compare on that cut and report the error there. Mean absolute error against the holdout, in the cut, in percentage points.
4. Publish the error next to the number. The augmented figure travels with its error or it does not travel.
5. Repeat when the wave changes. A model calibrated on last wave's material is a claim about last wave.

UNIQUE INSIGHT Every one of those five steps is ordinary survey practice with a different subject. None of them requires believing anything about artificial intelligence. The reason they are skipped is not technical. Running a holdout means paying for real respondents you then refuse to use in the headline. Nobody wants to explain that cost to a client who came here to avoid a cost.

Who answers when the augmented cut reaches the slide

PERSONAL EXPERIENCE The failure we watch is not a modelling failure. It is a labelling failure, and it happens after the analysis is finished.

The augmented cut goes into the chart with the same colour, the same bar width and the same decimal places as the cuts that came from people. The distinction lives in the methodology appendix. The deck is then cut down for the executive readout, and the appendix is the first thing dropped. A quarter later somebody sets regional investment against a bar without knowing that half of it was estimated, and there is no longer anyone in the room who could tell them.

The fix is boring and it is in the chart, not in the model. We made the same argument about dashboards built from the decision: the reader of the slide is the reader who counts. Augmented cells carry a visual mark that survives every copy of the file, and the error against the holdout is written in the footnote of the chart itself rather than in a separate document. If a number cannot carry its own provenance into a screenshot, it will eventually be read without it.

Where Cassi.ai comes in

@Cassi.ai is a software engineering company specialized in the pains of market research, innovation and insights, founded by Cassiano Albuquerque. We work with enterprise research teams and agencies across banking, health, retail, media and consumer goods.

Synthetic Lab is a Cassi.ai product that turns the research already done into synthetic personas of the target, for synthetic focus groups and quantitative studies with quotas.

Our work on this axis runs on the client's own material, and the discipline above is the part we do not negotiate. What the model learned from is written down. What was kept out of it is written down. And the comparison is made in the cut the committee is going to read.

We govern this with a four level ladder of trust: possibilities, qualitative attitudes, quantitative reads, macroeconomic simulation. Most client work sits on the first two rungs, and we say which rung out loud, because presenting a screening instrument as a quantitative read is how this field loses the argument it is currently winning.

When augmentation is the wrong instrument

UNIQUE INSIGHT There are cuts that should not be augmented at any level of model quality, and a supplier who will not name them is selling rather than advising.

The first row is the one teams try to argue their way around, because it is the row where the need is most acute. Acute need and available evidence are unrelated quantities.

FAQ

What is synthetic sample augmentation?

It is the practice of adding model generated records to a real dataset so that a specific cut becomes readable, typically a geography or a low incidence profile that the panel could not fill. It is distinct from replacing a study with simulated respondents, and the houses that offer it, including Big Village, state that boundary explicitly.

What is the difference between synthetic data and synthetic respondents?

A synthetic respondent answers a stimulus you show it, and is validated by whether it ranks options the way people do. Synthetic data in the augmentation sense estimates how an underrepresented group would have answered a question your study already asked, and is validated against a holdout of real answers in the same cut. Same vocabulary, different jobs, different tests.

How accurate is synthetic data at the subgroup level?

It depends on whether the model is interpolating inside measured material or extrapolating outside it. Verasight measured mean absolute error above 10 percentage points at the subgroup level when personas were assembled from attributes, against roughly 4 points on the most polarized national question. Big Village reported error comparable to, and in several cases lower than, its own holdout when modelling audiences adjacent to ones it had measured.

How do you validate an augmented sample against real respondents?

Set the holdout aside before modelling, choose the cut the decision will actually be read on, compute mean absolute error against the holdout in that cut, and publish the error alongside the number. A match on the total sample is not evidence about a cell.

When should a research team refuse to augment a cut?

When there are no real respondents from that audience to learn from. When the question is new and nobody in the study answered it. And when the output carries legal or regulatory weight, because the identity of respondents has to be auditable. For regulated work, the written AI policy research agencies now need is the place to state that rule.

An augmented cut quoted without the error it scored against a holdout is an opinion with decimal places. The decimal places are what make it travel.

If your last feasibility meeting dropped a market or accepted a thin base, the useful question for the next one is not whether synthetic augmentation works. It is who is holding the real respondents you would test it against, and whether anyone has asked them to keep some back.

About the author

Cassiano Albuquerque is the founder of @Cassi.ai. He has spent twenty years in market research operations, from fieldwork and tabulation to the reports that reach steering committees, and now builds AI systems for research teams. To discuss a feasibility problem on your own study, contact him on LinkedIn or see the cassiai.com portfolio.

Published by Cassi.ai. Read the full article at https://www.cassiai.com/blog/synthetic-sample-augmentation.