Synthetic Data Privacy Research: Who Carries the Proof

Synthetic data privacy research points one way: output built from personal data stays personal until someone shows the risk is remote. What that asks of agen...

The agency holds three waves of a tracker, a folder of interview transcripts and a customer file the client sent for sample matching. Someone proposes generating a synthetic copy of all of it, so analysts, a coding partner and a new vendor can explore without touching anything real. The proposal passes in five minutes, because synthetic sounds like anonymous. What synthetic data privacy research has found, in regulator guidance and in attack papers, runs the other way. It moves the burden of proof onto whoever pressed generate.

Key Takeaways

• Synthetic data generated from personal data is presumed to stay personal data until someone shows the re-identification risk is remote.
• UK regulator guidance says synthetic data is anonymous only if the source records cannot be inferred from it.
• An ICML 2025 paper showed that synthetic text alone, with the model kept private, still leaks which records trained it.
• Differential privacy closed that leak in the same paper, and the regulator warns it costs utility.
• 2026 guidance from three research associations adds a second owner: a synthetic output can still carry the client's property.

What the privacy claim actually says

Synthetic data is data produced by an algorithm that learned the patterns of a real dataset, built so that analysis of the copy gives results close to analysis of the original. Kantar presents it as a complement to real data, used to augment, to create and to simulate.

A privacy enhancing technology is a technique that lets an organisation analyse or share information while reducing what can be learned about the people in it. The UK Information Commissioner's Office files synthetic data in this family, next to differential privacy and homomorphic encryption.

Re-identification is the act of linking a record, or a fact about a person, back to that person. It does not require a name in the file. A rare combination of age, postcode and occupation is often enough.

A membership inference attack is a test that tries to tell whether a specific record was part of the data a model learned from. It is the standard way security researchers measure whether synthetic output leaks its source.

The sales version of the claim is short: synthetic data contains no real people, so privacy is handled. The regulator version is longer, and every clause in it matters.

Where the source material comes from in a research operation

UNIQUE INSIGHT Most sectors that discuss synthetic data can at least imagine a source that is not about people. Market research cannot. Every file a synthetic model would learn from in an agency was collected from, or about, a person.

1. Interview and focus group transcripts, which carry voice, employer, family detail and the story that makes one participant recognisable to a colleague.
2. Panel and tracker bases, with demographics, geography and behaviour at respondent level across waves.
3. Client customer records, sent for sample matching, weighting or enrichment, and governed by the client's contract with its own customers.
4. Raw field data before cleaning, including open ends, where people write things nobody planned to collect.

All four are personal data at source. That is the starting position for everything below, and it is why the question in the next section is not academic for a research team.

Is synthetic data personal data?

The most careful legal review of the question was written for health data, and it transfers to research almost word for word. The PHG Foundation report, commissioned by the UK medicines regulator and published in May 2023, reviewed guidance from European data protection authorities.

Its finding is that regulators follow what the authors call an orthodox approach. Synthetic data is treated as a privacy enhancing technology. If the input or training data are personal data, models and output data are presumed to remain personal data unless effective anonymisation can be demonstrated with confidence.

The GA4GH policy brief of April 2024 restates the presumption in stronger terms. The output stays personal unless it is shown with a high degree of confidence that re-identification threats are minimal and well safeguarded.

The regulator's own text is more measured, and it points the same way. The ICO guidance on synthetic data, published as final guidance in June 2023, asks directly whether synthetic data is anonymous. Its answer is that it depends on whether the personal information behind it can be inferred from the synthetic data itself.

The same page adds the sentence that should sit on the first slide of any proposal. The more synthetic data mimics real data, the more useful it is, and the more likely it is to reveal someone's personal information. Utility and exposure rise together.

The PHG report is also honest about the cost of this position. It says the approach is likely to push developers toward risk averse choices and places a high burden on them, and it recommends that regulators consider a shifted presumption for some forms of synthetic data. That debate is open. Until it closes, the presumption is the working rule.

Three families of generation hide behind one label

"Synthetic" names an output, not a method. The GA4GH brief separates three families of generation: statistical methods, noise based methods and machine learning methods. The output of any of them can be fully synthetic, where every variable is generated, or partially synthetic, where only some are.

The ICO's draft chapter on privacy enhancing technologies, consulted in September 2022, drew the same line between partial and full synthesis. The final guidance adds a hospital example: replace location and admission time with generated values, and keep the real causes of admission.

UNIQUE INSIGHT Translate that into a tracker. A partially synthetic base that keeps real verbatims and real purchase behaviour, and regenerates only age and region, has changed the least identifying columns and kept the most identifying ones. It is still called synthetic in the deck. The PHG review records the same distinction from the legal side: a fully synthetic set with no one to one match to training records poses different risks from a hybrid of real and generated rows.

So the first questions a data protection officer asks are about the family, the generated variables and the training base.

What the attack papers show

For years the privacy argument ran on intuition, because testing leakage from a released synthetic file was hard. That changed with two papers that attack the output alone, without access to the model that made it.

The Canary's Echo, presented at ICML 2025, fine-tuned a language model on private text, generated synthetic text from it, and gave the attacker only the synthetic text. The authors report that membership inference still worked. On three labelled datasets the attacks reached an AUC of 0.74, 0.68 and 0.77, where 0.50 is a coin flip.

AUC is a score for how well a test separates two groups, here records that trained the model from records that did not. A score of 0.50 means the test is guessing, and 1.00 means it never misses.

The same paper measured the gap between the two kinds of attacker. The authors report that a planted record would need to occur eight times more often to be as exposed through the synthetic text as through direct access to the model. That is real protection, and it still leaves a leak.

What an attacker learns from the synthetic text alone

Two panels from The Canary’s Echo, presented at ICML 2025, where a language model was fine-tuned on private text and the attacker saw only the synthetic text it generated, never the model. Panel one shows membership inference AUC, where 0.50 is a coin flip and 1.00 never misses: a coin flip reference at 0.50, then 0.74 on a reviews dataset called SST-2, 0.68 on a news dataset called AG News, and 0.77 on an inference dataset called SNLI, and finally 0.50 for the strongest attack when the model was trained with differential privacy at a privacy budget of epsilon 8. Panel two says a planted record had to occur 8 times more often to be as exposed through the synthetic text as through direct access to the model. A closing line says less leakage than the model is still leakage. Every number belongs to the source named beside it and none of them is a Cassi.ai measurement.

Synthetic text leaks less than the model that wrote it, and more than nothing. Differential privacy is what brought the attack back to a coin flip.

A second paper moved the same question to tabular data, which is closer to a tracker base. Privacy Auditing Synthetic Data Release through Local Likelihood Attacks proposes an attack that needs no knowledge of the generator. The authors report it outperforms earlier attacks, especially at low false positive rates. They trace the leak to generative models overfitting particular regions of the training data.

UNIQUE INSIGHT Low false positive rates are the regime that matters for a research file. An attacker does not need to expose everyone. Confidently exposing the one senior buyer in a small B2B cell is enough to break a confidentiality promise.

The trade nobody mentions in the deck

Differential privacy is a mathematical framework that adds calibrated noise during generation, so that the output changes very little whether or not any single person's record was included. The size of the allowed change is called the privacy budget.

It works. In The Canary's Echo, fine-tuning with differential privacy dropped the strongest attack on synthetic text to an AUC of 0.50, which is guessing. The ICO names it as a way to protect outlier records from linkage attacks, and immediately states the price. It may reduce the utility of the information and introduce unpredictability in its characteristics.

How to DP-fy Your Data, a practical guide revised in July 2026, lays out every component such a system needs. The list runs from handling and preparing the sensitive data, through tracking how the budget is spent, to empirical privacy testing of the output. The guide is written for engineers. Its length is the point for everyone else: this is a system somebody has to build and run.

UNIQUE INSIGHT Here is the collision research teams walk into. The records most in need of protection are the outliers, and the outliers are often the reason the study was bought. The small segment, the rare profile, the one region with a thin base. Noise that protects them also blurs them. We made the subgroup argument from the accuracy side in synthetic sample augmentation, and it returns here from the privacy side. The ICO says as much: in some cases you will not reach an acceptable balance, for example when the purpose needs the outliers.

The client's property does not dissolve either

Privacy law is about the participant. The research contract adds a second owner, and generation does not remove that one either.

In June 2026, Research Live reported joint guidance from MRS, ESOMAR and the Insights Association on client data in AI tools. Contracts should state that project data belongs to the client. De-identified sets, aggregated summaries and synthetic outputs can still carry the client's strategic thinking. Agencies are asked to keep approved tool lists, data classification and retention rules.

UNIQUE INSIGHT That closes the second exit. Even when a synthetic base clears the re-identification test, it is still derived from what the client paid to learn. Handing it to a vendor or reusing it to train a model for another account is a contract question before it is a privacy one. The same logic applies to twins built from a brand's own interviews, which we covered in customer digital twin research.

What changes when the burden moves

UNIQUE INSIGHT The practical difference between "synthetic solved it" and "we have to show it" lands in four places.

The timeline gains a step. A re-identification assessment runs before the synthetic file leaves the team that built it, and it has to be redone when a new wave, a new linked source or a new attack changes the picture. The PHG report is explicit that the controller keeps the assessment under review.

The contract gains a clause. It says what the synthetic output may be used for, by whom, and whether it counts as client property.

Somebody signs. An assessment nobody owns is a draft. Someone with authority over data protection puts a name on the conclusion, and that person is rarely the analyst who proposed the idea.

The proposal changes tone. The line promising that synthetic data removes privacy risk comes out. In its place: we will generate a synthetic file, test it for re-identification, and tell you what the test found.

Where the burden of proof falls

Two readings of the same synthetic file. The top row is the reading in the sales deck: a real base of transcripts, panel waves and client records, which is personal data, becomes a synthetic copy, and the copy is treated as settled and shared with vendors and partners because the word sounds anonymous. The bottom row is the reading in regulator guidance: the same real base becomes a synthetic copy that is presumed to stay personal data; a re-identification test is run on the output, recorded and signed; and only then is the file released, within what the test and the client contract allow. The source line says the presumption is as reviewed by the PHG Foundation in 2023 and restated by the GA4GH policy brief in 2024, and that ICO guidance from 2023 says synthetic data is anonymous only if the personal information behind it cannot be inferred from it. A closing line says the word synthetic moves nothing, and the test and the record move the burden.

The top row is how synthetic data is usually proposed. The bottom row is what the guidance expects before the file leaves the team that made it.

Three lines to write before the model runs

The discipline fits on half a page. It has to be written before generation, because afterwards the details are reconstructed from memory.

1. Write down the source base. Which files, which waves, which variables, collected under which consent and which client contract, and where the base physically sat while the model learned from it.
2. Write down the generation technique. The family, whether the output is fully or partially synthetic, which columns were generated, and whether differential privacy was applied with which budget.
3. Write down the re-identification test. Which attack or assessment was run against the output, on which records, what it found, and who signed the conclusion.
4. Write down the permitted use. Who may receive the file, for what purpose, and when the assessment will be repeated.

Where the model runs belongs in the first line, and it is not a detail. A generator trained inside the agency's own environment and one trained through an outside service are different data flows, which we covered in open weight models in market research.

Who answers when the client's counsel asks

PERSONAL EXPERIENCE The question never arrives during the project. It arrives six months later, by email, from someone in the client's legal team who was not in the kickoff.

It asks what the synthetic file was trained on, whether it can be re-identified, and who received it. The analyst who proposed the idea has moved accounts. The vendor who received the file has a copy nobody logged. The answer the agency can give is a recollection of a meeting where everyone agreed synthetic meant safe.

A recollection does not answer counsel. A dated document with the three lines above does, even when what it says is uncomfortable. The same discipline sits behind the written AI policy research agencies now need: the record is written when the work happens, not when somebody asks.

Where Cassi.ai comes in

@Cassi.ai is a software engineering company specialized in the pains of market research, innovation and insights, founded by Cassiano Albuquerque. We work with enterprise research teams and agencies across banking, health, retail, media and consumer goods.

Synthetic Lab is a Cassi.ai product that turns the research already done into synthetic personas of the target, for synthetic focus groups and quantitative studies with quotas.

On this axis the work starts by writing down three things before any model runs: which base the model learned from, which generation technique was applied, and which re-identification test was run against the output. That is working discipline. We give no legal opinion and issue no certification, and the deliverable says so.

Our synthetic work runs on the client's own material, and the Super Agents line can run inside the client's environment, on premises, in a private cloud or air gapped. That narrows where the source base travels. It does not answer whether the output is personal data, and we do not pretend it does. The rest is in the cassiai.com portfolio.

When a synthetic copy is the wrong instrument

UNIQUE INSIGHT Some uses survive the presumption easily. Others should not be attempted, whatever the generator.

The fourth row is where teams argue longest, because the rare segment is often the reason the synthetic copy was proposed. The ICO guidance already concedes that for some purposes no acceptable balance exists.

What this article is not

UNIQUE INSIGHT This is not legal advice. The sources above are themselves careful to say the legal position is unsettled: the PHG report calls for regulators to revisit the presumption, and the GA4GH brief expects uncertainty to remain for some time.

What the article offers is the operational side. Write the three lines, test the output, and take the question of whether your synthetic file is personal data to your own counsel and your data protection officer. They decide. The record is what lets them decide quickly.

FAQ

Is synthetic data still personal data under GDPR?

It can be. The approach European authorities follow, as reviewed by the PHG Foundation, presumes that output built from personal data remains personal data. The presumption holds unless effective anonymisation can be demonstrated with confidence. The decision on a specific file belongs to the controller and its counsel.

Does synthetic data remove the need for consent?

Not by itself. Generating synthetic data requires processing the real data first, and the ICO notes that this may involve processing personal information. The lawful basis for that step still has to exist, and the client contract still governs what the output may be used for.

Can a synthetic dataset be re-identified?

Published attacks say yes, to a measurable degree. The Canary's Echo reached AUCs of 0.68 to 0.77 in telling training records apart using only synthetic text, where 0.50 is guessing. A later paper showed a similar attack on tabular synthetic data without any access to the generator.

What do regulators say about synthetic data as anonymisation?

The ICO says whether synthetic data is anonymous depends on whether the underlying personal information can be inferred from it. It also warns that the more the output mimics real data, the more likely it is to reveal someone. It lists membership inference and model inversion among the known attacks.

Does differential privacy solve it?

It is the strongest defence in the published work, and it has a price. Differential privacy dropped the strongest attack in The Canary's Echo to chance. The ICO warns it may reduce utility, and for purposes that depend on outliers an acceptable balance may not exist.

A synthetic file with no record of what trained it is a promise nobody can check. The promise gets checked eventually, usually by someone who was not in the room.

If your team is about to generate a synthetic copy of a client base, write the three lines first and send them to counsel with the proposal. The ones worth working with will be glad you did.

About the author

Cassiano Albuquerque is the founder of @Cassi.ai. He has spent twenty years in market research operations, from fieldwork and tabulation to the reports that reach steering committees, and now builds AI systems for research teams. To discuss how a synthetic data proposal on your own study should be documented, contact him on LinkedIn or see the cassiai.com portfolio.

Published by Cassi.ai. Read the full article at https://www.cassiai.com/blog/synthetic-data-privacy-research.