ESOMAR 20 Questions for AI Vendors: What to Ask First

The ESOMAR 20 Questions turn an AI vendor's accuracy claim into something a research team can check: against what reference, on which task, in which language...

An insights team is about to sign for an AI tool. It codes open ends, or summarizes interview transcripts, or fills a sample with synthetic respondents. The vendor ran a clean demo on its own data. The deck says the model is accurate, sometimes with a percentage beside the word. IT has sent its security questionnaire, procurement has compared the commercial terms, and the research director signs. In most buying processes I have watched, nobody asked the question that decides whether the tool belongs in a deliverable. Accurate against what? The ESOMAR 20 Questions exist to make that question, and nineteen others, routine.

Key Takeaways
ESOMAR issued its 20 Questions to Help Buyers of AI-Based Services in March 2024, grouped in five sections that end with human oversight and data governance.
Question 7 asks how the supplier validates output for accuracy and whether that process is documented. For synthetic data, it asks for a comparison with real-world results.
The NIST AI Risk Management Framework says accuracy should always be paired with realistic test sets that represent the conditions of expected use.
The AAPOR task force report of May 2026 asks for a task-specific test dataset for each new application, because performance does not carry over between uses.
The MRS guidance on AI requires researchers to demonstrate the validity of AI-generated insights and to let clients check output quality.

What the ESOMAR 20 Questions are

The ESOMAR 20 Questions are a checklist for researchers who commission AI-based services, published by ESOMAR, the global association for research, insights and analytics. The checklist itself was issued in March 2024. Its stated aim is transparency, trust and confidence between buyers and suppliers, with intellectual property, privacy and AI laws respected.

Its scope is AI used while doing research: research products, participant interaction, research operations and analysis. General business software and the use of research data to train a client's own AI are left out.

The questions sit in five sections. The first is about the supplier, the next two about the offer, and the last two about the people and the data behind it. Each question carries a short context note telling the buyer what a good answer should reveal.

Suppliers are invited to publish their answers. ESOMAR's list of companies that have answered, dated December 2025, names twenty of them, from Ipsos and Toluna to Yabble and Thematic. Published answers are a starting point. ESOMAR's own disclaimer says the answers are the companies' responsibility and that it cannot vouch for their veracity.

How AI tools still get bought in research teams

PERSONAL EXPERIENCE The sequence I see most often in agencies and client-side insights teams has nothing to do with research method. It runs like this.

Watch a demo on the vendor's sample data, chosen by the vendor.
Send the security questionnaire from IT and wait for the data processing agreement.
Compare the commercial terms against one or two alternatives.
Run a pilot on one live project, usually the one with the friendliest deadline.
Extend the tool to every study once the pilot did not embarrass anyone.

Every step there is reasonable, and none of them tests the output against anything the team already trusts. Teams still sign on a demo and a pilot. The pilot proves the tool runs. Whether it is right was never measured.

That is how a team ends up with several isolated tools and no evidence about any of them. We described where that leaves a team in AI adoption in market research: past the pilot.

Why "accurate" is not an answer

An accuracy figure has three hidden parts, and a vendor deck usually shows none of them. The reference it was compared with. The task it was measured on. The conditions of the test.

Accuracy is the closeness of results to the true values, or to the values accepted as being true. That is the ISO/IEC TS 5723:2022 definition quoted in the NIST AI Risk Management Framework. The second half of that definition is the part buyers skip. Somebody decided what counts as true.

A reference set is that decision made concrete. It is a collection of cases whose correct answer was settled in advance, usually by trained people, and the tool's output is scored against it. Coding work calls it a gold standard. Machine learning calls it a test set.

The NIST framework is direct about what has to travel with the number. Accuracy measurements should always be paired with clearly defined, realistic test sets that represent the conditions of expected use. Details of the test method belong in the documentation too. A percentage without that is a claim about the vendor's test, made in the vendor's conditions.

The test set can be wrong as well. The NIST profile for generative AI, NIST AI 600-1, warns that datasets commonly used to benchmark models can contain label errors. Those are the benchmarks teams use to choose a model.

Then the task. Section 4.1.2 of the AAPOR task force report gives an example any multi-country team will recognize. A model may translate an English questionnaire into Spanish and Chinese with reasonable accuracy, then do substantially worse in another language. Its recommendation is a task-specific test dataset for each new application. Section 4.1.3 adds time. Providers update models, prompts get revised, and performance seen at launch should not be assumed to hold.

Who answers when the number was wrong

The vendor does not sign the deliverable. The agency does, and the codes of conduct say so.

The MRS Guidance on Using AI and Related Technologies sums up its ethics as legal, transparent, considered and trustworthy, and turns that into rules. Practitioners must allow clients to arrange checks on the quality of AI outputs. They must be able to demonstrate the validity of insights, conclusions and recommendations generated with AI. They must give clients enough information to assess the validity of results produced on their behalf. They must keep records that make AI use auditable, replicable and traceable.

Read those rules against the buying sequence above. When a client questions a finding, a research director who signed on a demo has only the vendor's percentage on file, measured on data the agency never saw.

What an accuracy claim has to name

A grid comparing three vendor answers to the question of how accurate the AI is, against five things an accuracy claim has to name: the reference it was scored against, who built the reference, the buyer's task and language, the model version and test date, and a measure chosen in advance. The first answer, "Our model is highly accurate", names none of the five. The second, "Accurate against our labelled benchmark", names the reference, who built it and the measure, and leaves out the buyer's task and language and the model version and test date. The third, "Agreement with your coders, on your task", names all five. A closing band quotes the NIST AI Risk Management Framework 1.0, section 3.1: accuracy measurements should be paired with realistic test sets that represent the conditions of expected use. The figure is an illustration of criteria and contains no measured numbers.

Only the third answer can be checked by the team that signs the deliverable. The first two describe a test the vendor ran on its own terms.

The questions that carry the most weight for a research buyer

UNIQUE INSIGHT All twenty questions deserve an answer in writing. For a research agency or an insights team, eleven of them decide whether output can go into a deliverable. They cluster into five things to establish before signing.

Accurate against what. Question 7 asks how output is verified and validated for accuracy, whether the process is documented, and how validity is measured. Question 8 asks for the limitations and how they are mitigated. For synthetic data, question 7 adds two more. What validation compared synthetic outputs with primary research or real-world results? How does the supplier keep data from natural persons apart from synthetic data? Our article on synthetic respondents in concept testing shows what that comparison looks like.

Which model, and whose. Question 5 asks what model is used, and whether it was built in house or comes from a third party. The AAPOR report notes that model versions, configurations and system-level instructions are often not disclosed by vendors. Ask anyway, and ask to be told when the version changes.

What happens to client data. Question 6 asks whether client data or synthetic data goes into training and whether the buyer can opt out. Questions 18 and 19 ask who owns the data and what the supplier may not do with it, including where it is held. Question 20 asks who owns the output. Consider a coding tool that learns from every project. It can carry one client's codeframe into the next client's study, and the agency would never see it happen.

Where the human sits. Questions 10 to 12 cover transparency, ethical principles and human oversight. Human in the loop is a framework for integrating human judgement into an AI process, as ESOMAR defines it in a footnote. It can mean ongoing supervision, human overrides, final decisions made by people, or exception handling. Ask which of those applies, at which step, and who does it.

Where the inputs came from. Question 14 asks whether the origin and processing of training or input data is documented. Data lineage is that record of where each input came from and who processed it, so that a result can be traced back to its sources.

The 20 Questions, and the ones to ask first

A map of the ESOMAR 20 Questions to Help Buyers of AI-Based Services, issued in March 2024, laid out as five columns, one per section, with labels paraphrased from the text. Section A, Company profile: question 1, Experience in AI for research; question 2, Where AI helps research; question 3, Problems met in deployment. Section B, Explainable and fit for purpose: question 4, Role of AI in plain terms; question 5, Which model, built or bought (highlighted); question 6, Data that trains the model (highlighted). Section C, Trustworthy, ethical, transparent: question 7, How accuracy is validated (highlighted); question 8, Limitations and mitigation (highlighted); question 9, Duty of care to people. Section D, Human oversight: question 10, Clear when AI is used (highlighted); question 11, Ethical principles in practice (highlighted); question 12, Human oversight in the loop (highlighted). Section E, Data governance: question 13, Training data quality; question 14, Data lineage (highlighted); question 15, Privacy notice; question 16, Privacy law and consent; question 17, Resilience and security; question 18, Data ownership (highlighted); question 19, Data sovereignty (highlighted); question 20, Who owns the output (highlighted). Eleven questions are highlighted in orange as the ones that decide whether output can go into a deliverable, which is Cassi.ai's reading and not ESOMAR's: 5, 6, 7, 8, 10, 11, 12, 14, 18, 19 and 20. A closing band says to ask for every answer in writing, starting with question 7: how the output was validated, against what, and whether it is documented.

The highlighted questions are our reading of what a research buyer needs before signing. The full checklist still deserves an answer in writing.

What an acceptable answer contains

UNIQUE INSIGHT The table pairs the answer a buyer usually hears with what that answer leaves out and what a usable answer would contain.

ESOMAR question	Answer you will often hear	What it leaves out	What an acceptable answer contains
7, validation	"Our model is highly accurate"	Reference, task, language, date	The test set, who built the reference, the agreement measure, the result per task and language, the model version and test date
8, limitations	"It works for any research use"	Where it fails	Named tasks and conditions where performance drops, and what the supplier does about it
5, model	"Proprietary AI"	Which model runs underneath	Model family, provider if external, and how version changes are announced
6 and 19, client data	"Your data is secure"	Whether it trains anything	Whether client data enters training, the opt out, the storage region, written into the contract
12, human oversight	"Humans are always in the loop"	Which human, doing what	The step where a person reviews, overrides or decides, and who that person is
14, lineage	"Trained on carefully curated data"	Where it came from	Documented sources for training and input data, and how they were processed

Question numbers and wording come from the ESOMAR 20 Questions. The phrasing of typical answers and the pass criteria are ours.

How to test an accuracy claim before you sign

The cheapest validation is one you run on your own past work, because there the reference already exists and you trust it.

Pick a closed project where people already produced what the tool would produce. For example, the human-coded open ends from last year's tracker, or a summary checked by the analyst who ran the interviews.
Give the vendor the input without the answers and get the tool's output back.
Score that output against your reference with a measure chosen before you look. For coding, the usual choice is kappa, which is an agreement statistic that discounts the agreement two coders would reach by chance. We set out the coding side in the evidence chain for qualitative coding.
Repeat for every task, language and market you plan to use. AAPOR section 4.1.2 says performance does not travel between them.
Write the reference, the measure and the minimum result into the contract. The NIST generative AI profile lists, as action GV-6.1-006, contract clauses that let an organization evaluate the processes and standards of its AI suppliers.
Rerun the test when the vendor changes the model, and on a schedule. AAPOR section 4.1.3 treats evaluation as ongoing while the tool is in production.

Vendors that answer question 7 well will find this test easy. The ones that resist it have told you something useful.

Where Cassi.ai comes in

cassiai.com is a software engineering company specialized in the pains of market research, innovation and insights, founded by Cassiano Albuquerque after twenty years in research operations. We work with research agencies and corporate insights teams.

The AI Innovation and Automation Workshop is a Cassi.ai service that maps real work by area in three one-hour sessions, measures AI maturity and ends with a prioritized automation and AI plan.

The pain we hear most in AI maturity conversations is the one above. A team subscribes to several AI tools, each bought separately. None of them knows the business, and nobody has evidence on file about any of them. Our maturity work starts by mapping what a team already uses, step by step. The questions in this article are the ones to put to each tool on that map. Where the reference set does not exist yet, building it from the team's own past projects is the first real deliverable.

The same questions apply to us. Our own products, from survey testing to synthetic personas, should be bought this way, and a buyer is right to put question 7 to us first. A single model carries its own systematic tilt, and the vendor who built a tool is not the one to certify it. The rest of our work is in the cassiai.com portfolio.

FAQ

What are the ESOMAR 20 questions for AI?

A checklist ESOMAR issued in March 2024 for researchers who commission AI-based services. The 20 Questions are grouped in five sections: company profile, explainability and fitness for purpose, trust and ethics, human oversight, and data governance.

What should an AI vendor's accuracy claim include?

The reference the output was compared with, who built it, the task and language, the measure used, the model version and the test date. The NIST AI RMF says accuracy should always be paired with realistic test sets and details of the test method.

Does the ESOMAR checklist apply to synthetic data vendors?

Yes. Question 7 carries a supplementary block for synthetic data. It asks what validation compared synthetic outputs with primary research or real-world results, and how the supplier tells data from natural persons apart from synthetic data.

Who is responsible for validating AI output in a research project?

The research supplier that delivers it. The MRS guidance requires practitioners to demonstrate the validity of AI-generated insights and to allow clients to arrange checks on output quality.

How often should an AI research tool be revalidated?

Whenever the model, the prompts, the task or the market changes, and on a schedule in between. The AAPOR task force report treats evaluation as ongoing for as long as the tool is in production.

A percentage on a vendor slide describes a test somebody else ran, on data somebody else chose. The research team that signs the deliverable needs its own test, on its own work, before the tool touches a client study.

If your team already uses an AI tool on client work, find the answer to question 7 in the contract file. If it is not there, the first test to run is on a project you closed last quarter.

About the author

Cassiano Albuquerque is the founder of cassiai.com. He has spent twenty years in market research operations, from fieldwork and tabulation to the reports that reach steering committees, and now builds AI systems for research teams. To discuss how to test the AI tools your team already uses against your own past work, contact him on LinkedIn or see the cassiai.com portfolio.

Published by Cassi.ai. Read the full article at https://www.cassiai.com/blog/esomar-20-questions-ai-vendors.