Multi-Country Survey Testing: One Questionnaire, Nine Routes

Multi-country survey testing checks the translation and rarely the routes. How to confirm every country version keeps the master's filters, quotas and screen...

The master questionnaire was signed off in English three weeks ago. Since then it has become nine questionnaires. Each was translated, reviewed by a native speaker and programmed in its own language, and each will open in its own country on Monday. The master link was tested carefully. The other eight were opened, clicked through once and approved because the text read well. Multi-country survey testing usually stops there, and the defect that survives is almost never in the words.

Key Takeaways

  • Multi-country survey testing has two quality controls that rarely meet: translation review checks the text, and programming QA usually checks one version.
  • Established translation procedures such as TRAPD confirm meaning and equivalence. They do not run the programmed version to confirm its routes and quotas.
  • Every country version has to be tested against the same map of filters, skips, quotas, screenouts, blocks and validations as the master.
  • A local difference is only acceptable if it was documented before the test. Otherwise it is a defect.
  • Automated navigation handles the volume, and a person keeps low confidence cases, chained rules, dynamic pages, anti-bot protection and methodological decisions.

What a multi-country survey test has to verify

Three terms carry this article, and all of them get used loosely in project kickoffs.

A master questionnaire is the source version of a multi-country study, usually written in one language, that defines the questions, answer lists and logic every country version is meant to reproduce.

A country version is the translated and programmed questionnaire that respondents in one country actually see. It inherits the master's logic and may carry documented local changes, such as a national education list.

A route is the sequence of screens a respondent passes through, set by the answers they give as filters, skips and quotas send them one way or another. Two respondents in two countries giving equivalent answers should travel equivalent routes.

Multi-country survey testing is the check that each country version sends equivalent respondents down equivalent routes, and fills the same quota cells, as the master questionnaire does.

The standard for equivalence is well established on the text side. The European Social Survey describes its translation goal as asking the same question in every language, rather than rendering the words literally. That is the right goal. Routes need the same treatment, and they rarely get it.

How translation is checked today

The strongest public procedure is TRAPD, used by cross-national programmes such as the European Social Survey.

TRAPD is a team translation procedure made of five linked steps: Translation, Review, Adjudication, Pre-testing and Documentation. Two translators work in parallel, a reviewer compares versions, and an adjudicator decides.

The ESS Round 11 Translation Guidelines add translation verification by an external provider and consistency checks across rounds. They also note that verification, assessment or pretesting may lead to revisions, which then call for further adjudication and testing. The guidelines accept that some items are country-specific by design, and give national education questions as the example.

AI has entered this stage too. Exploring the Potential Role of Generative AI in the TRAPD Procedure found that a model given a zero-shot prompt flagged real translation problems, including inconsistent conceptualization, formality issues and concepts with no equivalent. The AAPOR report on responsible AI in survey research accepts that role, with a condition: final adjudication still depends on expert review and cognitive testing.

A chat assistant can draft nine translations in an afternoon, which makes the adjudication step more important rather than less. We covered where a general assistant stops in why ChatGPT is not a research platform.

All of this is good practice, and all of it looks at text.

The gap nobody owns

UNIQUE INSIGHT Translation review confirms that question 14 means the same thing in Portuguese as in English. Nobody in that process opens the programmed Portuguese link to confirm that question 14 still sends the respondent to block C.

The two controls sit with different people. Translators and reviewers own meaning. The survey programmer and whoever tests the link own logic, and they usually test the master, or the first country to be programmed. The route of each remaining country has no owner.

That gap is where multi-country defects live, because each country version is a second implementation of the same specification, often built by a different programmer. We described what a single untested path costs in survey testing before fieldwork. In a nine-country study, that problem is multiplied by nine and then compounded by every local change.

Where the routes diverge between countries

UNIQUE INSIGHT The divergences follow a small number of patterns, and knowing them tells the tester where to look first.

Local answer lists feeding a filter. Education, region and income lists are rebuilt per country. If a filter in the master says "higher education goes to block D", the country list has to map its own categories onto that rule. A category left unmapped sends respondents the wrong way without an error.

Quota cells defined on local categories. A quota by region or age band is set per country. If the cell definitions in the programmed version do not match the sampling plan, the study terminates people who should qualify, or overfills a cell.

Answer order changed by translation. A list sorted alphabetically in the master gets re-sorted in the new language. Any rule that refers to a position rather than a code now points at a different answer.

Screeners adapted to the market. A category that does not exist in one country gets removed or replaced, and the screenout logic around it has to change with it.

Validation formats. Dates, postal codes, phone numbers and decimal separators differ, and a validation rule copied from the master can reject a correct local answer.

Text that carries logic. Piped text, conditional wording and gendered forms can break a display rule when the sentence structure changes.

None of these shows up in a translation review, because the text in each case can be perfect.

One rule from the master, three country versions

An illustrative example, not data from a real study. At the top, a master questionnaire rule says respondents with higher education go to block D. The same respondent profile runs through three country versions. In country version A the local education list is mapped to the rule and the profile lands in block D: same route, marked as passed with screen evidence attached. In country version B the local list adds a vocational category that an approved local note sends to block E: a different route that is in the local notes, marked as a documented variation with the approver named. In country version C one local category was left unmapped and the profile skips block D without any error: a different route that is not in the notes, logged as a defect with the question, the rule and screenshots. A closing line says a local difference passes only if it was written down before the test ran.

Versions B and C both change the route. Only the note written before the test separates an approved adaptation from a routing defect.

The method, step by step

PERSONAL EXPERIENCE This is the sequence we follow for a multi-country test. It starts from documents and ends with evidence per question and per rule.

  1. Read the master questionnaire and the technical documentation of the study: sampling plan, quota specification, country notes.
  2. Build one map of every filter, skip, quota, screenout, block and validation in the master, each written as a testable rule.
  3. Turn the map into a scenario matrix: each rule becomes a row with the respondent profiles that exercise it, and each country version becomes a column.
  4. Write down every documented local difference per country before any test runs, with who approved it.
  5. Run assisted navigation through each country version, profile by profile, recording the screen, the answer and the rule at every step.
  6. Compare each observed route with the route the map predicts, and classify every difference.
  7. Send low confidence results, chained rules, dynamic pages, anti-bot blocks and methodological questions to a person.
  8. Feed each confirmed correction back into the map, so the next round and the next wave start from it.

Step four is the one that decides whether the test is useful. Without it, every legitimate adaptation looks like a defect and the report drowns the real ones.

The scenario matrix

A scenario matrix is a table in which each row is a rule from the questionnaire map and each column is a country version. Every cell records whether that version applied the rule as the master does, with the evidence attached.

Read row by row, it shows whether one rule survived translation everywhere. Read column by column, it shows whether one country version is safe to open.

If the observed route in a country versionThenBecause
Matches the master route for that profileMark the rule as passed, with the screen evidenceEquivalence is shown, not assumed
Differs, and the difference is in the documented local notesMark it as a documented variationThe adaptation was approved before the test
Differs, and nothing in the notes explains itLog a defect with question, rule and screenshotsAn undocumented difference is a routing error until proven otherwise
Cannot be observed because the page blocked or changedSend it to a person, do not force it forwardA forced run destroys the evidence
Depends on a methodological choice, such as a local quota definitionSend it to the study leadTooling cannot decide what the study should measure

PERSONAL EXPERIENCE The fourth row follows the rule we set for all questionnaire testing: when a run cannot pass a screen, it stops and records the blocker.

The scenario matrix: rules in rows, country versions in columns

An illustrative scenario matrix, not data from a real study, showing five of nine country versions. Rows are rules from the master questionnaire map and columns are country versions A to E. Each cell carries one of four states: passed, local note, defect, or to a person. Row one, screenout, an under-age respondent ends the survey: passed in all five versions. Row two, filter, category users see block B: passed in A, C and E, local note in B, defect in D. Row three, quota, the region cell counts the complete: passed in A, B and D, defect in C, local note in E. Row four, skip, none of the above skips question 9: passed in A to D, sent to a person in E. Row five, validation, a local postal code is accepted: defect in A, passed in B to E. Row six, piped text, the brand name fills the follow-up question: passed in A, B, D and E, sent to a person in C. Two reading notes say that a row shows whether one rule survived in every version and a column shows whether one country version is safe to open. A closing line says defects go to the programmer with evidence, and cells marked for a person are read first, before anything is sent back.

Country C has a defect and an open question, so it does not open on Monday. Country A fails one rule that every other version passes.

Why one run per country is not enough

The navigation in step five is increasingly done by agents that operate a browser. The research on those agents says to treat a single run with suspicion.

On the Reliability of Computer Use Agents, from April 2026, shows the same agent succeeding at a task and failing when the identical task is run again. The authors recommend evaluating with repeated executions and preferring strategies that stay stable across runs.

The environment adds its own noise. WAREX introduced network instability, pop-ups and site changes into three web agent benchmarks, and task success dropped significantly. Survey pages carry all three, plus consent banners that differ by country.

Even the verdicts need checking. How Benchmarks Mis-Score Computer-Use Agents audited 150 trajectories scored as failures and found 15.3 percent of those verdicts were wrong. Of those, 10.7 percent were evaluator false negatives and 4.7 percent were broken tasks.

For multi-country testing this means three things. Each profile runs more than once per country. Each reported failure carries the screen and the step. And a person looks at failures before they reach the survey programmer, so a flaky run does not send a correct version back for rework.

What stays with a person

Automation covers the volume: nine versions, dozens of profiles, repeated runs. It does not cover five kinds of case, and the test plan should name them in advance.

  • Low confidence results, where the evidence does not clearly show which route the respondent took.
  • Chained rules, where a filter depends on an answer several screens back that itself depended on a quota.
  • Dynamic pages, where content loads or changes after the answer, so the screen captured may not be the screen the respondent saw.
  • Anti-bot protection, which exists to stop exactly this kind of navigation and should never be bypassed.
  • Methodological decisions, such as whether a local quota definition still measures what the study was designed to measure.

The AAPOR report gives a reason that goes beyond survey logic. Its recommendation not to assume performance generalizes uses translation as the example. A model can translate a questionnaire into Spanish and Chinese with reasonable accuracy and perform substantially worse in another language. The report asks for a task-specific test for each new application. A test that worked on the English master has proven nothing about the Korean version.

The same logic sits behind AI agent evaluation without an answer key: the agent does not grade its own work.

Corrections that carry into the next round

A multi-country study rarely has one round of testing. Programmers fix, versions come back, a wave two follows a wave one. The expensive mistake is to test each round from zero.

Every correction a person confirms becomes part of the map. If one country's region list needed a mapping fix, the next round checks that mapping first, there and in every country with a similar list. AAPOR's recommendation that evaluation continues through production, because models and templates change underneath, applies to questionnaires just as well.

The vendors building agents for questionnaire testing have reached the same idea for a single version. Questra states that approved test recordings become regression tests, a feature listed as coming soon when we checked in September 2026. Pollia states that its agents reconcile the intended, programmed and rendered survey. It also reports links that remembered previous answers and produced hollow runs. Alchemer's test response generator can cover each branch, and its documentation notes that JavaScript actions do not fire in that mode. None of the three pages addresses language or country versions.

Where the research on adapting questionnaires stands

Language models are also being used to adapt questionnaires across contexts, not only to test them. Exploring LLMs for Automated Generation and Adaptation of Questionnaires tested a pipeline with 238 participants in the United States and 118 in South Africa. Participants found the LLM-adapted questions slightly clearer and less biased than conventional versions.

That result makes the routing question more pressing. If adaptation gets faster and looks better, more country versions will be produced with more local change. Every local change is a place where a route can quietly diverge from the master.

Where it goes wrong in practice

PERSONAL EXPERIENCE The scene I recognize is a country version that passes every text review and opens with a quota that never fills.

The translation was excellent. The regional list had been rebuilt by the local team, correctly for the country, and the quota specification still referred to the master's region codes. Respondents from two regions qualified, answered the whole questionnaire and were counted in no cell. Field ran for days before the incidence report looked strange enough for someone to open the link and walk it.

Nobody made a translation error. Nobody made a programming error in the usual sense either. The local list and the quota rule were each right on their own, and nobody had tested them together in that country. Everything downstream inherited the gap, down to the final deck, the argument we made in report automation starts upstream.

Where Cassi.ai comes in

@Cassi.ai is a software engineering company specialized in the pains of market research, innovation and insights, founded by Cassiano Albuquerque. We understand research first, and we automate projects with or without language models.

Survey QA is a Cassi.ai product that tests programmed surveys automatically, walking every screen on desktop and mobile and returning a verdict screen by screen, with screenshots.

Survey QA is the product built for this pain. The questionnaire is programmed and someone has to test the script before it goes to field. Most teams still do it by hand: an analyst walks the consistencies, filters, completes, layout, bugs and grammar, then walks it again with different answers. That consumes days of a senior analyst on checking work, and the error that escapes costs far more later.

In a multi-country study we apply the same discipline to every country version. One map from the master, local differences written down first, evidence per question and per rule, and a person on the exceptions. Each run is isolated and classified by outcome, and a run that meets a blocker stops and records it rather than forcing the screen. More in the cassiai.com portfolio.

FAQ

How do you test a survey in multiple languages?

Build one map of the master questionnaire's filters, skips, quotas, screenouts, blocks and validations. Write down approved local differences per country. Then navigate each country version against that map, recording evidence per question and rule, and send exceptions to a person.

Does a translated survey need separate testing?

Yes. Translation review confirms meaning, and procedures such as the ESS translation guidelines do that well. The programmed country version is a separate implementation, and its routes and quota cells can differ from the master even when every sentence is correct.

What is the TRAPD translation method?

TRAPD stands for Translation, Review, Adjudication, Pre-testing and Documentation. It is a team procedure in which parallel translations are compared by a reviewer and settled by an adjudicator, used by cross-national programmes such as the European Social Survey.

How do you check skip logic in a translated questionnaire?

Design respondent profiles that exercise each skip, run them through the programmed country version, and compare the observed route with the route the master predicts. Pay particular attention to rebuilt local answer lists, re-sorted options and conditional text.

Can automated survey testing handle multiple countries?

It can carry the volume, with limits. Research on computer use agents shows the same agent can pass and then fail an identical task, so each profile needs repeated runs. Dynamic pages, anti-bot protection, chained rules and methodological choices stay with a person.

A questionnaire that reads the same in nine languages has been checked for meaning nine times and for routing once. The routes are what the data is made of.

About the author

Cassiano Albuquerque is the founder of @Cassi.ai. He has spent twenty years in market research operations, from fieldwork and tabulation to the reports that reach steering committees, and now builds AI systems for research teams. To test the country versions of your next multi-country study against its master questionnaire, contact him on LinkedIn or see the cassiai.com portfolio.

Published by CassiAI. Read the full article at https://www.cassiai.com/blog/multi-country-survey-testing.