Survey Programming Errors: When Word and the Script Split

Many survey programming errors start as edits to the questionnaire in Word that never reach the live script. How the split happens and how to compare version...

The client approved version three of the questionnaire on Tuesday, and the programmer started building the script that afternoon. On Wednesday the brand manager asked for one more brand in the awareness list and a softer wording on the purchase question. On Thursday the research team moved the usage block ahead of the attitudes block. Each change went into the Word file with tracked changes and left by email. On Monday the survey goes live with version three, half of version four and none of version five. Plenty of survey programming errors look like this. The code does exactly what a document said. It was just an older document.

Key Takeaways

• Many survey programming errors start as edits to the questionnaire document after programming has begun. Each edit reaches the script only if someone remembers to send it and retest it.
• Decision Analyst's published QA process starts programming once the questionnaire is finalized, then checks the program, runs 100 artificial respondents and reviews the live topline after the first 12 hours.
• Qualtrics keeps a dated version history of the script, and Word can compare two versions of the document. Neither records which document version a script implements.
• The General Social Survey's November 2024 release notes list programming errors corrected in later data releases, including one on a variable that other questions skip on.
• The fix is a comparison of rules, version by version: filters, quotas, codes and piping, checked against the script before sign-off.

How a questionnaire becomes a script

Drive Research puts the job in one line: survey programming transforms a questionnaire from a document into a working survey that people can interact with. The document is usually a Word file written by the research team and approved by the client. The working survey is a script on Qualtrics, Decipher, Forsta or whichever platform the study runs on.

A few terms carry the rest of this piece. A survey script is the programmed version of the questionnaire, the thing respondents actually see. Skip logic is the set of conditions that send a respondent past or into a question or block, based on earlier answers. Piping is the insertion of an earlier answer, such as a brand name, into the text of a later question. Answer codes are the numbers behind each answer option, which filters and quotas read.

Careful agencies document the handover. Decision Analyst's quality assurance page describes a questionnaire review and a project walk-through with one person from every department. Then comes the line that matters here: "Once the questionnaire is finalized, programming the survey in Logician® begins."

After that come a review of the program, an automated search for 25 common questionnaire errors, a simulated topline with 100 artificial respondents, and a review of the live topline after the first 12 hours of fieldwork.

Every one of those checks assumes the questionnaire stopped changing at the word "finalized".

Every check after programming reads the finalized file

One agency's published quality assurance sequence in six steps, each with a note on the right about what it reads or compares. Step one, questionnaire review: quotas, skips, validations, piping and wording in the document; it reads the questionnaire document. Step two, project walk-through: one person from every department reviews the details before programming; it reads the questionnaire document. Step three, questionnaire finalized: programming of the survey begins; from here on every check assumes this file stops changing. Step four, survey program review: logic, quotas and text, plus a computer search for 25 common errors; it compares the script with the finalized questionnaire. Step five, test topline: 100 artificial respondents run through the logic and the bases are checked; it compares simulated bases with the finalized questionnaire. Step six, live topline: after the first 12 hours of fieldwork, bases and text are checked again; it compares real bases with the finalized questionnaire. A closing band says a change made after step three enters this sequence only if someone sends it to programming. Source: Decision Analyst, Quality Assurance page; the steps and numbers are theirs and the notes on the right are Cassi.ai's reading.

The process is careful, and every step after programming checks the script against one file. A later version of that file is outside it.

How changes travel today

PERSONAL EXPERIENCE In most operations I have worked with, the questionnaire keeps moving for days after programming starts. The client circulates it internally and comments come back. The research executive edits the Word file with tracked changes, saves it as v4, and emails the programmer with "small changes, highlighted in yellow". The programmer applies what is highlighted. Then v5 arrives, sometimes from someone else on the team, sometimes as a reply to an older thread.

What travels well is what is visible on the page: a new answer option, a reworded question. What travels badly is what lives between questions. Codes renumbered when an option was inserted. A filter that still reads Q12, when Q12 now sits after the block it was meant to control. A quota cell defined on an age band that changed in the screener.

The tools do help, separately. Word can compare two versions of a document and show every difference as revision marks. Qualtrics saves a dated version of the survey each time it is published, with an optional description. Publishing only affects new respondents, and responses in progress stay on the old version. So the document has a history and the script has a history. Nothing connects the two. Most teams still keep that connection in an email thread.

Two version histories, no line between them

An illustrative sequence in two rows. The top row is the questionnaire in Word: version 3 on Tuesday, approved by the client, with programming starting from this file; version 4 on Wednesday, with one more brand in the awareness list and softer purchase wording; version 5 on Thursday, with the usage block moved ahead of the attitudes block. The bottom row is the script on the platform: built from version 3, with filters, quotas and codes as written on Tuesday; version 4 applied where highlighted, so the new brand is in and the softer wording is not; and live on Monday with version 3, half of version 4 and none of version 5. Version 3 reaches the script fully, version 4 is partly sent and version 5 is never sent. A closing band says each history is complete on its own and neither says which document version the script implements. The sequence was written for this article and is not a measured case.

Word keeps the document history and the platform keeps the script history. The link between them usually lives in an email thread.

Why the mismatch escapes the test

Because the test checks the script against whatever document the tester has open, and only on the paths the tester walks. A mismatch introduced in v5 is invisible to someone testing against v4, and a filter that broke when a block moved only shows on the path that passes through it.

DataDiggers names the mechanism. Rushed programming and logic changes at the eleventh hour lead to logic loops, broken skip patterns and missing routing, especially when the changes are not retested across all paths. Broken skip logic, in the same piece, takes respondents to irrelevant questions or past required ones, which distorts response paths and base sizes.

Even the General Social Survey, one of the most carefully documented surveys in the United States, finds these late. Its release notes for the 1972 to 2022 data, Release 4, dated November 2024, list several programming errors corrected in later releases. In one, a programming error put some cases into the no answer and don't know codes of PARTNERS. Other questions skip on PARTNERS, so the variables downstream were affected too. In another, respondents who answered "No religion" on one of two versions of a question, assigned by questionnaire form, were coded "No religion" on both.

The failure modes come from DataDiggers and the checks in Decision Analyst's process. The detection column is our own reading.

Who answers when the version and the script disagree

The person who signs off on the questionnaire for fieldwork answers for it. That person is often testing against the wrong version. Manual testing already costs days of a senior analyst, because every combination of filters and quotas opens a different path through the script. Redoing all of it for a change described as small is the step that gets skipped.

PERSONAL EXPERIENCE I have seen a study's fieldwork saved by a test on a questionnaire that was already approved. The errors were found before launch, the script went back to programming, and the team had to go back to the client and reopen a conversation everyone thought was closed. Uncomfortable, and far cheaper than finding the same errors in the tables.

The worst version of this defect is the quiet one. A respondent routed by an old filter finishes normally, the record looks complete, and the base sizes only look odd weeks later. The soft launch, covered in what to check in the first data file, can catch some of it. By then the slice of real respondents has already answered the wrong version.

Compare version by version against the rule map

The comparison that catches this is between rules, not between words. A rule map is the list of every filter, quota, validation and piping instruction in the questionnaire, each with the question it reads and the condition it applies. Traceability is being able to say, for each rule in the script, which version of the document it came from.

UNIQUE INSIGHT Word's compare view shows that a sentence changed. It does not show that the filter on Q18 now points to a question asked later. A rule map does, because it turns each version of the document into a list that can be compared line by line. This is the sequence we recommend.

1. Number and date every version of the questionnaire, and freeze the one that goes to programming.
2. Extract the rule map from each version: filters, quotas, validations, piping and answer codes.
3. Compare the rule maps between versions, so the change list names rules, not paragraphs.
4. Send the programmer the rule changes, with the document version they belong to.
5. Write in the script's version note which document version it implements.
6. Retest every path that touches a changed rule, and every path downstream of it.
7. Sign off only when the script's version note matches the latest approved document.

Step five is the cheapest and the one most teams skip. It turns "which version is live?" from a search through email into a line in the platform.

Public statistics already writes this into its standards. The US Census Bureau's quality standard for data collection instruments requires verifying that an instrument meets all its specifications, including skip patterns, wording, fills and edits. It also requires keeping the documentation needed to replicate how the instrument was developed, specifications and testing results included.

Where Cassi.ai comes in

@Cassi.ai is a software engineering company specialized in the pains of market research, innovation and insights, founded by Cassiano Albuquerque after twenty years in market research. We automate research projects with or without language models.

Survey QA is a Cassi.ai product that tests programmed surveys automatically, walking every screen on desktop and mobile and returning a verdict screen by screen, with screenshots.

Survey QA is our product for testing the programmed script before fieldwork. It reads the questionnaire, the quota plan and the script as one contract, and plans one route per objective. Each route is walked on desktop and on mobile, as a respondent would walk it, and the report shows what broke, with the screen and the step. Because the questionnaire document is an input, a new version of the document is a new contract, and the routes run again against the live script. The reasoning behind testing every route instead of one click-through is in survey testing before fieldwork.

None of this needs a language model to decide anything. Comparing two rule maps and walking the paths they imply is engineering with a defined contract, the difference we drew in why a general chatbot is not a research platform.

FAQ

What are the most common survey programming errors?

Broken skip logic, missing routing, wrong answer codes behind filters and quotas, and text piped from the wrong question. DataDiggers links many of them to late changes that were not retested across all paths.

Why do questionnaire changes not reach the programmed survey?

Because the document and the script are versioned in different tools. Word and Qualtrics each keep their own history, and neither records which document version the script implements.

How do you control questionnaire versions during programming?

Number and freeze each version, compare rule maps between versions, and write the document version into the script's version note before sign-off.

Does a soft launch catch a version mismatch?

Sometimes, as odd base sizes or a missing option. By then real respondents have answered the wrong version, so the comparison belongs before launch.

Before the next round of changes

UNIQUE INSIGHT A script can pass every test and still be the wrong questionnaire. Testing proves the script matches a document. Only traceability proves it matches the right one.

Open the live script of your current study and find the line that says which version of the questionnaire it implements. If there is no such line, the next small change from the client is where your next programming error will come from.

About the author

Cassiano Albuquerque is the founder of @Cassi.ai. He spent twenty years in market research operations before building software for research teams. To talk about questionnaire changes and script testing in your operation, contact him on LinkedIn.

Published by Cassi.ai. Read the full article at https://www.cassiai.com/blog/questionnaire-version-script-mismatch.