Survey Soft Launch: What to Check in the First Data File
A survey soft launch only works if someone opens the first data file. Five checks for the first completes, who decides to open field, and the stop rule.
The questionnaire passed testing on Thursday, and on Monday the panel invited the first slice of the sample. By Tuesday morning the first completes are sitting in the platform. That is the whole point of a survey soft launch: a small group of real respondents goes through the live script so that somebody can open their data before everyone else is invited. In a lot of research operations, nobody opens it. The counter goes up, nobody complains, and the full field opens on the strength of a file that was never exported.
Key Takeaways
- A survey soft launch releases a live questionnaire to a small slice of the sample, usually 5 to 10 percent in published guidance, before the full field opens.
- In a SurveyEngine worked example, overall completion in a 40-respondent soft launch was an acceptable 71 percent, while mobile completion was 34 percent against an expected 55.
- Five checks on the first data file catch what a script test cannot: base per filtered block, quota fill, interview length, open ends and breakoff by device.
- The person who signs off on the questionnaire for fieldwork answers for what escapes, so the stop rule is written before the first invitation.
What a survey soft launch is supposed to catch
A survey soft launch is the release of a live questionnaire to a small slice of the target sample, with the rest held back until someone has checked what came in. Full launch is the moment the remaining sample is invited or the quota caps are lifted.
Medallia's documentation says plainly that a soft launch does not replace thorough testing of the survey. It calls it an additional precaution in case testing missed something, such as a country missing from the answer list. The suggested rhythm is 5 percent of the target group on day one, 20 percent on day two and the rest on day three, with at least a day between rounds.
The script test, covered in survey testing before fieldwork, checks the script against the questionnaire document with answers typed by the team. The soft launch checks it against real people on their own phones. Veridata Insights draws the line: unless someone reviews the incoming data, checks the routing and reads the open text, a soft launch is only delivery.
How the soft launch runs today
PERSONAL EXPERIENCE In the operations I have worked alongside, the soft launch is almost always in the timeline. What varies is whether anyone reads it. The survey programmer sets a cap on completes. The panel sends the first invitations. The project manager watches the completes counter. The analyst who will eventually open the data is still closing the previous study. After a day or two without a complaint, someone lifts the cap, and the first time anyone opens the file is when tabulation starts.
Nobody complained, so it works. But the defects a soft launch exists for never complain. A routing error sends a respondent past a block and lets them finish normally. A quota built on the wrong variable fills quietly. A screen that breaks on one phone loses those respondents without an error message reaching anyone.
What the first data file shows that the script test cannot
The clearest public example is in SurveyEngine's guide to soft launches and fieldwork monitoring, published in July 2026. A study of 400 commuters ran a soft launch capped at 40 respondents, or 10 percent. Overall completion came back at 71 percent, judged acceptable. The defect was in the device split: completion on mobile was 34 percent against an expected 55 percent. A choice table on iOS ran past the edge of the screen. It was fixed, a second soft launch of 20 respondents confirmed the fix, and the full study opened.
The overall rate passed. The device split did not.
Two panels from one published soft launch, each with its own unit. Panel one, in respondents: the study targeted 400 commuters, the first soft launch was capped at 40 respondents, which is 10 percent, and a second soft launch of 20 respondents confirmed the fix. Panel two, in completion rate: across all devices the first soft launch completed at 71 percent, which the team judged acceptable; on mobile the expected completion was 55 percent and the observed completion was 34 percent. The cause was a choice table on iOS that ran past the edge of the screen. Source: SurveyEngine, Soft launch and fieldwork monitoring for transport SP surveys, July 2026, a worked example from a mode choice study. A closing line says the headline number looked healthy and only someone who split the file by device found the defect. Every number belongs to the SurveyEngine example and none of them is a Cassi.ai measurement.
An acceptable overall completion rate hid a mobile problem. The file had to be opened and split by device before anyone could see it.
The defect appeared only with real respondents on their own devices, a condition the earlier testing had not reproduced. And the headline metric looked healthy, so somebody had to open the file and split it by device to see anything wrong. American Directions Research Group makes the same bet. It keeps watching early completes even after four layers of QA, to catch problems within the first few percent.
Five checks to run on the first data file
UNIQUE INSIGHT The checks below fit on one page. Each one compares the soft launch file with something written down before the field opened, which is what turns reading a file into checking it.
A routing map is the list of every filter in the questionnaire, with the condition that sends a respondent into or past each block.
A quota cell is one combination of characteristics the study has to fill, such as women aged 25 to 34 in one region, with a target number of completes.
Breakoff is the point where a respondent who started the questionnaire stops answering and never reaches the end.
A speeder is a respondent whose interview is so far below the median length that they could not have read the questions.
| Check | What to compare | What a defect looks like | Usual cause |
|---|---|---|---|
| Base per filtered block | Respondents who saw each block, against the routing map | A block seen by nobody, or by everybody | A filter condition inverted or keyed to the wrong variable |
| Quota fill per cell | Completes per cell, against the targets | One cell racing ahead while another sits at zero | Quota defined on the wrong question, or a targeting error in the sample |
| Interview length | The median and both tails | A cluster far below the median, or one module taking most of the time | Speeders, a skipped section, or a block that runs too long |
| Open ends | Every verbatim in the slice, read in full | Blanks, gibberish, answers to a different question | Wrong text piped into the question, or a weak quality screen |
| Breakoff point | The screen where people stop, split by device | A spike on one screen or one device | A display problem, or a validation that rejects a valid answer |
The metrics come from Veridata Insights and SurveyEngine. The pairing of each check with its usual cause is ours. The open ends get read again at coding, as in the evidence chain behind qualitative coding.
Some methodology reports write the habit down. For Chapman University's 2025 Survey of American Fears, SSRS invited a limited number of panelists on March 24 and the full sample on March 25, after checking the soft launch data for questionnaire content and skip patterns. Its data processing ran a program built from the questionnaire's skip patterns to confirm every question had the appropriate number of cases. That is the first row of the table, automated. Development research calls the same habit high frequency checks.
Who decides to open the full field
The programmer owns the script. The panel owns the sample. The project manager owns the calendar. Opening the full field is a research decision, and it belongs to the person who signs off on the questionnaire for fieldwork, because that person answers for whatever escapes.
PERSONAL EXPERIENCE I have seen a field saved by a check made before it opened. The questionnaire had errors, they were caught before it went out, it went back to programming, and the team had to reopen the conversation with the client. Found during analysis, the same errors would have cost the fieldwork itself.
When the soft launch file is never opened, the sign-off covers a check nobody ran. The calendar made the decision.
If the questionnaire changes after the soft launch, the first slice answered a different instrument. Veridata's guidance is to re-administer the revised items or exclude those answers from analyses of the edited questions. That call also belongs to the person who signed, and it is easier on day two than in the week the tables are due.
Write the stop rule before the field opens
A stop rule is a condition agreed before fieldwork that says which results in the soft launch file pause the field, who reads the file, and who can open the full sample. Veridata tells teams to set decision thresholds ahead of time, so nobody negotiates with a deadline. This is the sequence we recommend.
- Freeze the routing map from the script test, with the share of respondents expected behind each filter.
- Cap the soft launch per quota cell. SurveyEngine sets those caps at 5 to 10 percent of each cell's target.
- Name the person who exports and reads the file, and the hour it happens, before the first invitation goes out.
- Run the five checks against thresholds written in the same document.
- Read every open end in the slice.
- Decide: pause and fix, revise and test again, or open the full field.
- Record the decision, who made it, and which cases answered a version of the questionnaire that later changed.
A stop rule for the first data file
A decision flow in five checks, written before the first invitation, each with an exit to the right that pauses the field. Check one, base per filtered block: who saw each block matches the routing map. If a block was seen by nobody or by everybody, pause and send the filter back to programming. Check two, quota fill per cell: each cell fills at the expected pace. If a cell sits at zero or races ahead, pause and check the quota variable and the targeting. Check three, interview length: the median and both tails sit inside the written range. If there are speeders or one module runs far too long, revise and test again before opening. Check four, open ends: every verbatim in the slice has been read. If there are blanks, gibberish or answers to the wrong question, pause and check the piped text and the quality screen. Check five, breakoff by screen and device: no screen and no device shows a spike. If one does, pause, fix the display or validation, then run a second slice. A closing band says that when all five pass, whoever signed off on the questionnaire for fieldwork opens the full sample and records the decision. The figure contains no measured numbers.
Every exit is decided before field. The only judgment left on the day is whether the file matches what was written down.
Step three matters most: a file with a named reader and a set hour gets opened.
Where Cassi.ai comes in
@Cassi.ai is a software engineering company specialized in the pains of market research, innovation and insights, founded by Cassiano Albuquerque after twenty years in market research. We automate research projects with or without language models.
Survey QA is a Cassi.ai product that tests programmed surveys automatically, walking every screen on desktop and mobile and returning a verdict screen by screen, with screenshots.
Survey QA is our product for the step before this one: testing the programmed script before fieldwork, which most teams still do by hand, filter by filter, at the cost of days of a senior analyst.
The soft launch check is the next link in the same chain, and it needs no language model. Every check in the table counts and compares against something written in advance, so it can run the moment the slice closes, with the open ends sent to the person who has to read them. That is engineering with a defined contract, the difference we drew in why a general chatbot is not a research platform. More in the cassiai.com portfolio.
FAQ
What is a soft launch in survey research?
A release of the live questionnaire to a small slice of the sample, so someone checks the incoming data before the full field opens. Medallia's documentation treats it as a precaution that does not replace testing.
How big should a survey soft launch be?
Veridata Insights and SurveyEngine both point to 5 to 10 percent of the target sample.
What should you check in soft launch data?
The five checks above, against thresholds written before fieldwork. In one SurveyEngine example, the device split was the only place the defect showed.
Can soft launch responses be kept in the final dataset?
Yes, if the questionnaire did not change. If it did, Veridata Insights recommends re-administering the revised items or excluding those responses from analyses of the edited questions.
Who decides whether to open the full field?
The person who signs off on the questionnaire for fieldwork, following a stop rule written before the first invitation.
Before the next soft launch
UNIQUE INSIGHT A soft launch that nobody reads is a full launch with a delay. The respondents in that first slice cannot be invited twice, and their answers only protect the study when somebody opens the file.
If your last few studies had a soft launch in the timeline, check who exported the first file and when. If the honest answer is nobody until tabulation, the stop rule is the shortest document you will write for the next one.
About the author
Cassiano Albuquerque is the founder of @Cassi.ai. He spent twenty years in market research operations before building software for research teams. To talk about soft launch checks for your studies, contact him on LinkedIn.
Published by Cassi.ai. Read the full article at https://www.cassiai.com/blog/survey-soft-launch-data-check.