Survey Testing Before Fieldwork: Coverage Over One Click
Survey testing before fieldwork is usually one person clicking through once. What path coverage looks like instead, and what the untested path really costs.
The questionnaire comes back from programming on a Thursday. Field opens Monday. Between those dates sits a task that appears on nobody's timeline as more than a line item: someone has to check that the thing works. In most research operations that check is one person, one browser tab, one set of answers, and a message saying it looks fine.
Key Takeaways
- Survey testing before fieldwork is usually a single manual walk-through by one person answering one way.
- A questionnaire with filters, quotas and branching has dozens of distinct answer paths through it, and one walk-through touches one of them.
- The paths nobody walked become bad data that surfaces during analysis, after fieldwork has already been paid for.
- Systematic coverage runs each attempt in an isolated browser session and classifies its outcome instead of returning a single pass or fail verdict.
- When the questionnaire refuses to advance, the correct behavior is to stop and record the blocker with evidence, because forcing the step forward destroys the proof that a defect exists.
What testing a programmed questionnaire involves
A modern quantitative questionnaire is a piece of software. Conditional display, answer piping, validation rules, randomization, redirect URLs to the panel provider, termination points. The research team wrote it as a document, a survey programmer turned it into a script on Qualtrics, Decipher, Forsta or whichever platform the study runs on, and what has to be verified is that the script behaves the way the document said it would.
Two terms carry most of the weight here, and both get used loosely in briefing meetings.
A path is the specific sequence of screens a single respondent passes through from the first question to the end of the questionnaire, determined by the answers that respondent gives. Each filter, each branch and each quota rule sends different answers down a different sequence. Two people opening the same link can see two entirely different questionnaires.
A screener is the block of qualifying questions at the front of a questionnaire that decides whether a respondent belongs in the study. It is where most of the branching lives, which makes it where most of the defects live.
Standard field guidance treats this as scheduled work. J-PAL recommends planning one to three days of testing depending on questionnaire complexity once the survey is programmed, and notes that bugs in a freshly programmed survey are normal rather than exceptional.
How it is done today
The programmer sends a test link. A researcher opens it, answers as a plausible respondent, reaches the end screen, and replies that it works. Sometimes a second person repeats the exercise with a different age or category answer. If the study matters, someone prints the questionnaire and reads it alongside the screen, checking wording rather than logic.
Almost nobody enumerates the paths first. There is no list of which answer combinations produce which sequences, so there is no way to say which of them were tried. Testing guidance is explicit that the goal is to walk through all branches and logic paths, and that is the part a manual pass cannot deliver on a screener with eight qualifying questions.
Teams still do it this way for an unglamorous reason. Walking a questionnaire by hand takes eight to fifteen minutes per attempt. Forty attempts is a full day of a senior person's time, on a task that produces nothing visible when it goes well. It is the same constraint that turns eleven concepts into three in a meeting, covered in [synthetic respondents in concept testing](/blog/synthetic-respondents-concept-testing).
The path nobody walked
UNIQUE INSIGHT The defect a manual test misses is almost never in the main path. It is in the combination. The respondent who picks the third category and then answers "none of the above" on the next screen lands in a branch that was written but never exercised, and the questionnaire routes them past a block they should have seen.
The delay is what makes it expensive. A routing defect throws no error. The respondent finishes, gets paid, and the record enters the dataset looking complete. The problem appears weeks later, when the analyst opens the base sizes and finds a subgroup with 40 respondents where the plan said 300.
By then fieldwork is closed and the panel has been invoiced. Re-field the affected cells and absorb the cost, or report on a base you know is compromised. The insight lead answers for it in front of the client either way.
| Failure type | What the respondent experiences | When it surfaces today | Cost of finding it late |
|---|---|---|---|
| Routing defect | Finishes normally, sees the wrong blocks | During analysis, as odd base sizes | Re-field or a compromised base |
| Quota or termination defect | Terminated when they should qualify | During field, as a low incidence rate | Overpaid sample and a delayed close |
| Technical blocker | Cannot advance past a screen | During field, as complaints and drop-off | Lost respondents and a paused study |
PERSONAL EXPERIENCE Grouped from the run classifications we use in survey QA work at Cassi.ai. The routing defect is the one that survives fieldwork, which is why it is the expensive one.
Coverage instead of one click
Path coverage is the proportion of distinct answer sequences through a questionnaire that have been executed and verified, as opposed to the number of test attempts recorded. Twenty attempts designed against different filters and quota cells produce more coverage than a hundred attempts that all answer the same way.
PERSONAL EXPERIENCE The approach we build runs each attempt in an isolated browser session with its own cookies and answer profile, so attempts cannot contaminate each other. A screener that a person tests once by hand typically yields dozens of paths worth executing. Each run is classified by outcome, completed, screened out, quality termination, technical blocker or error, instead of collapsing into one pass or fail verdict for the questionnaire as a whole.
That classification is what makes the output usable. A questionnaire returning 60 completions and 40 screenouts is not broken. A questionnaire returning 40 screenouts where every profile was built to qualify is broken in a locatable way, and the run record says which screen sent them out. This is engineering work with a defined contract rather than a prompt, a distinction we take up in [why a general chatbot is not a research platform](/blog/why-chatgpt-is-not-a-research-platform).
Exploratory mode and strict QA mode
| Mode | Intent | Behavior on error | Recommended use |
|---|---|---|---|
| Exploratory | Try paths and record what the questionnaire does | Records the behavior and continues to the next attempt | Discovery and mapping, first pass on a new script |
| Strict QA | Verify the script against the specified logic | Fails early and stops the run at the first error | Regression testing and sign-off on a finished version |
| Speed profile | Run in parallel with minimum delays between answers | Records timing-dependent failures | Finding validations and traps that break under fast entry |
PERSONAL EXPERIENCE Mode definitions as implemented in our survey QA work. Exploratory runs first, on the version that came back from programming. Strict QA runs on the version about to be signed off, and again after any change to the script.
The speed profile deserves a note. Running fast is normally treated as respondent misbehavior. As a test it exposes validation rules that depend on timing, attention traps calibrated for a reading speed nobody has, and screens that break when the next answer arrives before the page finishes rendering.
The rule about never forcing the questionnaire forward
UNIQUE INSIGHT When an automated run reaches a screen it cannot pass, the tempting behavior is to try harder. Click elsewhere, fill the field differently, submit again, finish the run. That destroys the finding. A run that forced its way through a broken validation reports a completion, and the defect stays in the script until a real respondent hits it in field.
The correct behavior is to stop. The run is recorded as a technical blocker with the screenshot at the moment of failure, the page state, the console log and a probable cause: failed validation, unresponsive control, timeout, script error. A bug report that says only "it did not advance" starts a round of messages. One with the screen, the log and the last successful step starts a fix.
How a coverage run is set up
PERSONAL EXPERIENCE The sequence below is the one we run before a questionnaire goes to field.
- Get written authorization for the test link and confirm it points at a staging version, never a live study already collecting data.
- Enumerate the paths from the questionnaire document: every filter, every branch, every quota cell, every termination point.
- Build answer profiles as explicit hypotheses, one per path worth exercising, including profiles designed to terminate and profiles designed to sit at a quota boundary.
- Run the profiles in isolated sessions, in exploratory mode, capturing a screenshot and page state at each step.
- Classify every run by outcome and separate legitimate screenouts from routing defects.
- Hand the blockers to the survey programmer with evidence attached, then re-run in strict QA mode against the corrected version.
Step two is the one teams skip, and skipping it makes the rest impossible to measure. You cannot report coverage against a set of paths nobody wrote down.
FAQ
How long should survey testing before fieldwork take?
J-PAL field guidance suggests one to three days of testing after the questionnaire is programmed, scaled to complexity. That budget assumes manual testing. Automating it raises the number of paths covered inside the same window rather than shortening the window.
What is a path in a questionnaire?
A path is the specific sequence of screens one respondent passes through, determined by the answers they give as filters and branching send them in different directions. A screener with eight qualifying questions has many of them, and a single manual walk-through covers exactly one.
Does automated testing replace human review of the questionnaire?
No. Automation covers logic, routing, quotas and technical failure. Wording, scale balance, cultural fit and whether the questionnaire measures the construct stay with the researcher. UNIQUE INSIGHT Machines verify behavior, humans verify meaning. The same split applies downstream, in [how the evidence chain survives automated qualitative coding](/blog/qualitative-coding-evidence-chain).
Can this run against a live survey already in field?
It should not. Test runs against a live link generate real records, consume quota and distort the incidence rate. PERSONAL EXPERIENCE We require written authorization and a staging link before any run, which is why that is step one of our own process.
Where this leaves the pre-field check
A questionnaire that has been walked once has been verified for one respondent. Everyone else is walking a path nobody tried, and the people who hit the defect will never report it, because from where they sit nothing looks wrong. The pre-field check deserves to be measured in paths covered, not in whether someone reached the end screen.
Cassi.ai is a software engineering company specialized in the pains of market research, innovation and insights. We are research people who build software, with twenty years of fieldwork behind the methodology, and we automate research operations for enterprise teams and agencies. More on that at Cassi.ai.
If your last three studies each got one manual walk-through before field, it is worth counting how many distinct paths those questionnaires actually had. If you want to see coverage run against a screener of your own, we will set it up on a staging link and show you the classification of every run.
Published by Cassi.ai. Read the full article at https://www.cassiai.com/blog/survey-testing-before-fieldwork.