Tracking Study Questionnaire Changes: Test the Old Paths

Tracking study questionnaire changes get tested, while the old filters they touch do not. Regression testing runs last wave's paths on the new script first.

The new wave of the brand tracker starts the way it always does. The programmer copies last year's project on the survey platform and applies this year's change list. Two brands join the list and one is retired. A question on how people discover the category gets added. The analyst tests every one of those tracking study questionnaire changes, screen by screen, and they all work. Nobody opens the follow-up block for the retired brand's buyers. Its filter still asks for code 7, and this year code 7 belongs to one of the new brands. The block now opens for the wrong people, and nothing on screen says so.

Key Takeaways

  • A tracker repeats the same questionnaire wave after wave, so any change to the instrument can move the trend line on its own.
  • Crunch.io lists common changes between waves, such as giving an existing code to a new answer option or reversing the coding of a scale.
  • Understanding Society's support team counted 322 cases asked a wave 5 module a second time, and pointed to errors in the data the routing read. The filter itself used a code frame different from the released files.
  • Testing only the change list leaves every old filter that reads a changed code unchecked.
  • Regression testing runs last wave's test scenarios on the new script before fieldwork, and every difference not on the change list goes back to programming.

What a tracker is, and what changes between waves

Norstat describes a tracking study as research repeated over time with the same questionnaire, so that data points can be compared from one round to the next. It also says the quiet part out loud: every change in method or questionnaire may affect the time series.

A tracking study is a survey fielded in repeated rounds with the same core questionnaire, to measure change in brand, customer or market indicators over time.

A wave is one of those rounds. The script for each wave is usually a copy of the previous one, edited.

A filter is a condition in the script that decides who sees a question or a block, such as "ask this block only of people who bought brand 7 last".

Change is inevitable, and good tracker managers admit it. Jeffrey Henning of Researchscape, in Don't Sneeze on a Tracker, lists the usual pressures. Brands get renamed. An answer typed into "Other" becomes common enough to join the list. One client's study kept moving upmarket, so its qualification rose from firms with 100 employees to 250, then 500. His team filtered the historical results to match. He also warns that adding options to a select-all question lowers the share choosing each of the others.

How the new wave's script gets built today

PERSONAL EXPERIENCE In the trackers I have run and audited, the new wave is built from the old one in the same sequence every year.

  1. The research team agrees the change list in the annual review with the client.
  2. The programmer duplicates last wave's project on the survey platform.
  3. The programmer applies the changes: brands in and out, new questions, new answer options, a revised screener.
  4. The analyst tests the changed questions, walking the paths that the change list names.
  5. A soft launch goes out, and the full wave opens.

Step four is where the gap lives. The test is scoped by the change list, so it proves the new parts work. It says nothing about the old parts that read something the change touched. A brand code, a question number, a scale value or a quota variable can change meaning while every screen still displays correctly.

Which changes break the series without anyone noticing

Crunch.io has hosted hundreds of trackers, and its guide How to Conquer Tracking Surveys lists the changes that keep coming back. Among the avoidable ones: an old question relabeled with a retired number, an existing code given to a new answer option, scale questions with reversed coding, and a programmer who decided numbering should start at zero.

A code is the number stored in the data for an answer option, such as 7 for one brand on the list. Filters, quotas and tables all read the code, never the label on screen.

Change in the new waveWhat a test of the change seesWhat breaks somewhere elseWhat a regression test shows
A retired brand's code goes to a new brandThe new brand displays correctlyOld filters and quotas on that code now select the new brand's buyersLast wave's buyer of that brand takes a different path
A scale's coding is reversedThe question looks the sameThe top-box share flips in the tablesThe same answers store different values
Questions are renumberedThe new question worksA filter that reads the old number now reads another questionOld scenarios skip or enter the wrong block
The screener qualification changesThe screener worksThe trend base shifts, as in the Researchscape exampleLast wave's profiles qualify differently
An option joins a select-all listThe option displaysThe share for other options fallsNot a script error, a flag for the trend chart

The changes come from Crunch.io and Researchscape. The pairing with what breaks and what a regression test shows is ours.

A real filter that read a different code frame

Understanding Society, the UK household panel run from the University of Essex, documents a case in its public user support tracker. A researcher asked why some respondents had answered the employment history module in both wave 1 and wave 5.

The support team published the wave 5 filter for that module. It read a variable from the sampling database, and that variable used a different coding frame from the one in the released data files. A value coded 7 in the data was coded 1 in the variable the filter read. They also gave the count. 322 cases had been routed to the questions again, for reasons such as errors in the data the routing was based on, and gaps between the fed-forward data and the final dataset. Checked against the correct filter, 28 respondents who should have received the questions did not.

One filter, two code frames

A case from Understanding Society, the UK household panel run from the University of Essex, in wave 5, employment history module. Two boxes side by side describe the same households. On the left, the released data files store the household sample origin value as code 7. On the right, the sampling database, which the wave 5 filter read, stores the same value as code 1. Below, a bar chart in cases, from the panel support team: 322 cases were asked the module a second time, and 28 respondents should have been asked and were not. The source line says this is Understanding Society user support issue 852, that the reasons given for the repeat cases include errors in the data the routing read and gaps between fed-forward data and the final dataset, and that the 28 were counted against the filter checked with the correct code frame. A closing line says each frame was right on its own, and a check against the wrong one checks nothing. Every number belongs to Understanding Society and none of them is a Cassi.ai measurement.

The filter and the released data were both correct. Anyone checking the routing with the released codes would have checked the wrong thing.

This is a large academic panel with a methodology team and public documentation. The filter was correct in its own frame, and the released data was correct in its own. Anyone checking the routing against the released codes would have checked the wrong thing. The reasons given for the repeat cases point to the data the routing read. They surfaced years later, because a researcher compared two waves.

Fed-forward data is information from an earlier wave that the script loads in advance, so that routing can depend on what the respondent said before. Trackers with a panel sample use it too, and every fed-forward variable is a code that two waves have to agree on.

What it costs, and who answers for it

PERSONAL EXPERIENCE A filter error in a tracker almost never shows up as an error. It shows up as a movement. The follow-up block for one brand's buyers gets a different base, the satisfaction score for that brand moves, and the brand team reads it as the market.

By the time someone notices, usually at tabulation, the wave is closed. The choices are all bad. Re-field the block, footnote the trend, or drop the brand from this year's comparison. Each one costs the thing a tracker is bought for, a series the client can trust. We covered how hard it already is to tell a real movement from noise in our article on brand tracker significance. A script defect adds a third possibility nobody tests for.

Testing by hand makes it worse. An analyst walks the script, changes answers, and walks it again, because every combination of filter and quota opens a different path. On a tracker with a long brand list, the old paths outnumber the new ones many times over. They are exactly the ones a change-list test skips.

The person who answers is the one who signed off the questionnaire for fieldwork, and next to them the tracker lead who presents the trend.

Regression testing: run last wave's paths on the new script

Software teams named this problem decades ago. The ISTQB glossary defines regression testing as change-related testing to detect whether defects have been introduced or uncovered in unchanged areas.

Regression testing, for a tracker, is running the test scenarios from the previous wave on the new wave's script. Then every difference in path or stored value is compared against the change list.

A test scenario is a set of answers that walks one path through the questionnaire, with the path it should take and the values it should store written down in advance.

Crunch.io recommends two of the pieces already: a dry run with dummy respondents before fieldwork, and a side-by-side comparison of the metadata of the two waves. Here is the full sequence we recommend.

  1. Keep last wave's scenario set, each scenario with its expected path and stored values.
  2. Write this wave's change list, and next to each change, the scenarios it is supposed to alter.
  3. Compare the two scripts' metadata: variable names, codes, labels, filters and quota definitions.
  4. Run every old scenario on the new script.
  5. Sort each difference as expected, because the change list explains it, or unexpected.
  6. Send every unexpected difference back to programming before the soft launch.
  7. Run new scenarios for the changes themselves, then fold them into the set for next year.
  8. Record, per question, the waves it can be compared across, and flag new options on the trend charts.

A regression test between two waves

A flow in three steps that runs last wave’s paths on the new script before the soft launch. Step one, on the left, is last wave’s scenario set, each scenario with the path it should take and the values it should store. Step two, on the right, is this wave’s change list, each change with the scenarios it is supposed to alter. Both feed step three in the center: run every old scenario on the new script, then compare paths and stored values with last wave and check each difference against the change list. Two outputs come out of it. On the left, an expected difference, which the change list explains, meaning the new wave is doing its job. On the right, in orange, an unexpected difference that nobody wrote down, which goes back to programming before the soft launch. A closing band says new scenarios for this wave’s changes join the set, so next year’s test starts from there. The figure contains no measured numbers.

The change list is what separates a planned difference from a defect. Without it, every difference looks plausible.

UNIQUE INSIGHT Step five is the method. A difference that the change list explains is the new wave doing its job. A difference nobody wrote down is a defect, however plausible the new path looks. In our opening example, the buyer of the retired brand now enters the new brand's block. Nobody wrote that change, so it goes back to programming.

What a regression test will not catch

UNIQUE INSIGHT A regression test proves the new script does what the old one did, except where you decided otherwise. It cannot prove the respondent reads the question the same way.

Question order is the classic case. In a Pew Research Center poll from November 2008, 81 percent said Republican leaders should work with Obama when a question about Democratic leaders came first. Without it, 66 percent said so. The script was identical in both versions. Only the context changed. Pew's guidance for trends is identical wording and a similar position in the questionnaire.

Sample and mode are the same story. Henning's point that lists are like rivers applies to any panel. Pew also reports that its move from phone polls to an online panel complicated some of its trends, and no script test would have touched that. Those are methodological decisions. They belong in the change list as decisions, so that the trend chart can say what moved and why.

Where Cassi.ai comes in

cassiai.com is a software engineering company specialized in the pains of market research, innovation and insights, founded by Cassiano Albuquerque after twenty years in the field. We understand research first, and we automate research projects with or without language models.

Survey QA is a Cassi.ai product that tests programmed surveys automatically, walking every screen on desktop and mobile and returning a verdict screen by screen, with screenshots.

Survey QA is our product for testing the programmed script before fieldwork, the job most teams still do by hand. It walks the questionnaire's paths on desktop and mobile, the way a respondent would. It returns the verdict screen by screen, with a screenshot of every screen and the rule behind every finding. The list of paths comes from the approved questionnaire, and the analyst still signs the release to field.

For a tracker, last wave's path list becomes the regression set. The paths that proved last year's rules run again on this year's script, next to the paths for the changes. We described the earlier links in the same chain in survey testing before fieldwork and in the soft launch data check. The multi-country version of the same problem is in one questionnaire, nine routes.

FAQ

Why do small questionnaire changes affect a tracking study?

Because a tracker measures change by repeating the same instrument. Norstat notes that every change in method or questionnaire may affect the time series, and code changes can move results while the screens look identical.

What is regression testing for a survey?

Running last wave's test scenarios on the new wave's script and checking every difference in path or stored value against the change list. The idea comes from software, where ISTQB defines it as testing unchanged areas after a change.

Can you reuse an answer code for a new brand in a tracker?

Avoid it. Crunch.io lists giving an existing code to a new option among the most common avoidable changes. Retire the old code and give the new brand a code of its own.

How do you check a copied tracker script before fieldwork?

Compare the metadata of the two scripts, run every old scenario on the new one, and sort each difference as expected or unexpected. Unexpected differences go back to programming before the soft launch.

What should a trend chart show when a question changes?

The waves across which each question can be compared, and a flag for new answer options. Researchscape flags new choices so readers know earlier waves never offered them.

Before the next wave is copied

UNIQUE INSIGHT A copied script carries every rule from last year, including the ones this year's change list quietly broke. Testing what changed checks the new parts. The series depends on the old ones.

If your next wave starts from a copy of the last one, check whether last wave's test scenarios still exist. If they were never written down, this is the wave to start, and we can walk the old paths with you before the soft launch.

About the author

Cassiano Albuquerque is the founder of cassiai.com. He has spent twenty years in market research, was ESOMAR's representative in Brazil and speaks at IIEX, ESOMAR and ABEP. He now builds AI systems for research teams. To talk about regression testing for your tracker, contact him on LinkedIn or see the cassiai.com portfolio.

Published by CassiAI. Read the full article at https://www.cassiai.com/blog/tracker-wave-script-regression.