Gamification in Market Research: Same Table, New Frame

Gamification in market research changes how much respondents give and what they say. What the experiments found, and how to keep each wave comparable.

A researcher writes the questionnaire in a document: Q1 to Q45, scales from 1 to 10, a few open questions at the end. The programmer puts it into the collection tool, and every question comes out as a radio button, a grid or a drop-down. The respondent opens it on a phone and recognizes the format at once. It looks like a school form, and it gets the effort people give to school forms. Gamification in market research was the industry's attempt, more than a decade ago, to change that format. The experiments worked on effort. They also changed the answers, and that second result is the one a research team has to manage.

Key Takeaways

  • In Engage Research and GMI experiments with more than 5,000 participants, asking people to describe themselves in exactly seven words raised the share who answered from 85 to 98 percent.
  • The same program found a two-minute limit raised a list of favorite foods from an average of 6 items to 35.
  • In a systematic review by Keusch and Zhang, most studies reported more fun, interest and satisfaction, while the effect on completion, item omission and satisficing was unclear.
  • The frame changes the data, so it is chosen per study, fixed across the waves of a tracker, and tested on a split sample before any switch.

What gamification in market research means

The term covers a wide range, from rewording one question to building the whole survey as a game.

Gamification is the use of game elements, such as a story, a mission, a rule, a time limit or points, in an activity that is not a game. In a survey, it changes how a question is framed or how the answer is given. The question itself stays the same.

A survey frame is everything the respondent sees around the question: the wording, the scene, the input on screen and the sense of progress.

Florian Keusch and Chan Zhang reviewed the published and unpublished evidence in a systematic review of gamified surveys. Their list of game elements includes challenges, story or narrative, rewards, goals and badges. At one end sits rewording a question so it sounds like a game. At the other end sits a survey in which an avatar adventures through a fantasy land while the respondent answers.

How the standard questionnaire is built today

PERSONAL EXPERIENCE Across twenty years of market research operations, the build has barely changed. The questionnaire is written in Word and numbered question by question. It goes to the programmer, who picks a template in the collection tool. A brand diary, a satisfaction study and an employee diagnosis come out with the same screens: a question at the top, radio buttons, a grid, a progress bar. Nobody decides that the format should look like that. It is the default, and the default wins.

Research Live described the same picture in 2011. In An extra life for online surveys, radio buttons, grids and drop-down menus were still the rule. The interactive future the industry had been promised had not arrived. Jon Puleston, then at GMI, located the problem in the language itself: asking how much you disagree with a statement on a scale of one to ten is a cliché that turns people off.

The cost shows up in the data. Satisficing is the respondent's habit of giving an answer that is good enough to move on, rather than the answer they would give if they thought about it. It looks like the shortest open end, the middle of the scale and the first option that fits. A research-in-progress paper by Schacht, Keusch, Bergmann and Morana links dull web surveys to satisficing and to break-off. Its examples are speeding, which is rushing through the screens, and straightlining, which is marking the same column on every row of a grid. The team that delivers the dataset answers for it, and the client pays for the fieldwork that produced it.

What the game experiments found

The best known industry evidence is a year-long program by Engage Research and GMI. In More than a game, Deborah Sleep of Engage reports more than thirty experiments with over 5,000 participants, run for six clients including Sony Music, Heinz and Kimberly-Clark. The paper behind it, The Game Experiments, won the best methodological paper award at the 2011 ESOMAR Congress. The research method was simple: keep the question, change the frame, and compare.

Question as usually askedReframed with a game elementWhat changed
Describe yourselfDescribe yourself in exactly seven wordsDescriptors rose from 2.4 to 4.5 on average, and answering rose from 85 to 98 percent
Rate how much you like each music actYou own a radio station: build its playlistActs evaluated rose from 83 to 148 on average
List your favorite foodsYou have two minutes to list themItems listed rose from 6 to 35 on average
Which emotions do people associate with this?One point for every answer you guess rightTime per answer rose from 8 to 12 seconds, and enjoyment from 50 to 90 percent

Every figure in the table comes from Deborah Sleep's report of the Engage Research and GMI experiments. Research Live's earlier feature quotes the playlist result as 82 against 146, a small gap between two tellings of the same test.

Same question, new frame

Four panels from one program of survey experiments, each with its own unit and scale, comparing a question as usually asked with the same question reframed with a game element. Panel one, describe yourself, in percent of respondents who answered: 85 percent as usually asked, 98 percent when asked to describe themselves in exactly seven words. Panel two, rating music acts, in acts evaluated on average: 83 as usually asked, 148 when respondents were asked to build the playlist of their own radio station. Panel three, listing favorite foods, in items on average: 6 as usually asked, 35 with a two-minute time limit. Panel four, a question about emotions, in percent who said they enjoyed it: 50 percent as usually asked, 90 percent when respondents won a point for every answer they guessed right. Source: Engage Research and GMI, The Game Experiments, ESOMAR Congress 2011, as reported by Deborah Sleep in Research Live, More than a game; more than thirty experiments with over 5,000 participants for six clients. A closing line says effort rose in every test, the answers changed with it, and that part is the team's to manage. Every number belongs to the Engage Research and GMI program and none of them is a Cassi.ai measurement.

Each reframed question drew more from respondents. None of these tests says whether the extra answers were closer to the truth.

The effort result is clear. The data result is harder. Puleston said it plainly to Research Live: yes, the data is different. He read the difference as a sign of better data, because a more engaged mind gives more thoughtful feedback. That may be true. It is also an argument, and a team cannot assume it for its own study.

Why the frame changes the answers too

The peer reviewed record is thinner than the trade press. Keusch and Zhang note that only a few experiments on gamified surveys had appeared in peer reviewed journals. Most studies they found reported a positive effect on fun, interest and satisfaction. The effect on behavior was more unclear: completion, break-offs, item omission, satisficing and the answers themselves.

A tracker is a study repeated in waves with the same questions, so the client can read change over time.

That gap matters most in a tracker. If wave three moves to a new frame and the brand score rises, the team cannot say whether the brand moved or the frame did. The same Research Live feature records two other risks. Gamifying a few questions among many plain ones only highlights how boring the rest are. And game mechanics that are too engaging can pull attention away from the survey goals.

A chapter by Kartik Pashupati and Pushkala Raman, Gamification in Market Research, frames the question the same way. It reviews empirical studies on data quality and engagement, then compares a conventional, text-heavy survey with a gamified version of the same survey.

Choose the frame per study and keep the table

UNIQUE INSIGHT The way out is to separate two things that the standard tool welds together: the frame the respondent sees and the table the client receives.

The results table is the set of variables, codes and bases a study delivers. It holds the same columns whatever screen the respondent saw.

A split-sample test fields the old frame and the new one at the same time, on two random halves of the sample. The gap between the halves measures how much the frame moves the answers.

  1. Write the questionnaire first, with every question, code and filter, as if no frame existed.
  2. Fix the results table from that questionnaire before any design work starts.
  3. Choose the frame for the study and the respondent: a mission map for a diary, a scene for a packaging test, a board for a diagnosis.
  4. Map every screen of the frame back to a question and a code in the table.
  5. Keep game points tied to progress and never to the content of an answer.
  6. Hold the frame fixed across every wave of a tracker.
  7. Before switching a running tracker to a new frame, run a split-sample test and report the gap with the results.

Three frames, one table

A diagram with three frames at the top, each designed for a different study: a mission map for a brand diary with levels and badges, a scene showing a store aisle for a packaging test, and a board of pinned notes for a diagnosis by area. Arrows from all three lead down to a single band that says every screen maps back to a question and a code, so there is one results table for every frame. Below it, three rules keep the waves comparable: one, choose the frame per study, set at design and written down with the method; two, hold it across waves, so a tracker keeps one frame from wave to wave; three, run a split sample first, fielding both frames before a switch and reporting the gap. The figure contains no measured numbers.

The respondent sees a frame built for the study. The client receives the same table either way.

Step five carries the most risk. A point for a correct guess works in a quiz about emotions, as the experiments showed. In a satisfaction question it would reward a particular answer, and the score would be measuring the game.

Where Cassi.ai comes in

@Cassi.ai is a software engineering company specialized in the pains of market research, innovation and insights. It was founded by Cassiano Albuquerque after twenty years in market research. We automate research projects with or without language models.

Survey Studio is a Cassi.ai product that runs immersive research, giving every study its own format, such as gamified missions or a narrative in scenes, on one engine with a live fieldwork cockpit.

Survey Studio is the Cassi.ai product for long online questionnaires. The Cassi.ai team designs a format for each study: missions with levels and badges for a brand diary, a store aisle with hotspots for a packaging test, a board of pinned notes for a diagnosis by area. Underneath runs one engine with five question types and conditional logic checked before fieldwork. All answers land in the same data structure, and the results board is the same for every format. The missions format follows step five: its points celebrate progress through the blocks, and answers are never scored.

The questionnaire gets tested the same way as any other, as in survey testing before fieldwork. The participant-side argument is the one we made about keeping research communities where people already are. Respondents give more when the study meets them in a form they want to use. On the client side, the board is built from the decision the study serves, as in a market research dashboard built from the decision.

FAQ

What is gamification in market research?

The use of game elements, such as a story, a mission, a rule, a time limit or points, to frame survey questions or the way respondents answer them. The question stays the same and the frame changes.

Does gamification improve survey data quality?

Most studies in Keusch and Zhang's review reported more fun, interest and satisfaction. Its effect on completion, item omission and satisficing was unclear. It changes the answers, so quality has to be checked study by study.

Do gamified surveys get longer answers?

In the Engage Research and GMI experiments, yes. A two-minute limit raised a list of favorite foods from 6 items to 35 on average.

Can a tracker switch to a gamified format?

Only with a split-sample test first, so the team can separate a change in the market from a change caused by the frame.

Should points be tied to answers?

Not in an attitude or satisfaction question. Points that reward an answer push respondents toward it. Tie them to progress through the questionnaire.

The format is a research decision

UNIQUE INSIGHT A questionnaire that looks like homework gets homework effort. A frame built for the study gets more effort and different answers. Both outcomes belong to the method, so the frame gets chosen, written down and held steady like any other part of the design.

If your next long study is still going out as the default grid template, ask who chose that format. If the answer is the collection tool, the frame is the first design decision the study has not made.

About the author

Cassiano Albuquerque is the founder of @Cassi.ai. He spent twenty years in market research operations before building software for research teams. To talk about the format of your next long study, contact him on LinkedIn.

Published by Cassi.ai. Read the full article at https://www.cassiai.com/blog/gamified-survey-design.