Crosstab Analysis: Reading the Banner Book Nobody Reads
Crosstab analysis lives in a banner book of hundreds of tables with letters under the figures. How to read them, where they mislead, and what to check first.
Fieldwork closes and the data processing team runs the tables. Every question in the survey is crossed by the same set of groups: gender, age bands, regions, users and non-users of the brand. Each cell shows a column percentage, the base sits at the top, and small letters appear under some of the figures. The result is the banner book, often hundreds of pages long, and it is still the standard deliverable of crosstab analysis in a quantitative study. The client opens page one, scrolls for a while, and sends an email asking what the answer is.
Key Takeaways
- In a crosstab, the letter B under a figure in column A means column A is significantly higher than column B. Sawtooth's documentation does not test any column with fewer than 30 respondents.
- At a 5 percent significance level, the chance of at least one false positive across 20 independent tests is 64 percent, and across 100 tests it is 99 percent.
- A banner of 10 columns compares 45 pairs of columns in every row. On an 8-row table that is 360 tests, and about 18 letters would appear by chance even if no group truly differed.
- Pew Research Center puts the margin of error at about 3 points for 1,067 interviews and 8 points for a subgroup of about 160.
- Base size, the test used, the correction for multiple tests and the size of the gap all have to be read before a letter goes into a deck.
What a crosstab, a banner and a banner book are
A crosstab is a table that shows the answers to one question broken down by the categories of another. Displayr's definition calls it a table that displays the relationship between two or more categorical variables. In commercial research the rows are historically called stubs and the columns banners.
A banner is the fixed set of groups running across the top of every table, such as age bands, region and brand users. The base is the number of respondents behind each column. A banner book is the full set of tables for a study, every question crossed by the same banner, usually delivered as a spreadsheet workbook or a long PDF.
The tables come from a written specification. The earlier article on report automation covers that tabulation plan and why it is where errors are born. This article is about what happens after the tables arrive.
How the banner book is read today
PERSONAL EXPERIENCE In the operations I have worked in, the banner book is read the same way almost everywhere. The researcher prints it or opens it on a second screen. They go page by page with a highlighter, marking every letter. The marked cells are copied into PowerPoint, and those become the findings. When the client asks for a cut that is not in the banner, such as heavy users in the south, it becomes a request to data processing and another export a few days later.
Outside the research team, almost nobody reads the book at all. The marketing director, the brand manager and the sales lead get the deck. The deck shows the letters someone decided were worth showing. The other pages, with their own letters, stay in the attachment.
So the default output answers no business question on its own, and every new question costs another round trip.
How to read the letters in a crosstab
The letters come from a test that compares columns two at a time. Sawtooth's documentation is precise about the convention. When the cell in column A shows the letter B, column A is larger than the lettered column, tested with z-tests for proportions and t-tests for means. Alchemer's help pages call the same procedure the column proportions test, which compares all pairs of columns by row within a banner question.
Statistical significance is the judgment that a difference between two groups is larger than sampling error alone would usually produce, at a chosen confidence level such as 95 percent. It says nothing about whether the difference is large enough to matter.
| What is in the cell | What it tells you | What it does not tell you |
|---|---|---|
| A percentage | The share of that column's base giving that answer | Whether the base is big enough to trust |
| A letter, such as B | This column is significantly higher than column B | How much higher, or whether it matters |
| No letter | No pair passed the test at the chosen level | That the groups are the same |
| A low base flag or no test | The column was too small to compare | Anything about that group, in either direction |
The tools differ in their safeguards. Sawtooth does not test columns with fewer than 30 respondents. Alchemer runs the column proportions test only when the overall chi-square for the table is significant. Q, the tabulation software from Displayr, applies a False Discovery Rate correction by default when it shows column comparisons. A PDF banner book usually carries one line about the test in a footnote, if it carries any.
Where the banner book misleads: small bases and many tests
The first trap is the base. Pew Research Center gives a clean example. A sample of 1,067 interviews has a margin of error of about plus or minus 3 points. A subgroup of about 160 people in that same sample has a margin of about plus or minus 8 points. A banner that cuts region by age by user status reaches bases like that very quickly. A 10-point gap between two such columns can easily be noise.
The second trap is the number of tests. The multiple comparisons problem is the rise in false positives when many tests are read together. MathWorld gives the formula. If each of m independent tests uses significance level α, the probability of at least one false rejection, when nothing is really different, is 1 minus (1 minus α) to the power of m.
UNIQUE INSIGHT Run that formula on an ordinary banner book. At α of 5 percent, one test has a 5 percent chance of a false letter. Twenty tests have 64 percent. A banner of 10 columns compares 45 pairs in every row, which puts the chance of at least one false letter at 90 percent for a single row. An 8-row table runs 360 comparisons, and about 18 letters would appear by chance alone. Real tests in one table are not fully independent, so these numbers are a guide. The direction is not in doubt.
Enough tests and chance writes the letters
A bar chart in percent showing the chance of at least one false significance letter when each test uses a 5 percent level and no group truly differs. One test, 5 percent. Five tests, 22.6 percent. Twenty tests, 64.2 percent. Forty-five tests, which is one row of a crosstab with a 10-column banner compared in all pairs, 90.1 percent. One hundred tests, 99.4 percent. The formula comes from MathWorld, Multiple Comparisons Problem: one minus one minus alpha to the power of m. The values are our arithmetic, assuming independent tests, and tests inside one crosstab are not fully independent, so they are a guide. A closing line says an 8-row table on that banner runs 360 comparisons and about 18 letters would appear by chance. None of the numbers is a Cassi.ai measurement.
The more columns in the banner and rows in the table, the more certain it becomes that some letters mean nothing.
So a banner book with hundreds of pages will always contain letters. The question is which ones survive a second look. The NIST/SEMATECH e-Handbook of Statistical Methods states the principle plainly. Running pairwise comparisons over and over for all possible pairs will not, in general, work, because the overall significance level is no longer the one set for a single pair. Its advice is to choose the comparisons that matter before seeing the results.
Who answers for a letter that was chance
Data processing answers for the table being computed as specified, and it usually is. The client's marketing team answers for the decision taken on the finding. Between them sits the research lead who chose which letters went into the deck and signed the debrief.
That person carries the risk. If a regional campaign is built on a letter that came from a base of 70 and 360 comparisons, the table was technically correct, and the finding was still wrong. Nobody will blame the software. They will ask who said the south was different.
What to check before a letter goes into the deck
The methodology for reading a crosstab fits in a short list. These are the checks we recommend, in order, for every letter someone wants to present.
- Read the base of both columns. We treat a base below 100 as a lead to confirm, not a finding.
- Find out which test produced the letter and at what confidence level.
- Find out whether a correction for multiple comparisons was applied, and which one.
- Look at the size of the gap in points, next to the margin of error for those bases.
- Check whether neighbouring rows and related questions point the same way.
- Look for the same pattern in another wave or another source before anyone acts on it.
- Write down which letters went into the deck and why, so the next wave can test them again.
Five checks before a letter goes into the deck
A decision flow in five checks, each with an exit to the right. Check one, the base of both columns: read the respondents behind each figure. If the base is small, treat the letter as a lead to confirm in a larger sample. Check two, test and level: know which test produced the letter and at what confidence. If nobody knows the test, ask data processing before quoting it. Check three, correction for many tests: check whether one was applied and which. If there is no correction on a wide banner, rerun with one or discount isolated letters. Check four, size of the gap: compare the points with the margin of error for those bases. If the gap is close to the margin, report it as uncertain, not as a difference. Check five, neighbours and another wave: related rows and another source point the same way. If the letter stands alone in one row, hold it until the next wave tests it again. A closing band says that when all five pass, the letter becomes a finding and the deck records which table it came from. The figure contains no measured numbers.
The exits are agreed before the debrief. A letter that fails a check is not thrown away, it waits for a larger base or the next wave.
Step 5 does more work than it seems. A real difference in satisfaction for one group usually shows up again in recommendation and repurchase. A lone letter in one row, with nothing around it, is the most likely chance result in the book.
What changes when the cross and the test live in a dashboard
The banner book is a fixed export. A filterable dashboard can keep the crosstab and the test in one place, recomputed for the cut the reader actually wants.
| Question | Banner book | Filterable dashboard with significance |
|---|---|---|
| A new cut, for example heavy users in one region | A request to data processing and a new export | A filter, applied by the reader |
| The base behind each figure | Printed at the top of the column, easy to skip | Shown next to the figure, updated with every filter |
| The test and its level | A footnote, if anywhere | Next to the figure, with the p-value |
| Correction for many tests | Depends on the tab software and its settings | Has to be designed in and stated on screen |
| Who can read it | People trained to read letters | Anyone who can read a base and a marker |
The third row of that table is not automatic. A dashboard that lets anyone filter freely also lets anyone run hundreds of tests by clicking. The correction and the minimum base have to be rules the dashboard enforces, not choices left to each reader. The earlier article on the dashboard built from the decision covers which questions the screen should answer. This one adds the statistical floor under it.
Where Cassi.ai comes in
@Cassi.ai is a software engineering company specialized in the pains of market research, innovation and insights. It was founded by Cassiano Albuquerque after twenty years in market research. We automate research projects with or without language models.
Dash Lab is a Cassi.ai product that builds made-to-measure research dashboards, with matrices, segmentation, driver analysis, significance testing and tracking, and the base next to every number.
Dash Lab starts from the pain clients describe to us: the default panel that comes with the collection platform opens, has charts, and does not answer the business question. Dash Lab turns the study into an interactive, custom statistical application. Every cut is a filter, the base sits next to every figure, and the significance test appears beside the number it qualifies, with the p-value on screen.
The analyst still decides what the finding is. The difference is that a reader who wants another cut gets it in a click, with the base and the test attached, instead of waiting for the next export. The broader case for engineering over generic chat tools is in why a general chatbot is not a research platform. More in the cassiai.com portfolio.
FAQ
What is crosstab analysis in market research?
Reading the answers to each question broken down by groups of respondents, such as age, region or brand use, to see where groups differ. Displayr defines the crosstab as a table of the relationship between categorical variables.
What do the letters in a crosstab mean?
In the common convention, the letter B in a cell of column A means column A is significantly higher than column B. Sawtooth's documentation describes it this way.
What base size is too small for significance testing in a crosstab?
There is no single rule. Sawtooth does not test columns under 30, and Pew shows a subgroup of about 160 carrying a margin of about 8 points.
Why do banner books show differences that are not real?
Because they run hundreds of tests at once. At 5 percent per test, the chance of at least one false positive passes 60 percent at 20 tests.
How do you correct for multiple comparisons in a crosstab?
Use the correction your tabulation software offers, such as False Discovery Rate in Q or Holm in Sawtooth, and state on the table which one was applied.
Before the next banner book goes out
UNIQUE INSIGHT A banner book with every cut is the raw material for an analysis, and at hundreds of tests it is guaranteed to contain letters that mean nothing. The answer the client asked for is the short list of differences that survive the base, the correction and a second source.
If your last debrief quoted a letter from a column of under 100 respondents, find the table and check what the base was. That check takes five minutes, and it is cheaper than a campaign built on chance.
About the author
Cassiano Albuquerque is the founder of @Cassi.ai. He spent twenty years in market research operations before building software for research teams. To talk about how your tables and dashboards are read, contact him on LinkedIn.
Published by Cassi.ai. Read the full article at https://www.cassiai.com/blog/crosstab-banner-tables.