Statistical Significance Versus Meaningful Significance

Statistical significance – It’s the tool that identifies which differences between consumer segments are meaningful to the research, and which differences are due to random variation. One issue I’ve noticed in my experience is sometimes statistically significant differences are called out in reports simply because they’re significant, but not relevant to the story.

How to understand statistical significance

I find the easiest way to explain statistical significance is to think in terms of margin of error.

If you’ve taken a stats course, you probably remember that the margin of error in a percentage is determined by the size of the sample. More respondents in your sample will increase the confidence that your results represent the true population. The percentage is less likely to fluctuate due to random variation.

That’s what it means when you see something like “n=x respondents were surveyed… yielding a margin of error of +/- y%” at the beginning of a research report.

It’s there to indicate that the true, top-line results are within y percentage points of what you’re seeing with whatever confidence level you select, be it 90%, 95%, etc.

When comparing two independent proportions, they’re only going to be statistically significant if their margins of error don’t intersect. Otherwise, you can’t tell if they’re truly different from each other or if their differences are just random.

The larger the sample size of those two proportions, the more their margins of error shrink. By increasing the likelihood, their differences are statistically significant.

Statistical significance applied to market research

I often work on experiential marketing research projects that last for months. We accumulate sample sizes so large that margins of error decrease to a couple of percentage points for most sub-groups.

When that happens, just about everything is statistically significant. That is when it’s important to comb through the results to identify the differences that are most meaningful to the research.

Yes, it’s great that a significantly greater percentage of unicycle riders (55%) prefer wearing top hats than jugglers do (51%), but that’s a four percentage point difference. Even though it’s statistically significant, it’s not as noteworthy and actionable as a 20 percentage point difference.

Key findings in market research occur when there’s not only statistical significance, but meaningful significance.

I guess I paid more attention in Stats 101 than I thought.

Photo Source: https://www.flickr.com/photos/cogdog/