OpenBrand

Key Takeaways

• Reliable statistics depend on transparency about who ran the study, who was surveyed, and how the questions were asked.

• Small sample sizes and wide margins of error can make results too uncertain to support strong marketing claims.

• A representative sample matters, since “cherry-picked” respondents can skew results even when the math is correct.

• Incentives and missing context can produce technically true numbers that still mislead the audience.

• If a statistic contradicts established evidence, treat it as a cue to verify sources and look for confirmation elsewhere.

Reliable statistics have become increasingly important as the world continues to put emphasis on quantitative results. They show up in medicine, education, politics, and our favorite, market research. Statistics are easily accessible to anyone anywhere with today’s technology. Companies are using these numbers to attract consumers with lines such as ‘9 out of 10 teachers recommend THIS product!’ or ‘#1 cancer care center.’ Likewise, statistics can make for a compelling hook in a blog post or a social media post.

What Are Reliable Statistics?

Reliable statistics come from clearly described studies with sound methods, appropriate samples, and context that makes the numbers meaningful.

How to Spot Unreliable Statistics: 7 Factors to Look For

Businesses love to use statistics. Using data helps them sound impressive and authoritative. However, that may sometimes confuse consumers who don’t know how to evaluate for reliable statistics. Even businesses that contract out their marketing research or consumer data research may not fully understand the statistics that are returned to them. Numbers can be misleading for anyone, that is why we have identified seven characteristics to look for and avoid when searching for reliable statistics:

1. Statistics Benefit the Group Who Collected the Information

Suppose a toothpaste company releases results from a study showing those who use their toothpaste have fewer cavities. Just because this study was carried out by a company that benefits from the results does not mean that the results are unreliable. But we are not provided with any information on the study itself: information that is crucial to judging whether the statistics are reliable. Consumers and even businesses need to look carefully at factors such as sample size, audience selection, and the slant of questions asked. It is important to know the questions asked, who was being asked these questions, and why.

When reporting findings, companies must ensure they are clear about how studies were conducted as well as their findings.

2. Small Sample Size

The sample size is the number of people that are being surveyed. This size is crucial to the ending results. Having a sample size that is too small will result in findings that are not conclusive, despite its convenience to businesses. Researchers, or those conducting the study, should always disclose the sample size to ensure the reliability of the findings.

3. Error Margins Are Too Large

One way to get an idea about the sample size is by error margins. The smaller the sample size, the larger the error margins should be. It is also important to look at error margins for comparable research to see if the margins for the statistics in question are relatively small or large.

4. The Sample Representation Is Inaccurate or Biased

If a survey were conducted on how much businesses spent on marketing research, the findings would not be reliable if research were done only on small businesses. The cherry-picking of respondents does not lead to fair results. The best samples are probability samples because they are designed to account for the characteristics of only the group being studied. Other ways in which sample representation can be inaccurate or biased are if the survey was only carried out in one form or aren’t double-blind. For instance, if a survey were conducted through landline telephone calls only, those who only have cell phones, or no phone at all would not have been surveyed. As a caveat, such a project could be accurate if the study was for issues that only apply to landline telephone users. Double-blind studies are when participants in a study do not know which group they’re in. One example of this could be a double-blind taste test where participants do not know which cola drinks are in which glasses. This is the only way studies can avoid the risk of giving subtle cues to respondents or introducing biases, however unconscious, into the conclusions.

5. Incentives are Inappropriate for the Sample

Incentives are used, and sometimes required, to encourage people to participate in a research study. While this is not always the case, in some instances incentives can lead to poor participation and result in inaccurate reporting. A sign of biased and unreliable statistics is when respondents are only given incentives if they respond in a certain way. As an example, if a pollster says, “We will give you a free cellphone for answering this survey”, and the questions center on if consumers prefer that cellphone brand to other brands, the results cannot be conclusive.

There are other ways to look at incentives, to ensure reliable reporting. If a survey is listed to offer $30 cash for simply filling out the survey, respondents are most likely not going to put much thought into their answers. These answers may provide some results but are not fully reliable statistics.

6. The Context Is Not Reported

We often hear about a given statistic in the context of the end findings and results of research, but we are not given much more information. When looking for reliable statistics, this can be a red flag. Many times, journalists or news anchors are the ones reporting the findings, but these are not the ones who have conducted the study themselves. This means they may not fully understand the nature of the study, and this can sometimes be misleading.

Remember: just because something sounds authoritative does not mean it is authoritative or reliable. When a statistic says that people are now twice as likely to die from something, that could be an example of context not being reported. What were the odds of dying from that cause in the first place? If they were something like 0.00003 percent, then being two times more likely to die from it could technically be true but is still very misleading, as death from this cause is still rare.

7. The Statistic Flies in the Face of Precedent

What might someone think if a survey came out tomorrow saying that skin cancer is not all that common? (It is the most common of all cancers, according to the American Cancer Society and many other organizations.) Beware of statistics that go against the grain. They’re not necessarily wrong, but they are worth approaching with caution. Look at the groups sponsoring or carrying out the research. Are these reliable?

Why Do Reliable Statistics Matter for Businesses?

1. Many consumers are savvy. They know when something is not on the up and up, and it is often best for businesses to be straightforward about how they conducted research and reached their conclusions.

2. Businesses need to understand the statistics they quote or present to the world as well as how it was conducted. In this age of social media, it is too easy to share a cool statistic without doing due diligence.

3. Businesses need to be sure that the companies they work with for, say, tracking consumer data, are presenting information accurately and clearly.

FAQs

How Does Sample Size Affect Reliability?

Bigger samples reduce uncertainty, but quality matters too, since a large sample that is biased can still produce misleading results.

What Is Margin of Error, and Why Does It Matter?

Margin of error shows how much results could swing, so wide margins are a warning sign that the estimate may not support bold claims.

What Makes a Sample Representative?

A statistic is only as good as the people surveyed, so check whether the sample matches the population the claim is meant to describe.

How Can Incentives Skew Survey Results?

Incentives can help participation, but they become a problem when they attract low-effort answers or reward people for responding a certain way.

What Context Should Always Accompany a Statistic?

Numbers without baseline context can mislead, so ask what is being compared, over what timeframe, and how common the outcome is.

What Should You Do When a Stat Conflicts With Precedent?

When a stat clashes with established evidence, verify the source and look for replication before you treat it as a reliable signal.

Does Sponsorship Make Statistics Unreliable?

A sponsor’s stake in the outcome isn’t a dealbreaker, but you need to review the study details to judge methods, sampling, and question design.

Learn More About Proper Market Research With OpenBrand

By being aware of these pitfalls of misleading data and looking at signs such as sample size, methodology, and sample representation, a company can get a good idea of whether research is being performed accurately. Curious about what can happen when companies get the data wrong (or ignore the data entirely)? Check out our blog post: Top 5 Examples of How NOT To Do Market Research