How to read survey data

As it gets closer to our (American) mid-term elections, we’re about to be inundated with surveys and polls. But, even between elections, surveys are everywhere, for better or worse.

To help filter the signals from the noise, here is my list of tips for critically reading reports based on survey data that I’ve collected over the years.

If you’re a reader of survey data, use these tips to help you interpret survey data you see in the future.

If you’re publishing survey data, be sure to consider these as well, especially if your readers have read this post.

To critically read survey data, you need to know:

  1. Who was surveyed and how
  2. What they were asked
  3. How the results are stated

Let’s look at  each of these a bit more…

Continue reading “How to read survey data”

Lies, damn lies, and statistics

Chart of science-public split on science-related issues (from fivethirtyeight.com)
Science-public split on science-related issues (from fivethirtyeight.com)

The headline from stats web site, http://fivethirtyeight.com/, says Americans And Scientists Agree More On Vaccines Than On Other Hot Button Issues while the headline from Mother Jones, reporting on the same data, says, This Chart Shows That Americans Are Way Out of Step With Scientists on Pretty Much Everything. You wouldn’t know from the headlines that they were reporting from the same data.

If you ignore the text and just look at the data in each chart, while the chart from http://fivethirtyeight.com/ breaks the data out by political-party affiliation, the numbers do appear to be the same (or the same enough). The rhetoric and visualizations, however, are quite different.

Things that make you go, “hmmmm…..”

Or, another way to look at it is that it always pays to go to the source data.

Chart showing opinion differences between public and scientists from Mother Jones
Opinion differences between public and scientists from Mother Jones