FAQs: Weighting

Why do we weight our results?

Accurate polling requires the demographic breakdown of a survey to closely resemble the same breakdown for the population you are trying to measure. For example, North Carolina likely voters breakdown about 53%-47% women to men and about 77%-18%-5% white to black to “other” and we try to have our surveys match those ratios as close as possible.

One drawback of IVR polling is that you are not sure of exactly who you are interviewing until the end of the survey. We can’t set quotas for demographics like traditional pollsters, so we just let the telephone calls run and then work with the data after the fact.

Traditional pollsters can manipulate their respondents during the survey by beginning each survey by asking for the “second oldest woman in the household” or some other method so that they reach their quotas for demographic groups like gender and race.

The most common demographic “problem” for us is that more women answer the surveys relative to men, and not enough African-Americans answer our surveys. To achieve relatively accurate demographic breakdowns we have to employ weighting schemes.

How do we weight?

The first step in weighting we achieve by surveying more than enough people. That allows us to go back and randomly reject individual surveys from demographics that are overrepresented. For example, if you saw one of our surveys with 500 respondents, in actuality 600 people may have answered the survey, but we had to reject 100 female responses. It’s like using a quota but after the fact, and our random selection eliminates any potential bias from the rejections.

We also employ a mathematical weighting scheme that will assign a weight based on one demographic. For example, if a survey is 82% white and 13% black, but needs to be 77% white and 17% black the weighting formula can take care of that mathematically.

It’s hard to be exact on each demographic when weighting, but we try to get our numbers as close as possible. This is when polling becomes somewhat of art. But our end results are available for all to see and to scrutinize.

Pre versus Post weighting

You would think that weighting would have an impact on the survey results. It does, but I am always amazed by how little an impact. Usually the numbers for our substantive polls questions only move 0-2% from before weighting to after weighting.