In an column published yesterday by the New York Times Ross Douthat aruges we shouldn’t take outlandish conclusions from polls seriously. He cites the recent Vanity Fair and 60 Minutes poll* showing that 24% of Americans don’t think Obama was born in the US. He writes that this number is misleading because many Americans are ignorant—they are unaware of law or geography, therefore the number does not encompass the majority of Americans’ beliefs.
While he may be correct on some of his assumptions, I think he misses the bigger picture. This isn’t about the public’s IQ or common sense. The fact that some Americans don’t believe Hawaii is a state is not only important for us to know (proof that we need more funding for our schools) but it is also a factor in shaping public opinion. Their opinion is just as valid as Douthat’s or mine and the ballot box certainly doesn’t discriminate between us. The voter who doesn’t know that Hawaii is a state may be more easily persuaded by Birthers that Obama isn’t lawfully the President than someone who does know that Hawaii is a state. While there may be some grain of truth in Douthat’s assumption that conspiracy theorists have high IQ and low common sense, it doesn’t mean that their followers do.
This isn’t about the public’s IQ or common sense it is about the pollster IQ and common sense. It is about asking smart questions that unfold many of the nuances in the American public perception. Douthat used our poll to prove his point and I am going to use it to prove mine.
The question in the Vanity Fair and 60 Minutes didn’t extract many facts and was poorly worded. It could have been confusing to someone who doesn’t know their geography. It asked one question forcing respondents to not only know that the President was born in Hawaii but that Hawaii is part of the United States. While that may seem fair—remember a year ago 6% of Americans didn’t know Hawaii was part of the US and 4% were unsure.
PPP however asked several simple questions that helped us discover why people may think Obama wasn’t born in the U.S. By asking more then one question and using crosstabs we were able to account for those who don’t know their geography, but still believe in the President was born in U.S. If a respondent didn’t believe Obama was born in the U.S. we then asked follow up questions, allowing us to gather more information and have better more accurate results.
Crosstabs can often tell us more about the public than the mere question. It is not simply the singular question that matters but the series of questions that counts in drawing conclusions. Broadly both polls had similar results: PPP found that 62% of Americans believe Obama was born in the U.S., 60 Minutes and Vanity Fair found 63% (ours was taken almost a year ago so our data is old). But when you look deeper into the results at crosstabs you’ll find differences.
By looking at our crosstabs PPP is able to tell that 14% of Americans, who don’t believe Obama was born in the US, don’t know Hawaii is part of the Union. 64% of those who don't think Hawaii is a state don’t believe Obama was born in the United States. Vanity Fair/60 Minutes’ poll assumed that people knew that Hawaii was in the US—many of those who answered Hawaii could have meant another country.
PPP’s poll asked respondents if they supported Obama. By looking at the crosstab of those who disapprove of Obama and are skeptical or don’t believe Obama lives in US, we are likely to discover the “Birther” group amongst us. A year ago 79% of those who are convinced he wasn’t born here and 67% of those who are unsure disapprove of Obama. While more questions would be needed to guarantee these voters believed this (like civics questions), I wouldn’t consider the others who don’t know where Obama was born “Birthers.”
CBS’s Brian Montopoli disagrees, he wrote that according to an ABC News/ Washington Post poll, one in three “Birthers” approve of the president’s job performance. He incorrectly defines “Birthers” as those who simply believe the president wasn’t born in the United States. As we have discovered through our polls some Americans don’t know geography or civics, they aren’t necessarily part of the “Birther” movement. The Birther movement is a group who believes the Obama is unlawfully President because he wasn’t born in the United States. In this case I agree with Douthat, this is using polling data to jump to outlandish conclusions.
While 60 Minutes speculates if it was the success of “right wing conspiracy” or the failure of our “academic instruction,” we at Public Policy Polling may be closer to having the answers.
A quality of a poll can often be found in the quality of the questions. We shouldn’t discount people’s opinions but polls that don't capture them well. Polling is about asking questions that everyone can answer. Polls are simply a form of data collecting. The respondent shouldn’t be forced to do the work, the pollster should.
(*The Vanity Fair/60 Minutes question was posted differently on their respective website pages, CBS does not typically publish all of their polling data. This analysis is based on the information that was available to the public.)
The bigger part of the puzzle is that people may not answer poll questions strictly on the facts if they're not wholly comfortable with the prompted choices.
ReplyDeleteWhen given a choice between evolution and creationism, many people who believe in evolution and also in God as creator will pick "creationism".
Those who don't like Obama might say he's not born in the US just as an anti-Obama response, while those who don't like Bush might say he's part of a 9/11 conspiracy just to register an anti-Bush response.
When pollsters ask stupid questions they're asking for stupid answers, and that's probably what they'll get. It's just dishonest when media outlets turn around and treat the crappy poll they designed as anything more than waste.
Good post. I often feel that the major failing of pollsters is in the sorts of questions that are asked, and the subtle biases presented by it.
ReplyDeleteOpinion Dynamics (Fox's pet pollsters) are masters of this art, both on the actual polling side, and in the way they report the polls. I recall reading a series of their poll results in early '09: it was a lengthy poll, about 40 questions, but the answers were published out of sequence and in an unrelated fashion, and there were several questions that were not published.
Question order is a very important mechanism in polling, playing on the psychological concept of priming: for instance, asking a question about culpability regarding 9/11, then asking if you think Democrats are soft on defense, or softball questions about how you feel about celebrities followed by a question about how you feel about the Iraq & Afghanistan wars.
The true misfortune is that the profligacy of these hucksters discredits all pollsters to a great degree, because your average person on the street won't know which polling companies do what, and who to trust, and so, if they even know that most polls are unadulterated BS, they're likely to discredit ALL polls, not just ones run by crappy pollsters.
Good post, thanks!
People read Douthat?
ReplyDeleteI think you do need more analysis. I think a good share of birthers have decided he wasn't born in the U.S. because they don't want him to be President. It's a simple explanation for something so complicated.
ReplyDeleteIf you don't approve of someone making him illegitimate reinforces your belief. I know a number of otherwise intelligent people who believe that George W. Bush didn't win the 2004 election because he "stole Ohio." When pressed to provide any evidence for this they say things like, "the exit polls were different" as if the presence of an oddity proves anything. I've never been able to shake any of them of this notion.
But the only question that anyone ever asks Democrats that make them look crazy are those about 9/11.
If you asked people you know what year Obama was born and what year Hawaii became a state I'd bet half can't give the exact right year. If Obama were born in Hawaii before it was a state I could understand people thinking it wasn't part of the U.S.
"64% of those who think Hawaii is a state don’t believe Obama was born in the United States."
ReplyDeleteI think you've got that turned around backwards. That doesn't sync with what I see in the poll data, or with common sense.
Thanks Loren, your right that was a typo!
ReplyDeleteWhen did Hawaii become a state?
ReplyDeleteI think a key point here is that the way a question is worded is so important. If I were asked whether I believe the President was born in the U.S., I'd answer "No", because it's not a matter of belief, it's a matter of fact. He was born in the U.S. Now, on the other hand, if asked simply whether the President was born in the U.S., I'd of course answer "Yes", because it is true.
ReplyDeleteSo, I don't think pollsters should use terms like "believe" for questions of fact. They should instead ask "Is this or is this not true?" By including a "Don't know" response, they could then dig further with those folks on belief. But using belief in the original formulation biases the response in my opinion.