Wednesday, January 27, 2010

Responding to ABC

ABC News Polling Director Gary Langer took on our survey about what tv news operations viewers do and don't trust yesterday on his blog.

I encourage you to read it but ultimately Langer's point is that you should only pay attention to polls that he finds acceptable, and to meet his standards for acceptability your poll pretty much has to be sponsored by a major news institution, foundation, or university.

Ultimately what I see coming through in Langer's writing is insecurity- not so much about the fact that ABC News came out poorly in the poll, but about the fact that he and the small fraternity of other polling organizations he finds acceptable no longer control the polling world. Organizations like Rasmussen and PPP that are getting more and more public attention and showing themselves to be as accurate or better than traditional pollsters are a serious threat to the folks who used to have the marketplace to themselves, and that's why you continue to see the likes of Langer cut us down whenever they have have the opportunity. They're still trying to fight the battle against people taking our polling as seriously as their polling, but they're losing it a little more day by day.

One of the most amusing things Langer and others in his cohort claim is that polls should not be judged by their accuracy- i.e. PPP, Survey USA, and Rasmussen are bad no matter how good their polls are at nailing election outcomes because he doesn't like the methodology. That attitude may still hold water in the small self appointed fraternity of elitist pollsters, but in our results oriented society most consumers of polls will continue to turn more and more toward organizations that have proven track records of accuracy.

One of the ironic things about ABC and similar polling operations is that they rarely poll anything they could be held to public account for. For instance the Massachusetts Senate race was undoubtedly of huge national significance but none of the outfits that pass Langer's litmus test of reliability conducted a pre election poll there that could have shown that they do- or don't- know what they're doing. We're just supposed to take their word for it that their polling is correct because they said so.

Langer went through our poll and nitpicked a bunch of stuff he didn't like from a demographic standpoint, generally because it didn't line up with his polls. You could go through and do that with any poll (and none of it would have significantly impacted the overall conclusions) but what I found most interesting was that he criticized the fact that the people we interviewed didn't report voting for Barack Obama by a 53-46 margin for President in 2008.

Beyond the fact (that I'm sure he would agree with) that we're not going to weight every poll for the next four years to the self reported 2008 vote, I find it most interesting that Langer levied no criticism of a June CBS/NYT poll where respondents reported having voted for Barack Obama by a 23 point margin. It's hard to justify that as a representative sample by any stretch of the imagination but Langer didn't see fit to say anything about that at the time probably because CBS is part of the insider club whose polls he sees fit for existence.

He also tries to undercut our finding that Fox News is the most trusted by citing a bunch of other polls while neglecting to mention a Sacred Heart University one last fall that found the same thing we did.

Whether ABC News does or does not cover our polls is really immaterial to our success as a company. Relying on the mainstream media to get your work out is so last century. But I do think organizations that continue to not report IVR polls do their readers/viewers a disservice. Three weekends ago two polls came out on the Massachusetts Senate election- one from the Boston Globe showing a 15 point lead from Martha Coakley and one from us showing a one point lead for Scott Brown.

Our poll got massive mounts of attention, but certainly not as much from the MSM gatekeepers as the Globe one. At this point it's pretty clear which of the polls gave a truer picture of the race but to Langer that doesn't matter- the Globe poll was still better because methodology is more important than results. He's entitled to that opinion- but I think few outside of the polling elite would agree.

12 comments:

MJ Napier said...

Kudos! Method is a good place to start but you're adding voices and broadening (an otherwise canned) converation. Congratulations, you're scaring them.

Matt Osborne said...

Silly pollster, with your silly statistical sampling methods! Don't you realize what matters isn't accuracy, but the MSM's exclusive right to be the filter of opinion?

Anonymous said...

Rasmussen can be a lousy polling organization and still predict "outcomes" accurately, because predicting outcomes is not the largest part of what they do. They have an interest in predictive methodology when they are actually predicting. Most of the time they are doing issue polling; that's when they get really cute with the methodology because they can. If they weight their poll to prove most Americans think Obama is a citizen of Kenya and born out of wedlock, there's not going to be a referendum on that issue to expose them. Election predictions are the bait for the public, while all these issue polls are the switch. They're weighted toward a Republican sample purely to give the impression that Americans have all joined the tea party. As different as the Wall Street Journal's reporters are from its loony editorial writers, that's how different its pre-electoral polling is from its issue polling.

Dave said...

I assume you do polling for clients. They are paying for accuracy, not methodology.

Bharat said...

Keeping them honest.
Good for you, Tom!

Pablo Kenney said...

The Scott Brown/Boston Globe example is not especially useful.

The Boston Globes'poll wasn't wrong because of their fielding methodology (which is much of what Langer is critiquing in your polls), but more because of how they constructed their "Likely Voter" model. Their model resulted in a self identified party breakdown of [DEM 56%, REP 27%, IND 15%] where as yours was [DEM 44%, REP 17%, IND 39%]. Even the Globe had Coakley polling extremely closely among self-described Independents [41 to 40]. PPP took a risk using a non-conventional LV model before it was obvious that MA would have a non-conventional election.

You deserve congrats for that risk, but it is not a paradigmatic example of bad MSM polls receiving more attention than a good PPP poll.

Christian Liberty said...

The Sacred Heart poll that showed that Fox was the organization most turned to for news also reported that "Respondents were asked if they selected their favorite because they offer objective reporting or because they view the issues as they did. In results that were nearly three-to-one, 59.0% suggested they made their selection based on objective reporting, while 19.0% chose their favorite because they share the same views on issues."

Once again, the data shows that people choose Fox News because Fox offers the most objective reporting.

John said...

That's why the media, at least old media, is dying.

Research said...

I just posted my own take on this, http://tinyurl.com/yacccwz. I'm as amused as you are at his "results don't matter" attitude, but I also think the "points" he tries to make about what's wrong with IVR research -- his claims of noncoverage, nonresponse, and so on -- are all garbage, and I get into why.

Old media is old.

mogando said...

haha i'm amused that people are trying to derail Rass' reputation even though it's now crystal clear who was the first one who saw the writing on the wall.

and funny how people think Rass always purposely skew their weighting to favor the right, while no one ever questions the big "D" after PPP.

stop shooting the messengers, esp the ones who have good track records of accuracy (on the long run) - e.g. SurveyUSA / PPP / Rass.... if a poll is unfavorable to your own candidate, slamming the pollster won't buy you any votes.

Anonymous said...

I'm amazed that teh guy at ABC would QUESTION the results of this Fox News poll --- when EVERYONE knows that Fox News has a HUGE lead in viewership. There's a reason for that. ya know?

Anonymous said...

Results don't matter...

you know doesn't matter???

Gary Langer.

the Don Quixote of polling...

 
Web Statistics