More Attacks on Bad Surveys

I know, I know, I can’t stay away from this topic. Any wonder why though? We not only have sites like “A Liberal Dose” mis-citing accurate statistics1 we also have, as The Numbers Guy points out, bad surveys and polls being conducted.
Again, many of you will remember my discussions (here and elsewhere) on the importance of carefully developed question development as well as care in interpreting and drawing conclusions, without over-reaching. While I have recently criticized the interpretation of published results, The Numbers Guy in his article points out that the American Association for Public Opinion Research is tackling the problem of poor surveys making it into the media–and the media unquestioningly accepting the results.
Two relevant quotes:

Faulty survey data takes many forms. Sometimes the questions are loaded, as with a survey about online gambling I wrote about in April. Other surveys have very low response rates, like a poll about the value of mothers’ work; or pollsters don’t disclose all of their questions nor results, raising fears they’ve cherry-picked those responses that reflect best on the polls’ sponsors. Also, many polls you may read about have been conducted online, usually among a panel of volunteers lured by online ads — considered a less-representative sample by most pollsters than respondents who are found by random-digit telephone dialing.


Polls with pitfalls shouldn’t be discarded automatically. But often they are accepted automatically by the press and rendered indistinguishable from polls conducted by more standard means.

So there you have it. More criticisms of bad polling, bad data, and blind acceptance. And from a reputable source, to boot!
The Prof
1. And most recently in a fit of “hyper-‘bowl’-e” misrepresenting students’ putting marijuana in muffins as “spiking a punch bowl,” two very different things, to be sure!