« Joe Klein's sedition rhetoric | Main | Radio interview on misperceptions »

May 04, 2010


I'm the reader who initially called this poll question to Brendan's attention, which is not to say that he wouldn't have found it on his own or that he necessarily shares all my views about it. I'd like to use this question as the jumping off point for a couple of observations about opinion polls and the way in which they're reported.

The Times's reporting about this question is clearly deficient, reflecting either a sad misunderstanding of what profiling means or a mindset so wedded to the chosen narrative that the reporters and editors are unable to interpret the poll question correctly. One wonders whether if it turns out that the Arizona law does in fact net 20 illegal Latino immigrants for every illegal immigrant of another ethnicity, the Times will regard that as confirmation that profiling has occurred.

But looking beyond the way in which the poll is reported, let's consider the poll question itself. If the pollsters were trying to get a response about the issue of profiling, they obviously did a poor job framing the question. On the other hand, in light of the demographics of illegal immigration, the wonder is that anyone could answer the question other than that people of certain ethnic groups are likely to be detained more frequently than others. My suspicion is that a large number of the people answering "not too likely" or "not at all likely" didn't consider the question any more carefully than the Times's reporters did. They understood that the question was aimed at profiling, and they wanted to express their disagreement that profiling would occur.

When questions are as intricately phrased as this one, or as the question Brendan discussed last month about tax burden (a question I also first called to Brendan's attention), how really meaningful are the answers of respondents who are asked the question over the phone, with no chance to read it over or parse it and without the training that some of us have had in textual analysis? If the goal of the present question is to ask whether racial or ethnic profiling will take place, maybe the question should be asked in just that simple a manner. Sure, there will be differences in how people interpret "profiling," but the result of such a question is almost certainly more useful than the result for the intricate and off-the-topic question that was asked.

Some bad questions can be avoided by thoughtful survey design. Others can be discovered if the pollsters go through an additional step of pre-qualification of questions, asking potential questions of a small group of respondents (not tabulated in the final survey) and inquiring about why they answered as they did, in order to discover flaws in the question not originally intuited by the survey designers.

But the truth is that sometimes the public's opinions on issues can be best understood not by survey questions but by using focus groups or even by reading comments on newspapers like the Washington Post (which unlike the NYT allows comments) or by reading blogs. The problem is that these methods aren't scientific and don't yield statistics that can be subjected to quantitative analysis. Yet all the highfalutin quantitative analysis in the world can't cure the inability of respondents to parse and give a thoughtful response to an intricate and difficult question.

So maybe we should consider poll questions as useful when they present easily understood questions (which candidate one supports, right track/wrong track, which issues are of concern, etc.), but not when it takes an advanced degree to figure them out. And maybe those political scientists who revel in statistical analysis should stop pretending that the answers to complicated and easily misunderstood questions, no matter how thoroughly t-tested and cross-tabulated, yield any truly valid insights.

The comments to this entry are closed.