Bad CBS/NYT poll question on AZ profiling debate

A New York Times article on the latest CBS/NYT poll (PDF) suggests that a majority of Americans believe the new immigration law in Arizona "would result in racial profiling":

[D]espite protests against Arizona’s stringent new immigration enforcement law, a majority of Americans support it, even though they say it may lead to racial profiling...

[T]the respondents broadly agreed that the Arizona law would result in racial profiling...

However, as a reader noted, the poll question featured in a sidebar to the article doesn't ask about racial profiling, which is typically defined as targeting individuals solely based on their racial or ethnic background. Instead, CBS and the Times asked the following:

How likely do you think it is that the new law in Arizona will lead to police officers detaining people of certain racial or ethnic groups more frequently than other racial or ethnic groups? Do you think that is very likely to happen, somewhat likely, not too likely or not at all likely to happen?

Given the composition of the illegal immigrant population, Latinos will almost certainly be detained more frequently than other racial or ethnic groups under any enforcement regime in Arizona or any other state (particularly in comparison with their representation in the population). The relevant policy issue is whether the Arizona law will lead to detentions of Latinos based solely on their ethnic background. The Times article vaguely notes that "the Arizona Legislature and Gov. Jan Brewer made changes to the law on Friday that they say explicitly ban the police from racial profiling," but doesn't specify that the changes bar consideration of race or ethnicity in enforcement "except to the extent permitted by the United States or Arizona Constitution" by removing the word "solely" from the following provision:

A law enforcement official or agency of this state or a county, city, town or other political subdivision of this state may not solely consider race, color or national origin in implementing the requirements of this subsection except to the extent permitted by the United States or Arizona Constitution.

The poll question should have asked if people believe that this provision will be upheld in practice -- that's the question in dispute right now.

[Cross-posted to Pollster.com]

Comments

[Rob]

I'm the reader who initially called this poll question to Brendan's attention, which is not to say that he wouldn't have found it on his own or that he necessarily shares all my views about it. I'd like to use this question as the jumping off point for a couple of observations about opinion polls and the way in which they're reported.

The Times's reporting about this question is clearly deficient, reflecting either a sad misunderstanding of what profiling means or a mindset so wedded to the chosen narrative that the reporters and editors are unable to interpret the poll question correctly. One wonders whether if it turns out that the Arizona law does in fact net 20 illegal Latino immigrants for every illegal immigrant of another ethnicity, the Times will regard that as confirmation that profiling has occurred.

But looking beyond the way in which the poll is reported, let's consider the poll question itself. If the pollsters were trying to get a response about the issue of profiling, they obviously did a poor job framing the question. On the other hand, in light of the demographics of illegal immigration, the wonder is that anyone could answer the question other than that people of certain ethnic groups are likely to be detained more frequently than others. My suspicion is that a large number of the people answering "not too likely" or "not at all likely" didn't consider the question any more carefully than the Times's reporters did. They understood that the question was aimed at profiling, and they wanted to express their disagreement that profiling would occur.

When questions are as intricately phrased as this one, or as the question Brendan discussed last month about tax burden (a question I also first called to Brendan's attention), how really meaningful are the answers of respondents who are asked the question over the phone, with no chance to read it over or parse it and without the training that some of us have had in textual analysis? If the goal of the present question is to ask whether racial or ethnic profiling will take place, maybe the question should be asked in just that simple a manner. Sure, there will be differences in how people interpret "profiling," but the result of such a question is almost certainly more useful than the result for the intricate and off-the-topic question that was asked.

Some bad questions can be avoided by thoughtful survey design. Others can be discovered if the pollsters go through an additional step of pre-qualification of questions, asking potential questions of a small group of respondents (not tabulated in the final survey) and inquiring about why they answered as they did, in order to discover flaws in the question not originally intuited by the survey designers.

But the truth is that sometimes the public's opinions on issues can be best understood not by survey questions but by using focus groups or even by reading comments on newspapers like the Washington Post (which unlike the NYT allows comments) or by reading blogs. The problem is that these methods aren't scientific and don't yield statistics that can be subjected to quantitative analysis. Yet all the highfalutin quantitative analysis in the world can't cure the inability of respondents to parse and give a thoughtful response to an intricate and difficult question.

So maybe we should consider poll questions as useful when they present easily understood questions (which candidate one supports, right track/wrong track, which issues are of concern, etc.), but not when it takes an advanced degree to figure them out. And maybe those political scientists who revel in statistical analysis should stop pretending that the answers to complicated and easily misunderstood questions, no matter how thoroughly t-tested and cross-tabulated, yield any truly valid insights.