Margin of error alert: Obama not up in Iowa

As the NYT's Janet Elder points out (via Andrew Gelman), journalists are once again hyping minor differences in polling while neglecting to mention the margin of error in their results.

Here's a great example from the Des Moines Register's article on its new poll for the Iowa Democratic caucus (my emphasis):

Barack Obama has pulled ahead in the race for Iowa's Democratic presidential caucuses, while the party's national frontrunner Hillary Clinton has slipped to second in the leadoff nominating state, according to The Des Moines Register's new Iowa Poll.

...Obama, an Illinois senator, leads for the first time in the Register's poll as the choice of 28 percent of likely caucusgoers, up from 22 percent in October. Clinton, a New York senator, was the preferred candidate of 25 percent, down from 29 percent in the previous poll.

Former North Carolina Sen. John Edwards, who led in the Register's May poll, held steady with 23 percent, in third place, but part of the three-way battle.

...The poll shows what has continued to be a wide gap between the top three candidates and the remainder of the field. The telephone survey of 500 likely Democratic caucusgoers was conducted Nov. 25 to 28 and has a margin of error of plus or minus 4.4 percentage points.

Contrary to the article's lede, the margin of error on the poll means that Obama and Hillary are actually tied statistically -- Obama isn't winning!

Update 12/3 12:35 PM: Let me elaborate in response to angry comments below. Yes, given the margin of error and Obama's lead, the odds are reasonably high that he's ahead in a naive sense. However, his lead falls short of the standard 95% confidence threshold. While I'm not a big fan of the .95 standard, which is certainly arbitrary, in my professional work, I think it's appropriate here to to call the race a statistical tie (meaning we can't have much confidence in who is ahead) given both the margin of error and the uncertainties of projecting who will actually turn out. Along those lines, here's Elder:

News organizations differ on how strictly to apply the margin of sampling error. But when looking at horse race numbers in a political poll, particularly in Iowa, with its quirky caucus system, historically low turnout (5 percent of Iowans participated in the Democratic caucus in 2004) and rules that change from one year to the next — this year Iowans can register to vote at the door on caucus night — the margin of sampling error is probably best applied in its strictest sense.

If you don't believe me, check out the new poll (via Michael Crowley) showing Hillary at 31%, Edwards at 24%, and Obama at 20% (margin of error 6%) -- numbers that should make it quite clear that the Des Moines Register numbers cannot be interpreted strictly.

In the end, the best approach is to consider all of the polls. The pollster.com compilation of all Iowa Democratic polls through 11/29 makes it clear that the race is too close to call:

Update 12/3 7:47 PM: Via this comment, I learned that the poll cited above was actually in the field before the Des Moines Register poll, not after. Here's an LA Times blog post on the timing issue:

The Hillary Clinton presidential campaign, buffeted by the Des Moines Register's Sunday front page trumpeting its poll that gave Barack Obama a slight lead over her (28% to 25%) among likely Democratic caucus-goers, responded today with a release spotlighting two surveys that put her ahead in the Hawkeye state.

The Associated Press/Pew Research Center poll reported Clinton backed by 31% of the likely caucus-goers, followed by Obama (26%), John Edwards (19%) and Bill Richardson (10%). Iowa State University weighed in with these numbers: Clinton, 30.8%; Edwards, 24.4%; Obama 20.2%; Richardson, 11.4%.

In all three of the surveys, the advantage for the leader is within the margin of error. So basically, they all confirm one obvious point -- the Democratic race in Iowa is very tight and very fluid.

A closer look at the polls, however, reveals a potentially key difference between the Register's survey and the other two: timeliness.

The newspaper's poll was conducted from Nov. 25 (Sunday a week ago) through last Thursday. The AP/Pew survey was conducted Nov. 7-25, while the Iowa State poll was in the field Nov. 6-18.

Pollsters like to refer to their findings as "snapshots in time." The AP/Pew and Iowa State polls strike us as a bit lengthy in the development stage.

I'll leave the technical details of the statistics debate to the comments.

Comments

[Lettuce]

And Edwards is how far back?

[Dave]

Read the Elder article, which is quite good, but I still think your analysis of the poll isn't quite right either. Obama and Clinton aren't so much "statistically tied," as there just isn't 95% confidence that Obama is definitely ahead.

I'm not a statistician so I don't know the exact formula, but I think a more correct headline would be "Obama not definitely leading in Iowa, but poll indicates X% confidence that he is."

[Dave]

Put another way, a true "statistic tie" would be a result of 28% to 28%, because at that point the statistics would be equally confident that either Obama or Clinton is ahead. These results don't show that, however; the statistics are more confident that Obama is leading, though not 95% confident.

[Larry]

No, no it doesn't. It means there's still a reasonable chance that Clinton is ahead, but it is in fact more likely that Obama leads, given the results of the poll.

Also, I love quoting the margin of error to 1 decimal place, but the poll results to integer percentages. With rounding, Obama might lead by very close to the margin of error here, which would give Clinton only a 5% chance of leading. That would still be a statistical tie, according to the conventions you advocate here, but Hillary would very likely be behind.

[cms]

You need to better understand the meaning of "margin of error." Here's a good primer:

http://www.washingtonmonthly.com/archives/individual/2004_08/004536.php

In the DMR poll, there is an 82% chance that Obama leads.

[Michael Young]

The phrase "statistical tie" is *wildly* misleading, far worse than suggesting that a 3% difference is significant here. To blithely say the leader "isn't winning" is even more misleading; the leader certainly isn't "losing".

It is still *much* more likely that the leader would win a poll of the entire population, just not quite to the standard confidence level. Had it been a 1% difference, I might have cut you some more slack. But a 3% difference here is not entirely trivial. Your attempt to trivialize it suggests that you don't really understand what "margin of error" means.

[Lettuce]

So, second guess:

I'm guessing your just ignoring Edwards?

[Larry]

The "new" poll was conducted over Nov 7-25, which is a very long time, and also ends as the DMR poll began. So, they're not measuring the same thing at all.

The statement:

But when looking at horse race numbers in a political poll, particularly in Iowa, with its quirky caucus system, historically low turnout (5 percent of Iowans participated in the Democratic caucus in 2004) and rules that change from one year to the next — this year Iowans can register to vote at the door on caucus night — the margin of sampling error is probably best applied in its strictest sense.

is nonsense. That's saying that not knowing about all kinds of inestimable systematic errors we should use the value of the sampling error as a flat confidence interval. I'm sure you've done posts before on the fallacy of using made up numbers as a starting point to revise into something useful, and that logic certainly would apply here to the statement above.

One should use the statistical confidence interval for its defined purpose or not at all.

It should also be noted that the sampling error does have its meaning altered when there are more than two choices and the measured probabilities are far away from 50%. I'll defer to the indispensible pollster.com for the details, though for the poll under discussion the change is minimal.

[Brendan Nyhan]

Thanks for the tip, Larry - I've added details on the poll's timing above. I want to be clear that I'm taking Elder's statement in a heuristic sense. Minor differences in numbers in a single poll are not particularly meaningful. The CI is not especially helpful, but requiring the press to use a .95 standard would at least constrain them from reporting tiny, fluctuating differences as "news" and telling stories about what caused those differences. The kind of big leads we could be confident are real are likely to exceed the .95 threshold.

PS This whole comment thread can be interpreted as a meta-debate about frequentist versus Bayesian statistics. Everyone implicitly wants to be a Bayesian and interpret p-values directly, but the frequentist paradigm is ill-suited to making those sorts of inferences.

[Larry]

I think I actually managed to get the dates wrong above, not noticing there were actually two old, lengthy polls puting Clinton ahead. That's all cleared up in your latest update.

True enough about the meta-debate, and the press is generally ill-qualified to be trusted to understand this to any level of detail beyond the 'statistical tie' nomenclature (assuming most of them even understand that). But, shouldn't the goal be to have a better informed press corps?

It is also true that the press over-reacts, looking at the polls in the horse-race sense and then writing stories "mind-reading" what the electorate must be thinking. But, again, I don't think the way to combat this is by forcing a simpler view of reading polls onto the press.

[Michael Young]

> In the end, the best approach is to consider all of the polls.

Agreed. If your point was "other polls suggest a closer race", then it would have been better to say so without suggesting that the (factually correct) reporting on this single poll was flawed.