Drudge is touting a new Zogby poll showing Hillary trailing top Republicans, but the Reuters article admits in the last paragraph that it's based on an unusual sample:
The poll of 9,355 people had a margin of error of plus or minus one percentage point. The interactive poll surveys individuals who have registered to take part in online polls.
Absent further details, it's not clear why we would believe this to be a useful finding. Zogby's summary and defense of its approach is here, but let's just say that the jury is still out. Here's some past coverage in which a political scientist expresses skepticism:
When reached by phone last week, Cliff Zukin, a political science professor and polling expert at Rutgers University, suggests that journalists should generally be wary of any Zogby interactive poll.
“The Zogby stuff, on scientific grounds, is quite questionable,” says Zukin. “Online, Internet, opt-in polling, where people volunteer to be respondents, doesn’t really have a basis in scientific validity. There are two kinds of samples in the world. There are probability samples, and there are non-probability samples.”
The Zogby interactive polls, says Zukin, clearly fall into the latter camp. “With probability samples, when everybody has a known chance of being selected, you can make pretty valid inferences about the population from which it is drawn,” says Zukin. “You can’t do that at all with self-selected surveys. That’s a problem.”
Another problem with Internet-based polling, says Zukin, is that, in general, Web and email-based surveys tend to overvalue the opinions of young people. A group that is notoriously lousy at showing up to actually vote.
“Internet coverage is now about two-thirds of the population,” says Zukin. “But it’s really age-skewed and, to a lesser extent, education-skewed, in the wrong way for voters. It’s younger people who are online. It’s older people who are not online. It’s older people who vote. And younger people who don’t.”
“It’s certainly not the gold standard,” says Zukin.
In short, without further confirmation from more traditional polling, don't take the Drudge headline too seriously.
Update 11/27 10:01 AM: The University of Wisconsin's Charles Franklin has a detailed post showing that the Zogby results are way out of whack with more traditional estimates:
The hugely surprising result is that the Zogby poll finds Sen. Hillary Clinton losing to all four top Republicans in head-to-head trial heats. What makes that surprising is that Clinton LEADS all four of those Republicans in the trend estimates based on all other polling by between 3.8 and 11.6 points.
...What is immediately clear is that the Zogby Clinton numbers are well below the estimated trend for Clinton in each of the four comparisons. Clinton is consistently 8-10 points below her trend estimate based on other polling.
Meanwhile, Hillary's Fact Hub blog points out that a more traditional Gallup poll was also released yesterday and it found results more in line with previous findings:
A new Gallup Poll finds Sen. Hillary Clinton with a slim but not statistically significant advantage over both former Mayor Rudy Giuliani and Sen. John McCain in head-to-head matchups for the 2008 general election for president. Clinton has much more substantial leads over former Sen. Fred Thompson and former Gov. Mitt Romney. Sen. Barack Obama also has significant leads over Thompson and Romney, but essentially ties with Giuliani and McCain.
...The results for Obama matched up against the Republican candidates are largely similar to those for Clinton. He has substantial leads over both Thompson and Romney, and is highly competitive with McCain and Giuliani. Though Clinton's margins of support over McCain (six points) and Giuliani (five points) are larger than Obama's margins of support against the same candidates (three points and zero points, respectively), the differences are not large enough to be considered meaningful from a statistical perspective.
Update 11/28 11:18 PM: Via Kos, an analysis of Zogby Interactive's track record provides further corroboration of my suspicions:
I looked at five pollsters that were among the most prolific: Rasmussen, SurveyUSA, Zogby (which releases separate telephone and online polls) and Washington, D.C.-based Mason-Dixon. For all but the latter, I used the numbers posted on the organizations' own Web sites. For Mason-Dixon, which keeps some of its poll data behind a subscriber wall, I used Pollster.com to find polls from the two weeks before the election. I checked the results against vote counts as of this Tuesday.
...In the Senate races, the average error on the margin of victory was tightly bunched for all the phone polls. Rasmussen (25 races) and Mason-Dixon (15) each were off by an average of fewer than four points on the margin. Zogby's phone polls (10) and SurveyUSA (18) each missed by slightly more than four points. Just four of the 68 phone polls missed by 10 points or more, with the widest miss at 18 points.
But the performance of Zogby Interactive, the unit that conducts surveys online, demonstrates the dubious value of judging polls only by whether they pick winners correctly. As Zogby noted in a press release, its online polls identified 18 of 19 Senate winners correctly. But its predictions missed by an average of 8.6 percentage points in those polls -- at least twice the average miss of four other polling operations I examined. Zogby predicted a nine-point win for Democrat Herb Kohl in Wisconsin; he won by 37 points. Democrat Maria Cantwell was expected to win by four points in Washington; she won by 17. (Zogby cooperated with WSJ.com on an online polling project that tracked some Senate and gubernatorial races.)
The picture was similar in the gubernatorial races (where Zogby polled only online, not by phone). Mason-Dixon's average error was under 3.4 points in 14 races. Rasmussen missed by an average of 3.8 points in 30 races; SurveyUSA was off by 4.4 points, on average, in 18 races. But Zogby's online poll missed by an average of 8.3 points, erring on six races by more than 15 points.
The first question one needs to ask about any Zogby poll is: who paid for it? Only then should you bother to look into the unconventional procedures.
I understand why Drudge might take Zogby seriously when the outcome suits him. Why anyone else does at this point is really beyond me, though.
Posted by: Michael Young | November 27, 2007 at 01:19 PM