What is John Zogby talking about?

Zogby International, the polling firm of John Zogby, announced its latest poll yesterday with a press release headlined "No Bounce." The release states that "President Bush's televised address to the nation produced no noticeable bounce in his approval numbers, with his job approval rating slipping a point from a week ago, to 43%, in the latest Zogby International poll."

But if you read the details of the poll, you will find out that it was conducted from June 27-29. Bush gave his speech during the evening of June 28. So how could Zogby tell if Bush got a bounce when a substantial portion of his respondents were interviewed before the speech? This seems like another example of Zogby's questionable professional ethics to me.

Yet despite this obvious flaw, the poll was touted by Bloomberg as showing that Bush did not receive a bounce:

Bush's Iraq Speech Fails to Rally Support, Poll Finds

U.S. President George W. Bush's public approval rating remained in negative territory after a nationally televised address on Iraq, a new poll has found.

The poll by Utica, New York-based Zogby International reported that 43 percent of those surveyed approved of Bush's job performance and 56 percent disapproved. The poll of 905 people was taken June 27-29 and had a margin of error of 3.3 percentage points.

What's especially strange is that reporter Ken Fireman notes later in the article that Bush's speech took place on June 28, but doesn't see fit to mention the contradiction between that date and the period in which the poll was conducted. (Atrios also picked up the release and quoted it without comment.)

As I wrote before, Bush is ultimately unlikely to get any significant boost from the speech. But to demonstrate that, we need a poll conducted after the speech. This is not complicated. So why can't Zogby and Bloomberg get it right?

Clarification 7/2: As a commenter correctly points out, I somehow missed this statement in the press release:

The Zogby America survey includes calls made both before and after the President’s address, and the results show no discernible "bump" in his job approval, with voter approval of his job performance at 45% in the final day of polling.

Apologies for the error. But it doesn't change my conclusion. The press release initially touts the overall finding of an approval level of 43% as evidence of a lack of a bounce. That's just wrong. And the fact that the approval level in the third day is close to his previous one isn't proof of anything. Assuming a proportional number of calls across the three days, Zogby made about 300 calls in the third day. As such, his margin of error for polling on that day is much larger than the +/- 3.3% margin of error for the survey as a whole, which means that he can't conclude much of anything from the 45% figure.

Update 7/2: A Gallup poll conducted after Bush's speech (6/29-6/30) shows Bush's job approval at 46%, which is not statistically distinct from the previous level of 45% (6/24-6/26). It also shows that responses to questions about Iraq appear to be relatively stable, though pro-war sentiment bumped up 4% on two questions, which is right at the limit of the poll's margin of error.

Comments

[fact checker]

Dukes standards have fallen as of late apparently.

If you actually read Zogby's press release, the organization address this directly:

The Zogby America survey includes calls made both before and after the President’s address, and the results show no discernible “bump” in his job approval, with voter approval of his job performance at 45% in the final day of polling.

-A friendly plastician.

[Brendan Nyhan]

Thanks for letting me know, and apologies for the mistake. A clarification is posted above.

[Foreign pollster]

Re: your updated point. Zogby's conclusion, while perhaps misleading because he didn't explicitly state that sample sizes pre- and post-speech are smaller, is not technically wrong. Even with sample sizes of 300, the margin of error would be about 5 to 6%, which is still respectable. His conclusion was that there was no significant change in rating before and after the speech, and that is correct. Until further polling, all that can be said is that Bush's speech doesn't appear to have had any impact on his rating.

[Brendan Nyhan]

The issue turns on what sort of a bounce we're expecting, and what the resulting empirical standards are. Yes, Zogby's June 29 numbers suggest some sort of massive bounce didn't occur, but that was already wildly implausible (see George Edwards' On Deaf Ears - most major presidential speeches have little or no effect on poll numbers). For more fine-grained questions about the effect of the speech, however, Zogby's numbers from his last day of polling establish little or nothing -- the margin of error is simply too large.