Beware context-free election analysis

Since the polls closed on Tuesday night, we've seen a rash of context-free analyses of the election results. Just as some commentators have interpreted the GOP's 60+ seat pickup as purely a backlash against Obama and his policies without considering the structural baseline, these analysts have typically focus on a single vote or characteristic shared by a group of House candidates without considering the context of those members' districts.

However, as John Sides notes, Democratic vote totals are very highly correlated with measures of district partisanship such as Obama's total in the district in 2008:

As such, any claim about influences on the election outcome needs to consider marginal effects relative to that baseline.

Here are just a few examples of how analysis can go wrong:

-NBC's First Read writes that "For all the talk of the Tea Party's strength - and there will certainly be a significant number of their candidates in Congress - just 32% of all Tea Party candidates who ran for Congress won." 32% sounds like a low number. However, most of the losing Tea Party candidates were running hopeless races against incumbents in heavily Democratic districts. Those in competitive districts seemed to perform about as well as other Republicans.

-Daily Kos campaign director Chris Bowers bashed conservative "Blue Dog Democrats," writing that "23 of 48 Blue Dogs standing for re-election have lost. Anyone who loses that badly doesn't get to tell Dems how to win." However, most Blue Dogs represented competitive districts and were thus most likely to be endangered by a GOP wave. The fact that they lost disproportionately doesn't mean that they were wrong about the Democratic message.

-Under the headline "Crushed," Josh Marshall noted that "Of the 39 Dems who voted against Health Care Reform, 12 are going to be returning in the next Congress," suggesting that opposition to reform was a cause of these members' defeats. However, he failed to note that most of anti-HCR members represented competitive districts and that vulnerable Democrats who opposed reform appeared to perform somewhat better than their pro-HCR counterparts.

-The Natural Resources Defense Council claimed to refute arguments that support for the cap-and-trade bill hurt Democrats by constructing the following graph:
Again, however, the opponents of cap-and-trade were more likely to represent marginal districts. The fact that they lost at higher rates doesn't mean that cap-and-trade didn't hurt its Democratic supporters.

For an example of a better approach, consider Eric McGhee's post on The Monkey Cage. Consistent with my finding that Democrats from competitive districts who supported health care reform appeared to perform worse on Election Day, he finds that House Democrats who supported health care reform, TARP, the stimulus bill, and/or cap-and-trade performed worse than those who did not controlling for the partisanship of their district. This finding directly contradicts the simplistic claims above about the effects of health care, cap-and-trade, and the Blue Dog agenda.

I replicated this estimate* and it appears to be robust. To illustrate the finding, here is a plot of the estimated marginal effect of an additional vote for one of the four controversial bills. I restrict my analysis to Democrats in potentially competitive seats (i.e. where President Obama received less than 60% of the vote in 2008):

The y-axis represents the estimated effect of an additional vote for a controversial bill on the predicted Democratic vote. The plot shows that this effect becomes increasingly negative as the district becomes less favorable to Democrats (lower Obama vote in 2008). While it is still very early, this finding contradicts the claims above, suggesting that the Democrats' aggressive legislative agenda may have provoked a backlash that was at least partly responsible for GOP overperformance in the House.

Update 11/4 9:09 PM: I should note that my Political Analysis article with Jacob Montgomery (PDF) reached a similar conclusion about the conditional effect of legislative extremism using a more comprehensive dataset:

[T]he effect of roll-call extremity on election results is moderated by party strength in the district (as measured by the CWBC presidential vote variable)... Substantively, these results indicate that members from very marginal districts suffer severe punishment for legislative extremity but the electoral cost of extremity declines rapidly as party strength in the district increases. In those districts in which the party is strongest, the marginal effect of roll-call extremity is actually either negligible (i.e. the 95% confidence interval includes zero) or positive. In other words, members are punished to the extent they are out of step with their district.

Update 11/5 7:52 AM: Chris Bowers objected to my quotation of his tweet above. I asked him to clarify his point -- here is what he wrote (compiled from a series of tweets):

The claim is that Blue Dogs have no special knowledge about how Dems can win elections. It's hard to phrase in a couple of tweets, but here goes... Basically, I don't think that the voting record or ideology of a Dem candidate mattered that much in electoral outcomes. What I do think is that the economic policies Blue Dogs prevented from passing would have helped Dem chances.

It's quite possible that a larger stimulus would have helped the Democrats; no arguments here on that point. As for the first claim, see above -- both Eric McGhee and I find that votes for highly controversial Democratic proposals may have hurt Democrats in marginal districts. But regardless of the merits of either claim, the fact that Blue Dogs lost disproportionately (Bowers's original point) is not evidence of much besides the fact that they represent swing districts.

* Like McGhee, I regressed the Democratic incumbent's vote total on an additive index of the number of the four bills that the member supported interacted with the two-party presidential vote for Obama in 2008. I also controlled for the district's Democratic House vote in 2008. However, I exclude the campaign spending variables McGhee included since they are potentially endogenous. I also exclude incumbents who did not have a Republican challenger.

[Cross-posted to HuffPost Pollster]

Comments

[MartyB]

Interesting analysis Brandan. Nice job on testing the narratives now floating around out there in left-leaning land.

James Tranto has some snippets of "interesting" narratives from left leaning pundits from just the past year and a half that add to the sense that liberal pundits and politicains really didn't realize that their legislative priorites might produce any sort of backlash:

http://tinyurl.com/237s8x5

Carville; May 2009: "Republicans have no hope of making serious inroads into Democratic advantages in 2010, or likely in 2012 and 2014 and so on."

Mark Halperin; April 2010, "the President is on a path to be a huge success by the time of November's midterm elections."

Ryan Grim and Amanda Turkel / Huffington Post; Sept 2010:
"Tom Perriello in Virginia, Carol Shea-Porter in New Hampshire, Alan Grayson in Florida, Mary Jo Kilroy in Ohio and John Hall in New York have locked down support among their base and are winning over independents".

Of course anybody can be wrong, but these are spectacularly wrong. Makes you think twice about whose analysis to listen to.

[Jinchi]

One thing you still haven't accounted for in your chart is the fact that a different subset of the population turned out in 2008 compared to 2010. I have little doubt that supporting liberal bills makes a candidate unpopular among conservatives, who apparently turned out in larger numbers Tuesday. But I also expect opposing liberal bills would make you unpopular with liberals.

The problem is that candidates don't know beforehand whether their vote will demoralize their own voters, energize their opponents or have no effect on turnout at all.

My guess is that your chart would be shifted well into the positive column if you ran the same test of the Marginal Effect on Vote Percentage against the candidates running in 2008.

But that doesn't mean those votes were motivators. It simply means that they weren't liked by the voters who showed up on this particular election day.

[bnyhan]

Jinchi, you're right that the electorates differ. This is the estimated marginal effect for the most recent election, not a claim about all potential electorates. It's certainly likely that things would have been different with the '08 electorate, but that was always unlikely to be repeated -- I'm not sure we've ever had a midterm wave election in favor of the party controlling Congress and the presidency.

[David in Cal]

With certain liberal elements finding ways to not believe that their agenda was unpopular, how do they explain their big loss? Some attribute it to a fantasy version of what conservatives want:

There were those who wanted to take us back to a time when children could be made to work in mines and factories, when workers had no legal rights to speak of, when the skies in every major city were heavy with industrial soot that would gather on sidewalks and windowsills like volcanic ash.

There were those who wanted to take us back to a time when women could not vote, or attend any but a few colleges, or get loans in their own names, or start their own businesses.

There were those who wanted to take us back to a time when blacks “had no rights that the white man was bound to respect,” – this being the official opinion of the Supreme Court before those awful days of judicial activism, now decried by the likes of you – and when people of color could legally be kept from voting solely because of race, or holding certain jobs, or living in certain neighborhoods, or run out of other towns altogether when the sun would go down, or be strung up from trees.

[David in Cal]

It's quite possible that a larger stimulus would have helped the Democrats

It's surprising that people believe in the effectiveness of stimulus, when stimulus has failed over and over. Obama's failed by his own standards -- that it it would keep unemployment under 8%. Bush had IIRC two stimuluses, both of which failed. And, FDR's failed. Henry Morganthau, Secretary of the Treasury under Franklin D. Roosevelt and one of FDR's closest advisers, said in 1939:

"We have tried spending money. We are spending more than we have ever spent before and it does not work." ... "after eight years of this Administration we have just as much unemployment as when we started. . . And an enormous debt to boot!"
http://townhall.com/columnists/ThomasSowell/2010/11/02/guess_who