Silver's flawed analysis of health care $

As I wrote back in May, Nate Silver is obviously a smart and energetic blogger, but he just isn't a social scientist. That's why it's frustrating when his quickie statistical analyses draw more attention than the relevant political science scholarship (of which Silver frequently seems unaware).

For instance, Silver published an analysis today that claims to measure "the impact of lobbying by the insurance industry on the prospects for health care reform." His statistical model predicts senators' support for the so-called "public option" using their estimated DW-NOMINATE ideal points (a measure of their policy views), health costs in their state, and their total campaign contributions from health insurance and HMO PACs. He finds that senators who receive larger industry contributions are less likely to support the so-called public option. Silver extrapolates from his model to a world in which special interest contributions are banned and concludes that "the insurance industry's influence appears to swing about nine votes against the public option."

However, as GW's John Sides notes on The Monkey Cage, "there is a potential endogeneity problem in the model that Silver constructs. Does health insurance PAC money make senators less likely to support the public option, or are Senators who tend to oppose or at least be skeptical about government health care programs more likely to get contributions from health insurers and HMOs?" In other words, we can't be confident that contributions are causing senators to oppose the public option. Silver acknowledges this caveat in a parenthetical, but then concludes his article by making the causal claim that "the money is why... Democrats are facing an uphill battle on the issue."

In addition, we can't say much of anything about a world in which health industry contributions aren't made. It's an unobserved counterfactual. Silver's estimate of nine votes being swung is essentially meaningless in its current form -- the model is extrapolating far beyond the available data.

What's so frustrating about Silver's post, which was (disappointingly) praised by Paul Krugman on his New York Times blog, is that there is an extensive literature on this subject by political scientists and economists. Over the last 30+ years, these studies* have typically found minimal effects of campaign contributions on roll call votes in Congress even when scholars use more sophisticated techniques to address the causal inference problems described above (which are widely understood at this point). Given these prior findings and the inherent problems with Silver's analytical approach, we should be extremely cautious about putting any weight on his conclusions.

(* See, for instance, pp. 112-117 of Ansolabehere, de Figueiredo, and Snyder 2003 [PDF] from the Journal of Economic Perspectives.)

Comments

[Jay Livingston]

1. Maybe someone should tell the insurance biz that for all their spending, they are getting only "minimal effects." They seem to share Silver's misperception about the impact of their contributions.

2. Is it possible that a large part of impact of spending falls in places other than roll-call votes? If the insurance lobby can prevent the public option from ever getting out of committee, they'll have won a big victory, but that effect will not turn up in data on roll-call votes.

[beezer]

I'm a little confused about a basic point. Does all the money spent on campaign contributions and K Street lobbying influence votes? Or not.

If the spending has "minimal effects" on roll call votes, what the heck are these people doing blowing all this money away? Are they delusional?

If the politician is four square in favor of a particular industry position, are you implying that creates the contributions? Or is someone "on the fence" going to reap more cash instead?

I'm willing to believe a politician's long held personal beliefs will be the major determinant in voting patterns. I'm also willing to believe that a politician's long term desire to stay in office (anyone in Congress is a long term politician)is also a strong determinant.

[Hans Noel]

It's not necessary for donors to be mistaken about their impact for their donations to be meaningful. They aren't blowing it away, they are just not necessarily "buying votes."

Consider a politician who is "four square in favor" of some position. That politician will face re-election challenges, and will need campaign money. Those who know what he stands for stand to gain if he is re-elected, and they will give him resources. But that is the other causal mechanism -- policy positions cause donations.

It is also likely that donations have other ways of mattering, as in Livingston's 2nd point. Recent work by Rick Hall and co-authors suggests that campaign contributions send a signal to an MC that the donor agrees with the MC, and so their information will be valuable. Since MC's rely on lobbyists for information, this signal is important.

There's a lot of work on this, as Brendan points out.

[rone]

Instead of ruing the flaws in Silver's work, why not try to work with him?

[dave]

even if ns does not have enough diclaimers to suit you, if even two or three votes of his nine hypotheticals are unable to be explained away by your counfouder (endogenousness of pac $), then ns is stll correct in his statement that 'money is the reason' the battle is uphill. get over yourself.