Measuring "epistemic closure"

While I obviously support more polling on misinformation, I don't agree with Ezra Klein's suggestion that it's the best way to determine whether there is "epistemic closure" among ideological or partisan groups:

The question is how do you measure epistemic closure?

The easy answer is you test for its product: Misinformation. What you'd want to do, I guess, is continuously poll a standard set of questions based on empirical facts. "Has GDP grown since President X's inauguration?" "Have global temperatures been rising or falling in recent decades?" "Does the United States have longer life expectancy than other developed nations?" "Do a majority of Americans approve of the president's job performance?" That sort of thing. Have representatives of both parties decide the questions and then see whether respondents from one party or the other get more questions right.

The problem is that misperceptions are not necessarily the result of a closed information loop. Someone with a relatively balanced media diet can still end up with false beliefs -- it all depends on how they interpret the news that they receive (i.e., the extent to which they are willing to accept information that is inconsistent with their preferences).

A better approach would be to measure (a) to what extent ideological elites on the left and right are failing to engage with outside sources of information and (b) to what extent their adherents are consuming political news largely or entirely from like-minded elites.

For an example of (a), see Hargittai, Gallo, and Kane on the insularity of linking patterns among liberal and conservative blogs in 2004 and 2005, a paper which includes this table comparing linkage patterns (more negative values indicate more insularity):

For an example of (b), see this new Gentzkow and Shapiro paper on ideological segregation online, which includes a table comparing the audiences of the online sources visited by liberals and conservatives:

Update 4/27 9:04 AM: See also this Baum and Groeling paper which finds that "Daily Kos on the left and Free Republic and Fox News on the right demonstrate clear and strong preferences for news stories that benefit the party most closely associated with their own ideological orientations":

Update 4/27 10:57 AM: I neglected to mention another relevant paper by Lawrence, Sides, and Farrell showing the ideological skew of readers of liberal and conservative blogs:

Henry Farrell, one of the authors of that piece, objects to part of my argument above at The Monkey Cage:

Julian Sanchez’s argument - which started this debate - seemed to me to be making a somewhat different point. While he focuses on the impact of an alternative sphere of media, his concern is with the consequences...

These consequences could plausibly manifest themselves either if conservatives (or liberals; or whoever) only consume conservative media or if they consume both conservative and non-conservative media, but tend to weight the arguments of the former much more heavily than the latter. And we simply cannot figure this out from data on media consumption patterns (or, for that matter, linkage patterns) alone...

Data on divergent patterns of media and information consumption is valuable in figuring out what people think. But people interested in this question aren’t so much worried about the actual patterns of consumption as about its putative consequences for political beliefs. So I think that first cut research to identify whether epistemic closure is a problem should focus on consequences, contra Brendan, looking at the extent to which individuals with different ideologies tend towards closure across a variety of politically salient issues. But it would be nice to see a second wave of research, extending the stuff that Brendan talks about to look at how variation in patterns of media consumption intersected with false political beliefs. And a third body of research could do some experimental work to figure out more precisely the underlying causal mechanisms...

I agree that the studies above don't quite get at the core of Sanchez's claim regarding the unwillingness of elites on the right to acknowledge outside sources of information as valid. Focusing on the putative consequences of closure, however, still strikes me as far too indirect. Instead, why not try to measure Sanchez's claim directly? For instance, one could code the sources referenced in the National Review and Weekly Standard and compare them to those referenced in The Nation and American Prospect. If Sanchez is correct, the sources cited by conservatives in a non-disparaging way are more likely to be fellow travelers in the movement than those cited by liberals.

(Note: See the papers linked above for more details on their data and the definitions of the measures in the tables.)

Comments

[Rob]

Unless I misunderstand them, Gentzkow and Shapiro classify media outlets as very liberal, somewhat liberal, etc., according to the percentage of viewership by those who self-identify as very liberal, very conservative, etc. So the greater the percentage of self-identified conservatives who read the New York Times, the less liberal the Times is. And the more that self-identified liberals avoid Fox News, the more conservative that media outlet's classification. This methodology may have the advantage of being an objective measure, but it seems like a measure so flawed that it makes the results largely meaningless. In other words, the percentage of liberal and conservative viewers is a very poor proxy for the ideological orientation of a media outlet. In addition, since the methodology of the study depends on individuals' self-identification, if liberals are more prone than conservatives to describe themselves as middle-of-the-road (or vice versa), that also makes the results unreliable. But Gentzkow and Shapiro did report their findings to the third decimal place, so forget what I said--the results must be very finely calibrated indeed.

It's probably a good thing I'm not in academia. I'd be the skunk at every garden party.

[David]

You'd never know it from the NY Times article, but conservatives think epistemic closure is greater on the left than on the right. Conservatives address their own EC, because they expect theif side to be more realistic and open minded.

A problem with using false beliefs as the measure is that some of these beliefs are somewhat subjective. We've addressed this point before on this blog.

E.g., consider "Does the United States have longer life expectancy than other developed nations?"
First of all, the question doesn't specifiy the age at which the life expectancy is to be compared.

Second, if it's life expectancy at birth, an adjustment needs to be made for treatment of premature babies. The US makes heroic efforts to keep premies alive, so our live births include premies who would have died in other countries. Some of these premies die at young ages, thus making the US life expectancy at birth appear worse in comparison with other countries than it really is.

The question of what the earth's temperature has done in recent decades depends on the decade. Temperature went down a bit in the decade from 2000 - 2010. It went up considerably in the 3 decades from 1970 - 2000. It went down (I believe) in the decade from 1960 -70.

So, if the question was simply "Have global temperatures been rising or falling in recent decades?", the correct answer might be "both."

[David]

One more ambiguity about the temperature change in recent decades. Suppose we decided to look at the 3 most recent decades. What years to they consist of (a) or (b)?

a. 2000 - 2010, 1990 - 2000, 1980 - 1990
b. 2000 - 2010, 1999 - 2009, 1998 - 2008

Both definitions are plausible. Under (a), the global temperature has risen in 2 of 3 most recent decades. Under (b) the global temperature has fallen in each of the 3 most recent decades.

[Rob]

In addition to the result quoted by Brendan, Baum and Groeling also found "some evidence that the self-consciously nonpartisan [ha!] Associated Press prefers stories critical of Republicans, which may constitute evidence supporting the oft-cited conservative claim of liberal bias in the mainstream news media. Of course, it could also reflect the exceptionally anti-Republican mood in the nation in the run-up to the 2006 midterm election, a period in which the news was dominated by stories about domestic political scandals enveloping the Republican party and the perceived failure of the administration’s policies in Iraq. Nonetheless, AP’s anti-Republican skew persisted even when these alternative explanations were explicitly controlled in our models." And btw, "some evidence" seems like a weak description of the actual results.

[MarkGisleson]

Nicely argued but I think there is a huge hole in all of your data. Daily Kos may be as biased as Hot Air, but has anyone counted the links out to reputable news sources?

I've been blogging since 1999 and I think I have a pretty good handle on right and left wing blogs. The right truly is a sealed system. Like the old campus Marxists, they only quote themselves. When they link to legit news sources, it's to criticize more often than not.

Your average lefty blog links to all kinds of news sources. For a while here in the Twin Cities a moderate blogger held Friday evening get togethers in his garage for left and right wing bloggers. It never took off because there was, incredibly, no common ground. The bloggers from the right knew their talking points, but not the issues. The bloggers from the left knew the news, but weren't always on top of what their party was doing in the lege or Congress.

This information orientation is painfully obvious if you look at aggregator blogs. The left has tons of them, the right, only a few and those few link relentlessly to the Wurlitzer.

I'd love to agree with you but the right and the left are not the same, even if they have switched places with each other since the '60s. (Today's Republican party is run by Southern Democrats, and today's Democrats sound more like liberal Republicans from the '60s than they do Ted Kennedy or LBJ.)

[David]

Mark, I'm unclear what you mean about Hot Air not linking to "reputable news sources". I went to the site and found 10 links in a box on top. The first was to the Weekly Standard, but that article immediately linked to the Atlantic. Among the other links was The Week, New York Times, Los Angeles Times, NPR, and ABC.

See http://hotair.com/

[howard.craft]

Too bad hot hotair.com is closed to new registrations. It certainly gives the impression that they are not open to dissenting views.

*****

It may be that the true distinction of "epistemic closure" isn't the presence of selective facts (or the presence of lies or distortions) but the overwhelming presence of purely subjective assessments. That would include the projection of motives and ascribed values toward those who offer opposing views.

It would be interesting to see to what extend it might be possible to quantify the prevalence and the weight of these factors in any given commentary. Or, to put it in another more qualitative way : if these attributes were removed what would be left of the main "argument"?

I suggest that anyone reading commentary (or any sort) should try to apply that type of a critical assessment to the commentary.

[Fred A Milton]

Since the shows' headliners in the following question are ideologically opposed, comparing them might be enlightening:

Are dissenting views (meaning views that dissent from the show's headliner's views) presented more frequently on Keith Olbermenn's show or on Bill O'Reilly's show?