FAIR hypes shaky estimate of Iraqi casualties

The political abuse of statistics continues.

In an action alert sent to its email list today, the liberals at Fairness and Accuracy in Reporting claim that Iraqi civilian casualty estimates of 16,500-20,000 offered by the network newscasts, including NBC's Brian Williams, were too low:

NBC's Williams seemed to be referring to an estimate of Iraqi civilian casualties that none of the networks saw fit to mention: According to a study published in the respected British medical journal The Lancet (10/29/04), about 100,000 Iraqi civilians have died as a result of the war. The majority of deaths were due to violence, primarily as a result of U.S.-led military action. One of the researchers on the project said that the estimate is likely a conservative one (New York Times, 10/29/04). It's certainly a more scientific estimate than the Iraq Body Count figure cited by ABC, which is, as that project's website notes, a "compilation of civilian deaths that have been reported by recognized sources.... It is likely that many if not most civilian casualties will go unreported by the media."

Saying that "about 100,000 Iraqi civilians" have died vastly overstates the precision of the number that was found. As Fred Kaplan explained on Slate, the 95% confidence interval of the Lancet study extended from 8,000-196,000 deaths. That means that the researchers are 95% sure that the true figure lies between those two vastly different numbers. While the Iraqi Body Count estimates may indeed be too low, can you blame the networks from avoiding such a shaky estimate?

My advice: don't trust FAIR (or any of the supposed media watchdogs) without checking the facts yourself.

(For more, see Ben Fritz's 2002 Spinsanity column on the pattern of distortions from FAIR and the Media Research Center.)

Update 3/22: As a result of an Instapundit link, there's a very useful comment thread below that's worth reading and includes a number of helpful links. A few key points: (1) Many analysts believe Fred Kaplan's article is deeply flawed. I was citing it mainly to support my claim that the study has a very wide confidence interval. That is correct. His other critiques of the methodology used in the Lancet study may not be valid, so I've deleted a statement referring to them above. (2) Critics point out that the breadth of the confidence interval does not mean that each point within it is equally likely. That is also correct, and I did not mean to imply otherwise. As I wrote in a comment posted below, I would have no objection to the networks saying that the Lancet study suggests that the total casualty figure is higher than 20,000 (and possibly much higher), but the wide confidence intervals attributable to the cluster sampling survey design make it understandable that they didn't. (3) The study estimates the total number of "excess deaths," not civilian deaths. Thus FAIR's claim is doubly wrong. Finally, here is a link to the Lancet study (PDF).

Comments

[Mark]

Not to start a debate here, but the Slate article wasn't exactely the model of rebuttal. It has been well debunked in turn at http://tinyurl.com/4z8er among other places.

Is the Lancet study right? I've read it, followed most of it, but fall down where it matters most: completely understanding the methodology. The above link brings that up. However, I know enough to know that the Slate article was horrible.

Do the networks avoid it because it's not reliable, or because of the controversy? Cthis then be an example of the media going soft and running the other way lest they get labelled as biased? Stating a lower number is undeniably safer politically.

[DavidS]

It's weird the way the Slate article is being used as if it was definitive. And odd, too, to simply state that using the Lancet study is 'political', as if casually dismissing it isn't. For non-statisticians, this Economist review makes an interesting comparison with the Slate piece.

http://www.economist.com/science/displayStory.cfm?story_id=3352814

[the other mark]

We do know how many Iraqi's have been killed by terrorists. Shouldn't the number be subtracted from the undetermined actual amount used in the Lancet study?
8000

Since the Lancet can speculate on the number killed by Americans, what is their estimate in civilians killed by Terrorists? Do they care? Does the left care?

Where is the video? Even Saddam kept showing the same scene, during shock and awe. Does the lancet study also add in Saddam loyalists killed by US forces? I'd spot the Lancet 30,000 on that one alone.

[roger]

I find it amazing that liberals cling so desperately to the figure of 100,000 deaths. They WANT it to be true. What twisted logic actually desires additional innocent deaths to support its political view?

[Brendan Nyhan]

The Crooked Timber post and Economist article linked above don't refute my main point -- the claim that "about 100,000 Iraqi civilians" have died as a result of the war is not a precise estimate. See this passage from Crooked Timber:

Finally, we come onto two critiques of the study which I would say are valid. The first is the one that I made myself in the original CT post; that the extrapolated number of 98,000 is a poor way to summarise the results of the analysis. I think that the simple fact that we can say with 97.5% confidence that the war has made things worse rather than better is just as powerful and doesn’t commit one to the really quite strong assumptions one would need to make for the extrapolation to be valid.

The second one is one that is attributable to the editors of the Lancet rather than the authors of the study. The Lancet’s editorial comment on the study contained the phrase “100,000 civilian deaths”. The study itself counts excess deaths and does not attempt to classify them as combatants or civilians. The Lancet editors should not have done this, and their denial that they did it to sensationalise the claim ahead of the US elections is unconvincing. This does not, however, affect the science; to claim that it does is the purest imaginable example of argumentum ad hominem.

To be clear, I would have no objection to the networks saying that the Lancet study suggests that the total casualty figure is higher than 20,000 (and possibly much higher), but again, the wide confidence intervals attributable to the cluster sampling survey design make it understandable that they didn't.

[Arrgh]

"about 100,000 Iraqi civilians have died as a result of the war. "

THE STUDY DOES NOT COUNT CIVILIANS! It estimates excess deaths. It does not distinguish between combatant and non combatant.

[papertiger]

Mark when you start your post with an obvious lie it doesn't inspire confidence in the rest of your post.

Of course you want to pick an argument. And your full of it for saying otherwise.

The gist of his tiny link is that a regular person can't possibly understand and therefor refute the lancet guesswork.

Far from proving it's contention, if a person can't even explain the way they arrived at their guesstimate without resorting to the "your stupid and can't understand it even if I explained it to you" argument, that you require years of statistical analysis classes to even begin understanding how Lancetarrived at their numbers, then this is even more reason to call the Lancet article a humbug.

Slate calls into question the baseline. What is the prewar deathrate of Iraq? How do we know this?
What would be the result of the Lancet's study done in a non involved country. Have them try it somewhere else and see if the numbers still make sense.

[getagrip]

It's weird the way the Slate article is being used as if it was definitive.

Some questions touched upon in the SLate article are not refutable. The confidence interval IS extraordinarily wide (CT: "it doesnt include zero." -so what?) Statistical sampling is NOT an appropriate method to estimate deaths. This isnt how crime statistics are calculated. There is a jurisdictional issue that has been completely glossed over. That epidemiologists and statisticians sign off on the procedure is irrelevant.

[Brian Jones]

If this figure were true in October of 2004, then where are the contemporary reports by US soldiers of having to wade through piles of civilian bodies?

[derek]

I'm not an expert on this stuff, but the Chronicle of Higher Education interviewed a number of experts who agreed this was a valuable study. Bradley Woodruff, a medical epidemilogist at the U.S. Centers for Disease Control, called it "the best possible methodology."

And getagrip, statistical sampling IS used for crime statistics in the United States. Check out the Bureau of Justice Statistics' National Crime Victimization Survey. It's one of the two major surveys for crime statistics - the other is the FBI's Uniform Crime Reports, which only includes crimes reported to law enforcement.

[DavidS]

Like the Economist, I'd simply like to know - eventually - how many people were killed by the war:

"The study is not perfect. But then it does not claim to be. The way forward is to duplicate the Lancet study independently, and at a larger scale."

[submandave]

Re Iraq Body Count: I always like to point out that their own published methodology includes all deaths resulting from combat or the "occupation". Under the latter category you can find those beheaded by terrorists and the victims of terrorist bombings in Hilla, Najaf, Mosel, etc. I have always found it disengenuous at best for a group to cite civilian deaths at the hand of terrorists as some sort of argument for withdrawing US troops and leaving the population at the whims of those very same terrorists.

Arrgh makes a very good point ("[The Lancet study] does not distinguish between combatant and non-combatant [deaths]."). Considering the size of Saddam's army and the terrible pounding we gave them, the 100K estimate of total deaths may not be that far off. Seriously speaking, we practically eliminated entire divisions in the drive to Baghdad.

[Greg T.]

Bringing an opinion from someone in the CDC into the debate doesn't help much. Much like the Lancet is currently doing, that organization destroyed a lot of its credibility a few years ago when they published a "study" on the effects of gun violence. Disease control = gun violence. Medical journal = estimating deaths in a war zone.

The Bush administration isn't the only side to politicize science. Pathetic.

[MisterPundit]

The Lancet study is a ridiculous waste of time. A range so vast (8000 - 196,000) is meaningless. Since the 95% CR extends across the entire range, you can't just take the midway point and hold that up as definitive.

One day the actual figures will be known, a fact the authors of the Lancet study (who admitted political intent) can't be thrilled about.

[Cody]

I think the point of the Crooked Timber piece is that the 95% figure does not "extend" across the entire range, rather the midpoint is the most likely and estimates become less likely as you move away from the mean in eiher direction.
But I am personally positive that there have been 0 civilian deaths in Iraq as our forces have demonstrated the upmost respect for human life.
Or maybe 20,000 deaths, the figure that has been used since the Lancet study was published last October (so presumably civilian deaths stopped at that point at its been 20,000 ever since).
Yes, it most certainly is the left clinging to beliefs they desperately want to be true.

[DavidS]

It's worth pointing out that the 100,000 figure is not made up by FAIR - it's used by the Lancet authors in their paper:

"Making conservative assumptions, we think that about 100 000 excess deaths, or more have happened since the 2003 invasion of Iraq. Violence accounted for most of the excess deaths and air strikes from coalition forces
accounted for most violent deaths."

[nikkos]

I too was very skeptical of the Lancet study when it was first published. 100,000 civilians? Impossible, I scoffed, in this era of modern warfare, laser-designation of targets, vetting of targets prior to attack, smart bombs, etc. etc. However, I have since read "Generation Kill," by Evan Wright, and while I feel that the 100,000 number is still high, my eyes have been opened. What Mr. Wright makes clear is that yes, we have the most highly trained and disciplined professional fighting force on the planet, and they are equipped with some of the highest-tech, "smartest" weapons available, and that when used properly these weapons certainly can minimize civilian deaths. But what is not reported in the media, or known by most folks, is that good old fashioned artillery is eing employed in urban centers in Iraq to devastating effect. These are mot smart weapons- there is no internal GPS or laser guidance unit. In fact, artillery has not changed much since Napoleon. It can be a very indiscriminate weapon, especially when employed carelessly. Mr. Wright witnessed several such instances. While his experiences are his alone and may not be representative of all US forces in Iraq, it does suggest that our precision in utilizing our weapons is not as surgical as has been made out by the Pentagon and the media. Whether we have killed 10,000 or 100,000 Iraqi civilians, the point is that this war is being fought in our names and we are all responsible.

[Mark Amerman]

There is I think a bigger problem with the Lancet
paper than the wide confidence interval and that
is that how little thought has apparently been given to
the possibility of bias. The Lancet paper found
1 violent death in their sample population for
about 15 months pre-invasion and 19 violent deaths
for a comparable period during the invasion and
post-invasion. That very small sample is then
extropalated to a large number of violent
deaths (about 59,000) for the 24.5 million population
of Iraq overall.

For the extrapolation to be valid the sample had
to be totally random. Even then, even if we have
total randomness, because of the smallness of the
sample we end up with this very wide range of
plausible violent death estimates. But what
happens if the sampling was not random?

Then of course the study for most practical
purposes ceases to have any meaning at all. One
of the curious things about this debate is how
many participants seem to take it for granted
that there was no bias.

The paper itself fails to address the issue.
It never mentions how vulnerable the conclusions
would be to bias and makes no mention of what
measures where taken to protect against it.

The article by Fred Kaplan, taken by many as
the major point of criticism of the paper, implicity
assumes no bias. Most of the posters on this
forum are implicitly assuming there was no bias.

So let's take a little time to examine how
realistic this widely shared assumption is.

First off look at the sampling procedure itself.
Many people problem assume that households
were randomly interviewed, but as the paper
itself discloses only 32 points where randomly
selected. From each of those points interviewers
fanned out to interview households nearby. There
is a heck of a lot of choice in this experimental
procedure. The fact is that the interviewers
were, to a significant extent, selecting who
they interviewed; that opens up the possibility
of unconscious bias.

The political attitudes of the interviewers
then become a significant question. By the
manner in which the study was put out we pretty
well know the biases were of the authors here in the
US but that doesn't prove the study itself was
invalid. The real question is the attitude of the
interviewers, particularly since they were to
some extent choosing who they interviewed.

I don't know what those attitudes were; certainly
it isn't disclosed in the paper. But from the
political orientation of the authors here in
the U.S. we might make a guess as to their
collaborators in Iraq.

I hope I've made clear how crucial this question
is but let's move on. Let's assume that the
interviewers were totally free of bias, either
conscious or unconscious and chose their subjects
totally randomly.

We have an even bigger problem. Representatives
of some 980 households were interviewed. They
know roughly what the survey is about. Either
because they can guess (How many people in your
household died in the fifteen months prior to
the invasion? How many died after?) or because
they were told directly. Now suppose ten percent
of these households were quite hostile to the
american invasion (probably a conservative
estimate), wouldn't they have some motivation
to invent deaths? Or if the death were real
wouldn't they be much more likely to blame that
death on americans rather than say the
Baathist or Al Quaeda insurgency?

Remember how tiny the numbers are: 1 violent
death to 19 violent deaths. Let's suppose the
real ratio for the whole population was 1 to 5.
That would imply about 14,000 more violent
deaths than normal for the first fifteen
months during and after the invasion.

Note that it would only take 14 out of the
some 980 households interviewed to misremember,
exaggerate, or just plain lie about who died
and why they died to get a widely different
estimate than reality.

Here's something more to think about. Some
may find it implausible that someone would
make up a death for political purposes, but
what about counting as a part of your household,
someone related but who -- without his death --
wouldn't have been thought of as part of the
household?

The authors of the study claimed to have
tried to verify that deaths really occurred,
but here's a question: if it's so simple to
verify random deaths in Iraq why aren't the
authors trying to count such directly rather
than sampling? The second question is how
do they know that a person that died would
normally be considered part of the household
in question. The third question is who
are they verifying the death with? Isn't
Iraq to some degree organized by clan and
tribe? And if a family is biased wouldn't
their neighbors be likely to be biased in
the same way?

Remember there were only 32 random sample
points.

What I hope I've made clear is that if the
interviewers were biased it would have had
far-reaching consequences and probably have
made this study meaningless and further that
this bias could have been unconscious.

More importantly, we know for certain that many
of the interviewed were biased and that this
changed the outcome. The only question is
how much.

[DWPittelli]

Remember also that the Lancet study tossed out the Falluja cluster.

A war opponent would point out that that cluster would have made the 100,000 figure about twice as high. One who favored the war would point out that 200,000 or more deaths is an even less plausible number.

Their fudging of the results show that even the Lancet's editors know that their 33-cluster methodology is unacceptable. Why did The Lancet toss out the Falluja cluster, other than to avoid what they would think was clearly a ridiculous result?

[Arrgggh]

not only doesnt it count civilians, it doesnt even attribute the 100k deaths to the US (except when it includes falluja, which it also supposedly discarded.) the airstrike claim comes from the falluja cluster.

[Robert]

Well, I'm 100% sure the real number is between 0 and 3 billion. Does that mean I get to pick 1.5 billion as my midpoint and go with it? Of course not.

If they said 102,000 give or take 94,000 they'd be acurate (well, 95%+ likely to be accurate). Of course that sounds a hell of a lot like "I dunno" so it isn't popular. but I can easily say 1.5 billion give or take 1.5 billion and be accurate (99%+ I'd say). However presuming a standard distribution within the 3 billion range would be obviously wrong.

Somehow assuming a standard distribution within their 188,000 isn't wrong, and therefore 100,000 is really really likely to be right?

Fine, get a new number (and confidence rating). If they could have said 90K - 110K (85% conf) they WOULD HAVE. C'mon, you think someone putting down 8,000 - 196,000 didn't realize that range was freaking huge? If they could have significantly reduced the range to something meaningful (while still keeping a reasonably large confidence rating) in order to have their numbers be meaningful, they would have done it in a heartbeat.

The fact that the study doesn't have a meaningfully narrow range within any confidence rating given seems to indicate that they simply couldn't give any useful or relevant numbers.

[nikkos]

The REAL question is how many Iraqi civilians are we willing to kill in order to "save" them? All politics aside (no, really!), this is what we should be asking ourselves. While I find the analysis of the statistical methodolgy intriguing, it's not the point. So far we've lost 1500+ American soldiers in this conflict- how many is too many? We've killed between 8000-194,000 Iraqi civilians- how many is too many?

[arboise]

We've killed between 8000-194,000 Iraqi civilians

THEY AREN'T CIVILIANS. read the damn study more carefully. and nothing in the study says "we've" killed the bulk of what the study DOES count.

Another obvious point is that a fair comparison doesnt just compare deaths during wartime to the status quo antebellum, but anticipated future deaths.

[The Apologist]

"So far we've lost 1500+ American soldiers in this conflict- how many is too many? We've killed between 8000-194,000 Iraqi civilians- how many is too many?" - Nikkos

How about less than Hussien, but more than Zarqawi? You're question is in bad faith. "We" are responsible for those "we" actually kill. "They" are responsible for those "they" actually kill. And the motives, methods, and results are all taken into account when assessing the "legitimacy" of the actions. I suspect you'll likely disagree with my contention but I think we stay and keep working at planting democracy in the Middle East for as long as we, the majority of Americans, believe we can be successful.

P.S. Your reference to the Peter Arnett canard "We had to destroy the village to save it" didn't go unnoticed. The truth is Arnett lied about this statement, it never happened.

P.P.S. This is one of the best discussion threads of this study that I've read anywhere. And I've read a lot of them.

[mistercalm]

I agree with the Apologist: I've seen a lot of discussion threads that treat the... uh... dubious methodology and conclusions of the Lancet study as sacred texts not to be defiled by examination. I've also seen the 100,000+ number casually thrown about in a letter to the editor in our local paper, as if the bodies had all been placed out on the ground and meticulously counted. I responded with my own letter to the editor the following day stating the same problems that make the Lancet study unreliable (along with debunking the previous writer's claim that "Bush has killed or injured over 30,000 of our soldiers".) For my trouble, the pinhead who wrote the previous letter called my home and accosted my wife on the phone, angry that anyone would DARE to disagree with her assessment. Oh, well... I feel pretty confident that the Lancet survey is probably correct, in the strictest sense that they are 95% certain the death toll is between 8,000 and 198,000... or whatever, but I doubt the the mean number is most likely correct.

[Kip watson]

Excellent post, excellent comments. This absurd study has been 'debunked' from more angles than a porn starlet. I have nothing more to add to those critiques, however there is one small contribution I wish to make.

Nikkos, modern artillery is every bit as accurate and high tech as other modern military technologies. US artillery pieces, guided by spotters using GPS, can land a shell rapidly in a precise location without the need for a 'spotter round'. For example the Crusader was designed to be able (using computer controlled velocity and trajectory) to land 8 artillery rounds within a few seconds on a target of 2 metre radius.

Bad news for bad guys. Utterly implausible that it could be responsible for mass unintended deaths.

(OK - the Crusader was abandoned, nevertheless this gives an idea of the current state of the art in artillery)

[nikkos]

Actually, the "destroy a village in order to save it" meme comes from Vietnam, not Peter Arnett. You know, "let us win your hearts and minds or we'll burn your damn hut down." You may disagree with what I have to say, but please don't accuse me of purloining that hack Arnett!

OK, let's throw out the Lancet numbers altogether. Let's say that U.S. forces have inadvertently, unintentionally killed Iraqi civilians. Can anyone dispute that? Given that we have killed civilians, albeit unintentionally, should we not be concerned with how many we killed simply because we didn't mean to kill them?

Furthermore, I did write my posts above in good faith. I have no political axe to grind here. I am truly interested in knowing what my fellow Americans think about the war's costs in terms of blood and treasure. Blogs offer a unique opportunity to obtain a wide cross section of opinions.

I know you suspect that I want the body count as high as possible in order to argue for a full withdrawal of American forces. However, I have not said that in any of my previous posts and the fact that you would assume it says a lot about the nature of the discourse surrounding this conflict. Since when is it anti-war to ask how many people have died? Since we are already in Iraq, I strongly believe that we must stay and finish the job. But since our government has never had an honest conversation with the American people about the costs of the war, we must have that conversation ourselves.

In regards to modern artillery, it, like all weapons systems (smart and dumb) are only as accurate as the human beings employing them. I cited "Generation Kill" because if you have a chance to read the book you wil find repeated instances of American officers calling in artillery strikes without the targeting round- just picking up the radio and calling in "fire for effect." The result is a lot of unnecessary destruction and loss of civilian life.

[triticale]

Actually, the "destroy a village in order to save it" meme comes from Vietnam, not Peter Arnett.

I'm embarrassed on your behalf, nikkos. That's a remark on the order of "Would you rather sleep with a pillow or the window open?" The quote, and it is a quote, not a youyou, came from Peter Arnett's reportage when he was in Vietnam, and there is no evidence that it is a quote from anyone other than Arnett.

[nikkos]

OK triticale, I'll concede the point about Arnett if only to bring your attention back to my original question.

My point is not the origin of the phrase but the point it makes. If you are going to disregard my point simply because someone you disagree with at one time made a similar point then I'll move on. However, if you would like to address the meat of my query then I am all ears.

If not, I'll assume that you and the many others that have sought to undermine my question by changing the subject continue to do so simply because you are afraid what you might find when you look under that particular rock.

[Richard Aubrey]

Nikkos: I'll take a shot, since I don't care what ambush is awaiting me.
The US has decided on what it needs to do. It will do so at the lowest possible cost in US lives and after that, in other lives. In fact, we have been combining the two since Viet Nam when there were multiple restrictions on the use of heavy weapons, and the attendant increased risks to the grunts caused casualties we would not otherwise have suffered.
Come to think of it, my father, who was in the Infantry in Europe, said they went light on artillery prep while in Holland because they were fighting in the built-up areas of an ally. That made it tougher on the guys in the rifle companies.
The number, in terms of actually enumerated bodies?
Doesn't matter.
It literally doesn't matter.
After the war, long after, somebody might do some poking around and say it wasn't worth it. They may even be right.
At this point, nobody knows.
The point now is to reduce the potential American dead in the war against Islamism to the smallest possible number. We don't know if the potential dead, without our actions, might be zero. We don't know if it's a quarter billion. We don't have a crystal ball. Since we don't know what catastrophe, if any, we're avoiding, nor what its cost would be, we have nothing to compare the current costs to.
Now, once we take the action, we change the future from that following not having taken action. We may well approach the zero number, giving future revisionists plenty of room to claim it wasn't worth it.
We do the best we can and let the partisans fight it out decades from now.
The key is not the numbers, but the effort to keep the numbers low.

[nikkos]

Mr. Aubrey: I know I'm supposed to ambush you, but rather, I'll thank you for your thoughtful and thought-provoking post!

Long live dialogue!

[ahem]

This absurd study has been 'debunked'

...except that it hasn't. And the 'debunkers' remain full of.... bunk. It's a litmus test for ineptitude w/r/t statistics, partisan hackery, and the belief that if you say something loud enough, it makes it true. And, sadly, Nyhan has passed with flying colours.