Cambridge Solidarity with Iraq: Briefing Note 04/11/04

THE LANCET IRAQ MORTALITY SURVEY: THE UK GOVERNMENT’S RESPONSE IS
INACCURATE AND MISLEADING

On Friday 29th October 2004, the Lancet medical journal published a study
of post-war mortality in Iraq which estimated that at least 100,000 excess
civilian deaths had occurred since the 2003 invasion; that most were caused
by violence; and that most of those violent deaths were caused by coalition
air strikes.

In his daily press briefing that morning, the Prime Minister's Official
Spokesperson (PMOS) dismissed the study since its methodology was, he
claimed, inappropriate:

“Asked if the Prime Minister was concerned about a survey published today
suggesting that 100,000 Iraqi civilians had died as a result of the war in
Iraq, the PMOS said that it was important to treat the figures with caution
because there were a number of concerns and doubts about the methodology
that had been used. Firstly, the survey appeared to be based on an
extrapolation technique rather than a detailed body count. Our worries
centred on the fact that the technique in question appeared to treat Iraq
as if every area was one and the same. In terms of the level of conflict,
that was definitely not the case. Secondly, the survey appeared to assume
that bombing had taken place throughout Iraq. Again, that was not true. It
had been focussed primarily on areas such as Fallujah. Consequently, we did
not believe that extrapolation was an appropriate technique to use.”
( http://www.number-10.gov.uk/output/Page6535.asp)

These criticisms of the Lancet mortality survey were repeated again on 1st
November. They are misleading, and almost entirely unfounded. This briefing
note examines the claims of the Prime Minister’s Official Spokesperson.

1) "Firstly, the survey appeared to be based on an extrapolation technique
rather than a detailed body count."

The fact that the survey uses an extrapolation technique does not
automatically mean that it is less likely to be accurate than a body count.
In fact, it is more likely to be accurate than existing attempts at body
counts of Iraqi civilians. This is because in Iraq, where there are so many
no-go areas, it would be impossible to count every casualty.

Attempts to do body counts through deaths reported in the press,
like Iraq Body Count, are also necessarily underestimates, since press
reports of casualties will be incomplete, not least because the areas where
people are being killed are the same areas into which journalists don't
dare go. As Iraq Body Count states on its website:

"Our maximum therefore refers to reported deaths - which can only be a
sample of true deaths unless one assumes that every civilian death has been
reported. It is likely that many if not most civilian casualties will go
unreported by the media." (  http://iraqbodycount.net/#position )

In these circumstances, population-based research like the Lancet study is
arguably a better source.

A detailed, comprehensive body count on the ground by military and medical
personnel would perhaps be more accurate, but it is a task which the US and
UK forces in Iraq refuse to carry out. For Downing Street to question the
best available study on the grounds of preferring a methodology which they
themselves refuse to operate is surprising, to say the least.

2) "Our worries centred on the fact that the technique in question appeared
to treat Iraq as if every area was one and the same."

The technique does not treat Iraq as if every area was the same. The survey
takes a sample in each of 33 areas throughout Iraq’s 18 governorates,
precisely so that regional variation is taken into account as much as
possible. All regions of the country were sampled.

The PMOS might be referring to the fact that due to the deteriorating
security situation in Iraq, the survey 'paired' 12 of the 18 governorates,
and took samples from only 1 of the pair in order to reduce danger to the
investigators by reducing the necessary travel. In these cases, however,
the process of assigning samples to one of the two governorates was
randomised, and weighted according to the population of the governorates,
thus 'still sampling from all regions of the country' and ensuring that
'the sample remained a random national sample' (Roberts et al p.2)

The PMOS may also be referring to the fact that the sample was not
stratified: each of the cluster samples contributed equally to the national
estimate. In this sense they were considered 'exchangeable' for the
purposes of analysis. Where the characteristics of a population are better
known, some surveys use a 'stratified' sample, whose composition is
weighted according to the known characteristics of the population e.g. they
contain the same proportion of men/women, social classes, proportion of
groups subject to violence etc. etc. In this case, the number of samples
attributed to each governorate was weighted according to the governorate's
population; but beyond that the samples were not stratified according, for
instance, to regional differences in exposure to violence, for the
methodologically sound reason that the distribution of violence in Iraq is
simply not known. As Dr Les Roberts, the lead author of the Lancet report,
stated in response to a query from Medialens (quoted at
www.medialens.org/blog ):

"We would have had a more accurate picture if we conducted a 'stratified'
sample, with some in the high violence areas and some in the low violence
areas. But, that would have involved visiting far more houses and exposing
the interviewers to even more risk. Secondly, we do not know how many
people are in the 'high violence' areas, so this would have involved large
assumptions that would now be criticized."

Importantly, though, this does NOT mean that all areas were treated as the
same, in the larger sense of extrapolating from a single area to the entire
national population, as the PMOS is perhaps trying to suggest. All areas of
Iraq were sampled to try to reflect regional variations as much as was
possible given the limitations of current knowledge and safety.

3) "Secondly, the survey appeared to assume that bombing had taken place
throughout Iraq. Again, that was not true. It had been focussed primarily
on areas such as Fallujah."

This is also untrue - see above. The survey was designed precisely to take
variations throughout Iraq into account. Moreover, since the mortality rate
due to violent deaths in the sample collected in Fallujah was so abnormally
high, the Fallujah sample was excluded from the study’s final estimate.
Thus far from overestimating the deaths caused elsewhere by extrapolating
them from those clustered in Fallujah, as the PMOS suggests, in fact the
figure of 100,000 deaths does not take into account the exceptionally high
mortality probably caused in the area of the most intensive bombing, namely
Fallujah. Had the Fallujah sample been included, the survey's estimate
would have been of an excess of about 298,000 deaths, with 200,000
concentrated in the 3% of Iraq around Fallujah (Roberts et al p.5).

In addition, contrary to the PMOS’s claims about the tight focus of
bombing, the Lancet study in fact found that in areas well away from
Fallujah the most significant cause of the deaths it recorded was still
violence, most attributed to coalition bombing (Roberts et al p.4).

4) "Consequently, we did not believe that extrapolation was an appropriate
technique to use."

Almost all surveys except censuses involve 'extrapolation techniques'
(using a sample rather than the entire population), since it is simply
impossible to survey every member of a country's population. Extrapolation
is a universally recognised method that governments and academics use to
obtain data, and it is remarkable to see a government spokesperson
rejecting it outright.

In addition, the specific sampling technique used in the Iraq mortality
survey, a clustered sample survey, is widely used by government and other
surveys to gather data from geographical clusters of households. See, for
example, the recent report from the Home Office Research, Development and
Statistics Directorate, 'Asylum seekers' experiences of the voucher scheme
in the UK, March 2002', which used cluster samples in order to reduce the
cost of the survey (  http://www.homeoffice.gov.uk/rds/pdfs2/asylumexp.pdf
- see pp.2-3 of this report).

5) Iraqi Ministry of Health Figures

(a) PMOS morning briefing, 1st November: "The Iraqi Department of Health
had issued figures that showed over a 6 month period there were about 3000
deaths, which was a long way short of the figures quoted in the "Lancet".
The Iraqi DOH measured those figures by the number of people who came into
hospitals throughout Iraq, and it was very difficult to rely on any such
figures quoted in the "Lancet" with any certainty."
( http://www.number-10.gov.uk/output/page6515.asp )

(b) Prime Minister, House of Commons, 4th November: "we do not accept the
figures released by The Lancet last week at all. The Iraqi Ministry of
Health has put out figures for the six months up to October, which suggest
just over 3,000 deaths, but that includes people who are either terrorists
or insurgents and those who have been the victims of terrorist attack."
( http://www.publications.parliament.uk/pa/cm200304/cmhansrd/cm041103/debtext/41103-03.htm#column_301 )

Like figures taken from Press Reports like Iraq Body Count, figures
obtained by counting the numbers of bodies arriving at hospitals, will also
necessarily be undercounts: many of those who have died or been killed will
never arrive at hospital in conditions of war, when access to roads and
health facilities are severely disrupted, and when it might seem pointless
to risk the journey for the sake of someone who is already dead.

It is unclear where the MOH figures referred to by the PM were released.
If, however, he is referring to the MOH figures obtained by Knight Ridder
Newspapers of "3,487 Iraqi deaths in 15 of the country's 18 provinces from
April 5 [i.e. not including the 3 Kurdish Governorates] - when the ministry
began compiling the data - until Sept. 19", then he is being misleading.
The MOH officials interviewed by Knight Ridder EXPLICITLY STATED THAT THEIR
FIGURES WERE AN UNDERCOUNT:

"Iraqi health and hospital officials agreed that the statistics captured
only part of the death toll. To compile the data, the Health Ministry calls
the directors general of the 15 provinces and asks how many deaths related
to the war were reported at hospitals. The tracking of such information has
become decentralized since the fall of Saddam Hussein's regime because both
hospitals and morgues issue death certificates now. And families often bury
their dead without telling any government agencies or are treated at
facilities that don't report to the government....The numbers also exclude
those whose bodies were too mutilated to be recovered at car bombings or
other attacks, the ministry said. Ministry officials said they didn't know
how big the undercount was. "We have nothing to do with politics," Jassim
said." ( http://www.realcities.com/mld/krwashington/9753603.htm )

The trends reported in the MOH figures also support some of the Lancet
study's findings: the MOH found that substantial numbers of women and
children were being killed by coalition violence (328 of their 3487
figure); and that more (two-thirds) were being killed by coalition than by
insurgent attacks (although the reliability of this latter distinction is
unclear).

***

Perhaps more serious than these misrepresentations of the mortality study
by the Prime Minister's Office is the UK Government's apparent disregard
for efforts to ascertain the numbers of civilian deaths its actions have
caused. Foreign Office Minister Baroness Symons promised the House of Lords
that:

“we have done our best to count them [civilian casualties] but I am bound
to say to the noble Baroness that the practicalities involved are very
difficult indeed. Certainly during the early part of the engagement, it has
not always been possible to know what civilian casualties there were on the
ground….We do our best to count civilian casualties now."

(Lords Hansard 27 Apr 2004: Column 686)

Yet neither the US nor the UK governments have made public any aggregate
data on civilian deaths in Iraq – a failure which an MOD spokesperson
defended by arguing that

“there is no reliable way of estimating the number of civilian casualties
caused during major combat operations.”

(BBC Online 22 September 2004)

The Lancet mortality survey argues that this is not the case:

“This survey shows that with modest funds, 4 weeks, and seven Iraqi team
members willing to risk their lives, a useful measure of civilian deaths
could be obtained. There seems to be little excuse for occupying forces to
not be able to provide more precise tallies.”

(Roberts et al p.7)

In the last few days the Government have attempted to disparage the Lancet
report on a different tack, by going back on its previous rejections of
casualty estimates and instead positively citing the much lower estimate of
the Iraq Body Count project, as Foreign Secretary Jack Straw did when
interviewed on the Radio 4 ‘Today’ programme on 29th October. Mr Straw
ignores the fact that Iraq Body Count itself makes clear that its figures
cannot be considered a comprehensive estimate of civilian casualties, since
‘it is likely that many if not most civilian casualties will go unreported
by the media.’ (see above). The Government’s new attention to the figures
provided by this media-report-based casualty estimate is also surprising
given the previous statement by an MOD spokesperson about Iraq Body Count
that

“ We would caution against taking the numbers quoted in media reports and
else where at face value. No source or combination of sources can produce a
reliable figure."

(BBC Online 22 September 2004)

and given Defence Minister Adam Ingram’s dismissive answer to a
parliamentary question in June 2003 asking “what information he has (a)
given to and (b) received from the organisation called Iraq Body Count”, Mr
Ingram replying simply that “Defence Ministers have had no contact with an
organisation called Iraq Body Count.” (Commons Written Answers, 3 Jun 2003
Col. 37W, question 114211)

The Government must accept that despite shortcomings due to the
difficulties faced by a small team of volunteer doctors collecting data in
a war zone, the Lancet report is the most comprehensive study of Iraqi
civilian deaths since 2003 currently available. If it is to rebuff its
findings, the Government must honour its own obligations by undertaking a
large-scale survey of civilian casualties. As the Lancet report states:

“ US General Tommy Franks is widely quoted as saying “we don’t do body
counts”. The Geneva Conventions have clear guidance about the
responsibilities of occupying armies to the civilian population they
control. The fact that more than half the deaths reportedly caused by the
occupying forces were women and children is cause for concern. In
particular, Convention IV, Article 27 states that protected persons “. . .
shall be at all times humanely treated, and shall be protected especially
against acts of violence . . .”. It seems difficult to understand how a
military force could monitor the extent to which civilians are protected
against violence without systematically doing body counts or at least
looking at the kinds of casualties they induce. In view of the political
importance of this conflict, these results should be confirmed by an
independent body such as the ICRC, Epicentre, or WHO.”

(Roberts et al p.7)


FURTHER INFORMATION:

1) L. Roberts, G. Burnham, R. Lafta, J. Khudhairi, R.
Garfield, ‘Mortality before and after the 2003 invasion of Iraq: cluster
sample survey’,  http://image.thelancet.com/extras/04art10342web.pdf


2) Iraq Body Count, the most comprehensive indirect casualty count:
 http://iraqbodycount.net/ This site includes a summary of another casualty count project, and an explanation of IBC methodology.

Author: Michael Lewis, Christ's College, Cambridge CB2 3BU, United Kingdom
 mhl24@cam.ac.uk