Prosecutor's fallacy

The terms "prosecutor's fallacy" and "defense attorney's fallacy" were originated by William C. Thompson and Edward Schumann in 1987.[2][3] The fallacy can arise from multiple testing, such as when evidence is compared against a large database. The size of the database elevates the likelihood of finding a match by pure chance alone; i.e., DNA evidence is soundest when a match is found after a single directed comparison because the existence of matches against a large database where the test sample is of poor quality may be less unlikely by mere chance.

The basic fallacy results from misunderstanding conditional probability and neglecting the prior odds of a defendant being guilty before that evidence was introduced. When a prosecutor has collected some evidence (for instance a DNA match) and has an expert testify that the probability of finding this evidence if the accused were innocent is tiny, the fallacy occurs if it is concluded that the probability of the accused being innocent must be comparably tiny. If the DNA match is used to confirm guilt which is otherwise suspected then it is indeed strong evidence. However, if the DNA evidence is the sole evidence against the accused and the accused was picked out of a large database of DNA profiles, the odds of the match being made at random may be increased, and less damaging to the defendant. The odds in this scenario do not relate to the odds of being guilty, they relate to the odds of being picked at random. While the odds of being picked at random may be low for an individual condition implying guilt, i.e. a positive DNA match, the probability of being picked at random for any condition grows to 1 as more conditions are considered, as is the case in multiple testing. It is often the case that both innocence and guilt (i.e., accidental death and murder) are both highly improbable, though naturally one must be true, so the ratio of the likelihood of the "innocent scenario" to the "guilty scenario" is much more informative than the probability of the "guilty scenario" alone.

Finding a person innocent or guilty can be viewed in mathematical terms as a form of binary classification. If E is the observed evidence, and I stands for "accused is innocent" then consider the conditional probabilities:

• P(E|I) is the probability that the "damning evidence" would be observed even when the accused is innocent (a "false positive").
• P(I|E) is the probability that the accused is innocent, despite the evidence E.

With forensic evidence, P(E|I) is tiny. The prosecutor wrongly concludes that P(I|E) is comparatively tiny. (The Lucia de Berk prosecution is accused of exactly this error,[5] for example.) In fact, P(E|I) and P(I|E) are quite different; using Bayes' theorem:

${\displaystyle P(I|E)=P(E|I)\cdot {\frac {P(I)}{P(E)}}}$

where:

• P(I) is the probability of innocence independent of the test result (i.e. from all other evidence) and
• P(E) is the prior probability that the evidence would be observed (regardless of innocence).

This equation shows that a small ${\displaystyle P(E|I)}$ does not imply a small ${\displaystyle P(I|E)}$ in case of a large ${\displaystyle P(I)}$ and a small ${\displaystyle P(E)}$. That is, if the accused is otherwise likely to be innocent and it is unlikely for anyone (guilty or innocent) to exhibit the observed evidence.

Note that

${\displaystyle P(E)=P(E|I)\cdot P(I)+P(E|\sim I)\cdot [1-P(I)]}$

• P(E|~I) is the probability that the evidence would identify a guilty suspect (not give a false negative). This is usually close to 100%, slightly increasing the inference of innocence over a test with false negatives. That inequality is concisely expressed in terms of odds:

${\displaystyle \operatorname {Odds} (I|E)\geq \operatorname {Odds} (I)\cdot P(E|I)}$

The prosecutor is claiming a negligible chance of innocence, given the evidence, implying Odds(I|E) -> P(I|E), or that:

${\displaystyle P(I|E)\approx P(E|I)\cdot \operatorname {Odds} (I)}$

A prosecutor conflating P(I|E) with P(E|I) makes a technical error whenever Odds(I) >> 1. This may be a harmless error if P(I|E) is still negligible, but it is especially misleading otherwise (mistaking low statistical significance for high confidence).

Though the prosecutor's fallacy typically happens by mistake,[6] in the adversarial system lawyers are usually free to present statistical evidence as best suits their case; retrials are more commonly the result of the prosecutor's fallacy in expert witness testimony or in the judge's summation.[7]

Suppose there is a one-in-a-million chance of a match given that the accused is innocent. The prosecutor says this means there is only a one-in-a-million chance of innocence. But if everyone in a community of 10 million people is tested, one expects 10 matches even if all are innocent. The defense fallacy would be to reason that "10 matches were expected, so the accused is no more likely to be guilty than any of the other matches, thus the evidence suggests a 90% chance that the accused is innocent." and "As such, this evidence is irrelevant." The first part of the reasoning would be correct only in the case where there is no further evidence pointing to the defendant. On the second part, Thompson & Schumann wrote that the evidence should still be highly relevant because it "drastically narrows the group of people who are or could have been suspects, while failing to exclude the defendant" (page 171).[2][8]

Another way of saying this would be to point out that the defense attorney's calculation failed to take into account the prior probability of the defendant's guilt. If, for example, the police came up with a list of 10 suspects, all of whom had access to the crime scene, then it would be very illogical indeed to suggest that a test that offers a one-in-a-million chance of a match would change the defendant's prior probability from 1 in 10 (10 percent) to 1 in a million (0.0001 percent). If nine innocent people were to be tested, the likelihood that the test would incorrectly match one (or more) of those people can be calculated as

${\displaystyle 1-((1-1/1000000)^{9})}$,

or approximately 0.0009%. If, however, the other 9 suspects were tested and did not return a match, then the probability of the defendant's guilt has increased from the prior probability of 10% (1 in 10 suspects) to 99.9991% on the basis of the test. The defendant might argue that "lists of suspects compiled by police fail to include the guilty person in 50% of cases" — if that were true, then the defendant's guilt would have increased from the prior probability of 5% (50% of 10%) to 49.99955% on the basis of the test — in which case "reasonable doubt" could be claimed to exist despite the positive test result.

Possible examples of fallacious defense arguments

Authors have cited defense arguments in the O. J. Simpson murder trial as an example of this fallacy regarding the context in which the accused had been brought to court: crime scene blood matched Simpson's with characteristics shared by 1 in 400 people. The defense argued that a football stadium could be filled with Angelenos matching the sample and that the figure of 1 in 400 was useless.[9][10]

Frequency tree of 100 000 battered American women showing the prosecutor's fallacy in OJ Simpson's murder trial

Also at the O. J. Simpson murder trial, the prosecution presented evidence that Simpson had been violent toward his wife, while the defense argued that there was only one woman murdered for every 2500 women who were subjected to spousal abuse, and that any history of Simpson being violent toward his wife was irrelevant to the trial. However, the reasoning behind the defense's calculation was fallacious. According to author Gerd Gigerenzer, the correct probability requires the context — that Simpson's wife had not only been subjected to domestic violence, but rather subjected to domestic violence (by Simpson) and murdered (by someone) — to be taken into account. Gigerenzer writes "the chances that a batterer actually murdered his partner, given that she has been killed, is about 8 in 9 or approximately 90%".[11] While most cases of spousal abuse do not end in murder, most cases of murder where there is a history of spousal abuse were murdered by their spouse.

Sally Clark, a British woman, was accused in 1998 of having killed her first child at 11 weeks of age and then her second child at 8 weeks of age. The prosecution had expert witness Sir Roy Meadow, a professor and consultant paediatrician,[12] testify that the probability of two children in the same family dying from SIDS is about 1 in 73 million. That was much less frequent than the actual rate measured in historical data – Meadow estimated it from single-SIDS death data, and the assumption that the probability of such deaths should be uncorrelated between infants.[13]

Meadow acknowledged that 1-in-73 million is not an impossibility, but argued that such accidents would happen "once every hundred years" and that, in a country of 15 million 2-child families, it is vastly more likely that the double-deaths are due to Münchausen syndrome by proxy than to such a rare accident. However, there is good reason to suppose that the likelihood of a death from SIDS in a family is significantly greater if a previous child has already died in these circumstances (a genetic predisposition to SIDS is likely to invalidate that assumed statistical independence[14]) making some families more susceptible to SIDS and the error an outcome of the ecological fallacy.[15] The likelihood of two SIDS deaths in the same family cannot be soundly estimated by squaring the likelihood of a single such death in all otherwise similar families.[16]

1-in-73 million greatly underestimated the chance of two successive accidents, but, even if that assessment were accurate, the court seems to have missed the fact that the 1-in-73 million number meant nothing on its own. As an a priori probability, it should have been weighed against the a priori probabilities of the alternatives. Given that two deaths had occurred, one of the following explanations must be true, and all of them are a priori extremely improbable:

1. Two successive deaths in the same family, both by SIDS
2. Double homicide (the prosecution's case)
3. Other possibilities (including one homicide and one case of SIDS)

It's unclear whether an estimate of the probability for the second possibility was ever proposed during the trial, or whether the comparison of the first two probabilities was understood to be the key estimate to make in the statistical analysis assessing the prosecution's case against the case for innocence.

Mrs. Clark was convicted in 1999, resulting in a press release by the Royal Statistical Society which pointed out the mistakes.[17]

In 2002, Ray Hill (Mathematics professor at Salford) attempted to accurately compare the chances of these two possible explanations; he concluded that successive accidents are between 4.5 and 9 times more likely than are successive murders, so that the '"a priori"' odds of Clark's guilt were between 4.5 to 1 and 9 to 1 against.[18]

After it was found that the forensic pathologist who had examined both babies had withheld exculpatory evidence, a higher court later quashed Sally Clark's conviction, on 29 January 2003.[19]

Sally Clark was a practising solicitor before the conviction. After her three-year imprisonment she developed a number of serious psychiatric problems including serious alcohol dependency and died in 2007 from acute alcohol poisoning.[20][21]

• Base rate fallacy – Statistical formal fallacy
• Confusion of the inverse
• Data dredging
• Lucia de Berk
• Ethics in mathematics – An emerging field of applied ethics
• False positive
• False positive paradox
• Howland will forgery trial
• Jurimetrics
• Likelihood function – Function related to statistics and probability theory
• People v. Collins
• Representativeness heuristic
• Simpson's paradox – A phenomenon in probability and statistics, in which a trend appears in groups of data but disappears when these groups are combined

• Discussion of the prosecutor's fallacy
• Forensic mathematics of DNA matching
• A British statistician on the fallacy