Statistics are often used to demonstrate the the value of any given conclusion.  I am right and you are wrong because this poll or that study says so.  The case is no different with DNA testing.  When a law enforcement agency finds DNA at the scene of the crime and compares it to the DNA of a suspect, if the two profiles are consistent, the lab will provide an “inclusion” or “rarity”  statistic to allow judges and juries to weigh the probative value of that “match.”

When a scientist gets up on the stand at trial and he states that the chances of the semen on the rape kit coming from someone other than the defendant is 1 in 35 quadrillion, most defendants should just start picking out their window treatments for their 6X9.

But it  begs a few questions that no one is really asking.  Are these statistics accurate ?  Are juries getting the right information to determine the accuracy of these numbers?  A recent article answers both these questions in the negative.  The Washington Monthly describes the problem in the context of a California cold hit case where the DNA match came from a very partial profile of a “mixed” sample taken from the victim:

Generally, juries in cold-hit cases are told to rely on FBI estimates when weighing the odds of a coincidental match between crime-scene DNA and the accused. When all thirteen markers are intact, these odds can be as slim as one in many trillions. In Puckett’s case, where there were only five and a half markers available, the San Francisco crime lab put the figure at one in 1.1 million—still remote enough to erase any reasonable doubt of his guilt. The problem is that, according to most scientists, this statistic is only relevant when DNA material is used to link a crime directly to a suspect identified through eyewitness testimony or other evidence. In cases where a suspect is found by searching through large databases, the chances of accidentally hitting on the wrong person are orders of magnitude higher.

The reasons for this aren’t difficult to grasp: consider what happens when you take a DNA profile that has a rarity of one in a million and run it through a database that contains a million people; chances are you’ll get a coincidental match.

This is a bit alarming considering that these inclusion statistics are considered infallible evidence that is consistently relied on by juries to put individuals in prison for life or even put them to death.  A recent study in Australia demonstrated looked at how individuals view DNA evidence:

Professor Goodman-Delahunty carried out a study of about 400 people to find how they responded to DNA evidence in a series of mock trials for a murder case.

She found that people with a low understanding of DNA convicted at a rate of 75 per cent, while those with better knowledge recorded a 42 per cent conviction rate.

In the study, jurors tended to regard DNA evidence as “infallible and so once it comes in, the very fact that the judge has admitted the evidence is often construed as an indication that it must be reliable, and therefore they rely on it without questioning it very thoroughly. “  This is a disconcerting trait considering the nature of these inclusion statistics.

Some preeminent, government authorized entities have suggested a way to properly calculate the inclusion statistic in database cold hit cases:

Given this fact, the two leading scientific bodies that have studied the issue—the National Research Council and the FBI’s DNA advisory board—have recommended that law enforcement and prosecutors calculate the probability of a coincidental match differently in cold-hit cases. In particular, they recommend multiplying the FBI’s rarity statistic by the number of profiles in the database, to arrive at a figure known as the Database Match Probability.

When one applies this formula to Puckett’s case (where the partial mixed profile with a rarity of one in 1.1 million was run through a database of 338,000 offenders) the chances of a coincidental match climb to one in three. Yes, you read that correctly.  The chances of having a hit to an innocent person is 33%.  And this counts as conclusive evidence.

But this is not some abstract notion.  We actually see this embellishment of inclusion statistics in studies and actual cases.  The Washington Monthly continues:

In 2004, detectives investigating a string of robberies on the city’s North Side found some skin cells that the culprit had left behind at one crime scene, which contained six DNA markers. When they ran this profile against Illinois’s offender database, they found it matched a woman named Diane Myers. There was just one problem: when the burglaries in question were committed, Myers was already in jail, serving time on drug charges.

Indeed, the little information that has come to light about the actual rate of coincidental matches in offender databases suggests the chances of hitting on the wrong person may be even higher than the Database Match Probability suggests. In 2005, Barlow heard that an Arizona state employee named Kathryn Troyer had run a series of tests on the state’s DNA database, which at the time included 65,000 profiles, and found multiple people with nine or more identical markers. If you believe the FBI’s rarity statistics, this was all but impossible—the chances of any two people in the general population sharing that many markers was supposed to be about one in 750 million, while the Database Match Probability for a nine-marker match in a system the size of Arizona’s is roughly one in 11,000.

Barlow decided to subpoena Troyer’s searches. To her surprise, she discovered that Troyer had unearthed not just a couple of pairs who shared nine identical markers, but 122. “That was a ‘wow’ moment,” Barlow recalls.

Studies of DNA databases elsewhere have revealed similar findings. In 2006, for instance, Illinois officials searched the state’s offender database, which at the time contained 233,000 profiles. They found 903 pairs with nine or more matching DNA markers. Among geneticists and statisticians, these findings have eroded faith in the FBI’s DNA rarity statistics, which were based on data from just 200 or 300 people and are used by crime labs across the country. Laurence Mueller, an ecology and evolutionary biology professor at University of California, Irvine, told me that anyone who knows statistics finds the figures “laughable.”

Yet, juries are not hearing this information.  In fact, the FBI is refusing to turn over important statistical information and, in Puckett’s case, the prosecution vigorously fought to keep the more accurate method of calculating inclusions statistics away from from the jury.  Nor do juries hear about the dozens of people in state DNA databases who, at 9 or more markers, possess the same profile.  In the end, the Judge in Puckett’s case refused to allow anyone to mention that Puckett was identified through a cold hit or that there were other, better ways to calculate the rarity of a DNA profile.  Puckett was convicted and remains in prison.

So what can be done.  The Australian researchers probably have it right:

Professor Goodman-Delahunty said participants were then shown a 20-minute tutorial about DNA evidence to find out what impact that would have.

“What we found is that was an effective way to increase people’s basic understanding of the science,” she said.

“As their knowledge increased – and we tested that with objective multiple choice questions before and after exposure to the tutorial – their trust and reliance on the DNA evidence decreased. And the conviction rate also decreased.

“I think it is a confirmation of concerns that have been expressed for a long time by courts, law reform commissions and lawyers about the impact of DNA evidence, possibly exacerbated by the popularity of shows like CSI and so forth, where this sort of evidence is rarely presented as including any potential for error.”

Knowledge is power and more liberal evidentiary considerations are necessary to arm the jury with enough information, however technical, to properly weigh this evidence that is often the linchpin of criminal prosecutions.