Saturday, August 22, 2015

Disentangling Two Issues in the Hair Evidence Debacle

Forensic-science practitioners commonly present findings regarding traces left at crime-scenes, on victims or suspects, or on or in their possessions. Such trace evidence can take many forms. Physical traces such as fingerprints, striations on bullets, shoe and tire prints, and handwritten documents are common examples. Biological materials, such as blood, semen, saliva, and hairs also are fodder for the crime laboratory. Comparisons of a questioned and known sample can supply valuable information on whether a specific suspect is associated in some manner with a crime. Viewers of the acronymious police procedurals—NCIS, CSI, and Law and Order SVU—know all this.

For decades, however, legal and other academics have questioned the hoary courtroom claims of absolutely certain identification of one and only one possible source of trace evidence. In the turbulent wake of the 2009 report of a National Research Council committee, these views have slowly gained traction in the forensic-science community. Indeed in the popular press and among investigative reporters, the pendulum may be swinging in favor of uncritical rejection of once unquestioned forensic sciences. Recent months have seen an episode from Frontline presenting DNA evidence as "anything but proven"; 1/ they have included unfounded reports that as many as 15 percent of men and women imprisoned with the help of DNA evidence at trial are wrongfully convicted; 2/ and award-winning journalists have spread the word that the FBI "faked an entire field of forensic science," 3/ placed "pseudoscience in the witness box," 4/ and palmed off "virtually worthless" evidence as scientific truth. 5/

The last set of reports stem from an ongoing review of well over 20,000 cases in which the FBI laboratory issued reports on hair associations. The review spans decades of hair comparisons, and it is showing so many questionable statements that the expert evidence on hair associations stands out as "one of the country's largest forensic scandals." 6/ Its preliminary findings provoked prominent Senators to speak of an "appalling and chilling ... indictment of our criminal justice system" 7/ and to call for a "root cause analysis" of ubiquitous errors. 8/ A distressed Department of Justice and FBI joined with the Innocence Project and the National Association of Defense Lawyers not only to publicize these failings, but also to call on states "to conduct their own independent reviews where ... examiners were trained by the FBI." 9/ Projecting the outcome of cases that have yet to be reviewed, postconviction petitions refer ominously to "[t]housands of . . . cases the Justice Department now recognizes were infected by false expert hair analysis" 10/ and "pseudoscientific nonsense." 11/

The hair scandal illustrates two related problems with many types of forensic-science testimony. The first is the problem of foundation—What reasons are there to believe that hair or other analysts possess sufficient expertise to produce relevant evidence of associations between known and unknown samples? For years, commentators and some defense counsel have posed legitimate questions (with little impact in the courts) about the reliability and validity of physical comparisons by examiners asked to judge whether known and unknown samples are similar in enough respects—and not too dissimilar in other respects—to support a claim that they could have originated from the same individual. To paraphrase Gertrude Stein, is there enough there there to warrant any form of testimony about a positive association? This is the existential question of whether, in the words in of the Court in Daubert v. Merrell Dow Pharmaceuticals, "[t]he subject of an expert's testimony [is] 'scientific . . . knowledge.'" 12/ Or, at the other extreme, is the entire enterprise ersatz—a "fake science" and a "worthless" endeavor?

As I see it, the harsh view that physical hair comparisons are pure pseudoscience, like astrology, graphology, homeopathy, or metoposcopy, is not supportable. The FBI's review project itself rests on the premise that hair evidence has some value. If the comparisons were worthless—like consulting the configurations of the stars or reading Tarot cards—there would be no need to review individual cases. In all cases of an association, the FBI would have exceeded the limits of science. But this only shows that the FBI thinks that there is a basis for some testimony of a positive association. What evidence supports this belief?

One reason to think that the FBI's hair analysts generally possess some expertise in making associations comes from an intriguing study (usually cited as proof of the failings of microscopic hair comparisons) done more than ten years ago. 13/ FBI researchers took human hairs submitted to the FBI laboratory for analysis between 1996 and 2000 and mitotyped them (a form of DNA testing) whenever possible. The probability of an FBI examiner finding of a positive association when comparing hairs from the same individual (as shown by mitotyping) exceeded the probability of this finding when comparing hairs from different individuals by a factor of 2.9 (with a 95% confidence interval of 1.7 to 4.9). A test with this performance level only supplies evidence that is, on average, weakly diagnostic of an association. Still, it is not simply invalid. 

The second problem lies in the presentation of perceived associations. Even if there is a there there, are forensic-science practitioners staying within the boundaries of their demonstrated expertise? Or are they purporting to know more than they do? The FBI's revelations about hair evidence are confined to this issue of overclaiming. What the FBI has uncovered are expert assertions in one case after another that are said to outstrip the core of demonstrated knowledge. Such overclaiming is one form of scientifically invalid testimony, 14/ but it is not the equivalent of an entire invalid science.

In considering the pervasiveness of the problem of overclaiming, the FBI's figure of 90+ percent is startling. It is so startling that one should ask whether it accurately estimates the prevalence of scientifically indefensible testimony. There are reasons to suspect that it might not. After all, the Hair Comparison Review Project was not designed to estimate the proportion of cases in which FBI examiners gave testimony that was, on balance, scientifically invalid. It is intended to spot isolated statements that claimed more than an association that, for all we know, could be quite common in the general population. The general criteria for judging whether particular testimony falls into this category have been publicized, but the protocol and specific criteria that the FBI reviewers are using have not been revealed. No representative sample of the reports or transcripts that are judged to be problematic is available, but a few transcripts in the cases in which the FBI has confessed scientific error indicate that at least some classifications are open to serious question.

It may be instructive to contrast the response to two very similar statements noted in a posting of May 23, 2015, about the court-martial of Jeffrey MacDonald: (1) "this hair ... microscopically matched the head hairs of Colette MacDonald"; and (2) "[a] forcibly removed Caucasian head hair ... exhibits the same microscopic characteristics as hairs in the K2 specimen. Accordingly, this hair is consistent with having originated from Kimberly MacDonald, the identified source of the K2 specimen." The first statement passed muster. The second did not.

If the review in this case is not aberrant, examiners can say that two hairs share the same features—they "match" and are consistent with one another—but they must not add the obvious (and scientifically undeniable) fact that this observation (if correct) means that they could have had the same origin or that they are "consistent with" this possibility.

Of course, one can criticize phrases like "consistent with" and "match" as creating an unacceptable risk that (in the absence of clarification on direct examination, cross-examination, or by judicial instruction) jurors will think the words connote a source attribution. But arguments of this sort stray from determinations that an examiner has made statements that "exceed the limits of science" (the phrase the Justice Department uses in confessing overclaiming). They represent judgments that an examiner has made statements that are scientifically acceptable but prone to being misunderstood.

To be sure, this latter danger is important to the law. It should inform rulings of admissibility under Rules of Evidence 403 and 702. It is a reason to regulate the manner in which experts testify to scientifically acceptable findings, as some courts have done. Laboratories themselves should adopt and enforce policies to ensure that reports and testimony avoid terminology that is known to convey the wrong impression. But it is misleading to include scientifically acceptable but psychologically dangerous phrasing in the counts of scientifically erroneous statements. Case-review projects ought to flag all instances in which examiners have not presented their findings as they should have, but reports ought to differentiate between statements that directly "exceed the limits of science" and those that risk being misconstrued in a way that would make them "exceed the limits of science." One size does not fit all.


This posting is an abridged and modified version of a forthcoming essay about the FBI's Microscopic Hair Review Comparison Review. A full draft of the preliminary version that is being edited for publication is available. Comments and corrections are welcome, especially before publication while there is time to improve the essay.

  1. Some inaccuracies in the documentary are noted in a June 24, 2015, posting on this blog.
  2. See the posting of July 3, 2015 on the blog (debunking this initial assertion of the Rand Corporation).
  3. Dahlia Lithwick, Pseudoscience in the Witness Box: The FBI Faked an Entire Field of Forensic Science, Slate (Apr. 22, 2015 5:09 PM).
  4. Id. These "shameful, horrifying errors" comprised "a story so horrifying . . . that it would stop your breath." Id.
  5. Erin Blakemore, FBI Admits Pseudoscientific Hair Analysis Used in Hundreds of Cases: Nearly 3,000 Cases Included Testimony About Hair Matches, a Technique that Has Been Debunked, Smartnews (Apr. 22, 2015) (quoting Ed Pilkington, Thirty Years In Jail For A Single Hair: The FBI's 'Mass Disaster' of False Conviction, Guardian, Apr. 21, 2015.
  6. Spencer S. Hsu, FBI Admits Flaws in Hair Analysis over Decades, Wash. Post, Apr. 18, 2015.
  7. Id. (quoting "Sen. Richard Blumenthal (D-Conn.), a former prosecutor").
  8. Spencer S. Hsu, FBI Overstated Forensic Hair Matches in Nearly All Trials Before 2000, Wash. Post, Apr. 19, 2015 (quoting "Senate Judiciary Committee Chairman Charles E. Grassley (R-Iowa) and the panel's ranking Democrat, Patrick J. Leahy (Vt.)").
  9. FBI, FBI Testimony on Microscopic Hair Analysis Contained Errors in at Least 90 Percent of Cases in Ongoing Review (Apr. 20, 2015).
  10. Petition for a Writ of Certiorari, at 2-3, Ferguson v. Steele, 134 S.Ct. 1581 (2014) (No. 13-1069).
  11. Id. at 19. Justice Breyer saw the “errors” referred to in the press release as emblematic of “flawed forensic testimony” generally and a reason to hold the death penalty unconstitutional. Glossip v. Gross, No. 14-7955 (June 29, 2015) (dissenting opinion).
  12. 509 U.S. 579, 590 (1993).
  13. Max M. Houck & Bruce Budowle, Correlation of Microscopic and Mitochondrial DNA Hair Comparisons, 47 J. Forensic Sci. 1 (2002).
  14. For this vocabulary, see Brandon Garrett & Peter Neufeld, Invalid Forensic Science Testimony and Wrongful Convictions, 95 Va. L. Rev. 1 (2009); cf. Eric S. Lander, Fix the Flaws in Forensic Science, N.Y. Times, Apr. 21, 2015 (“The F.B.I. stunned the legal community on Monday with its acknowledgment that testimony by its forensic scientists about hair identification was scientifically indefensible in nearly every one of more than 250 cases reviewed.”).

Monday, August 17, 2015

First NIST OSAC Forensic Science Standards Up for Public Comment

One of the responses to the 2009 NRC report on forensic science was the creation last year of an Organization of Scientific Area Committees (OSAC) for forensic science organized by the National Institute of Standards and Technology (NIST). This organization is developing new standards for forensic disciplines to follow.

Last week, NIST opened a 30-day public comment period for five standards from the Chemistry Scientific Area Committee. They are continuations or updates of existing ASTM (American Society of Testing and Materials) standards. The NIST OSAC News Release on the public comment period is at The five standards under consideration for inclusion on the OSAC Registry of Approved Standards are as follows:
  • ASTM E2329-14 Standard Practice for Identification of Seized Drugs
  • ASTM E2330-12 Standard Test Method for Determination of Concentrations of Elements in Glass Samples Using Inductively Coupled Plasma Mass Spectrometry (ICP-MS) for Forensic Comparisons
  • ASTM E2548-11e1 Standard Guide for Sampling Seized Drugs for Qualitative and Quantitative Analysis
  • ASTM E2881-13e1 Standard Test Method for Extraction and Derivatization of Vegetable Oils and Fats from Fire Debris and Liquid Samples with Analysis by Gas Chromatography-Mass Spectrometry
  • ASTM E2926-13 Standard Test Method for Forensic Comparison of Glass Using Micro X-ray Fluorescence (ยต-XRF) Spectrometry
Although they may seem technical and have forbidding names, some of these proposed standards should be of interest to lawyers as well as forensic scientists and statisticians who might want them to address how findings should be presented in court or in reports. For example, one standard involving glass fragments requires a difference of 3 standard deviations before the analyst can reject the hypothesis that the fragments on the suspect came from the crime scene. But 2.9 usually would be pretty good evidence that the suspect's fragments are from some other glass. What should the standard require or allow an expert to report in cases like this, where the suspect's fragments lie within the broad window for measurement error? Should there be an adjustment to the rejection range if more than one fragment has been tested? Should there even be a fixed window, or should the analyst simply report the probability of differences in the measurements as or more extreme as those observed if the fragments on the suspect came from the crime-scene glass? Better still, can a likelihood ratio be provided?

It appears that this 30-day period also offers an opportunity to view related ASTM standards on forensic science tests. Normally, ASTM, as the copyright holder, does not make its standards freely available.

Directions for subscribing to the OSAC newsletter and receiving announcements of comment periods, new standards, etc., are at the above URL and at

Disclosure and disclaimer: Although I am a member of the Legal Resource Committee of OSAC, the views expressed here (to the extent I have expressed any) are mine alone. They are not those of any organization. They are not necessarily shared by anyone inside (or outside) of NIST, OSAC, any SAC, any OSAC Task Force, or anyone else in the Legal Resource Committee.

Friday, July 24, 2015

What Proves that "the expert and his methods couldn’t possibly be reliable"?

I am developing an allergic reaction to the following kind of argument: "A forensic-science expert testified that a trace at the crime scene or on the victim was associated (to some degree of certainty) with the defendant. DNA later evidence exonerated the defendant. Therefore, the expert’s methods couldn’t possibly be reliable."

This reasoning is not very different from saying that a pitcher who does not strike out every batter couldn’t possibly be a reliable pitcher; that a polling firm that fails to correctly predict every election must be using methods that couldn’t possibly be reliable; or that a test for heart disease that sometimes errs couldn’t possibly be reliable. Without considering the success as well as the failure rate of the method, it is impossible to say that it is unreliable—or, by the same token, that it is reliable (in the sense of being worth relying on). 1/

Yet, the argument from cases of exonerations—lacking any comparison group—are legion in discourse on the use of trace evidence for identification. The latest example I encountered comes from a Washington Post blog site. Two days ago, Radley Balko wrote that
[A] defendant was convicted due to the testimony of a forensic expert who claimed that his “science” showed the defendant, and only the defendant, could have committed the crime. That conviction was later upheld by an appeals court in an opinion that explained in detail why the expert and his methods were legitimate and reliable. The defendant was later exonerated by DNA testing, thus demonstrating that the expert and his methods couldn’t possibly be reliable.
Mr. Balko went on to write that “not only did the courts continue to allow bite mark matching into evidence, every single time a defendant challenged its validity, that defendant lost.” 2/

Both the reasoning and the description of legal history are not quite right. To begin with, even absolute proof of innocence only shows that the test has a nonzero false-positive error rate (no surprise there) and that the witness should not have claimed to a certainty that no one else could have left the mark in question.

To be sure, some evidence that has found breathing space in the courtroom should be squeezed out entirely. But let’s face it—no scientific test meets the standard of perfection. If every case in which evidence that has produced false convictions meant “that the expert and his methods couldn’t possibly be reliable,” there could be no evidence. Overselling has occurred with every type of forensic evidence—from bitemarks to toolmarks to fingerprints to DNA. Courts, scientists, and criminalists should do their best to prevent this. Thus, whether through rules of evidence or through education and monitoring of analysts, testimony must be calibrated to the power of the scientific technique. The testimony should fairly express the known probative value of evidence from a validated method.

An example of testimony that violates this precept comes from Ege v. Yukins. 3/ In that case, a discredited dental expert testified as follows:
Q: Now, Doctor, with regard to your testimony, you indicated that it's highly consistent with the dentition of Defendant Carol Ege; is that correct?
A: Yes.
Q: Okay. With regard to—let me ask you a question. Let's say you have the Detroit Metropolitan Area, three, three and a half million people. Would anybody else within that kind of number match like she did?
A: No, in my expert opinion, nobody else would match up. 4/
The eventual outcome in the case contradicts the assertion that no challenge to the admission of bitemark evidence has succeeded. The state trial judge in Ege realized that the testimony was improper and only “denied [postconviction] relief because of the lack of a contemporaneous objection and a view that the showing of prejudice was insufficient.” 5/

A federal district court also concluded that “expert testimony identifying the petitioner as the only possible perpetrator of the alleged bite mark in the Detroit metropolitan area was improperly admitted.” 6/

The U.S. Court of Appeals for the Sixth Circuit agreed “with the district court that ‘Dr. Warnick's opinion that the petitioner was the only person in the entire Detroit metropolitan area who could have made the mark on the corpse carried an aura of mathematical precision pointing overwhelmingly to the statistical probability of guilt, when the evidence deserved no such credence.’” 7/ It affirmed the order for a new trial.

In short, the argument that a method of forensic identification that has been proved to be fallible is, for that reason alone, inadmissible proves too much. Likewise, the claim that no challenge to bitemark evidence has ever prevailed is exaggerated (although not by much). 8/

Please do not misunderstand me. The series of articles on bitemark evidence from which the remarks I have quoted were taken is impressive and useful. In offering these corrections to two small parts that seem a bit extreme, I am not arguing that bitemark analysis, which has little claim to validity, is either reliable (in the statistical sense that repeated analyses of the same marks give the same answers) or valid (in the sense that the answers are more often correct when marks from the same source are analyzed than when marks from different sources are compared). From the writing I have seen, bitemark analysis does not cut it.

I also believe that cases of false convictions should be studied and that the existence of a given type of scientific evidence in these cases should not be ignored. Finding a large number of false convictions with such evidence present is a warning signal. The evidence may come from a method that has a large false-positive rate, 9/ and that possibility must be investigated to decide whether the evidence should be excluded across the board or whether juries should receive the information -- together with an honest and clear explanation of the uncertainty in the results.

  1. See infra note 9.
  2. Radley Balko, A High-ranking Obama Official Just Called for the “Eradication” of Bite Mark Evidence, The Watch, Wash. Post, July 22, 2015.
  3. 485 F.3d 364 (6th Cir. 2007).
  4. Ege v. Yukins, 380 F.Supp. 2d 852, 871 (E.D. Mich. 2005), affirmed in part, reversed in part, 485 F.3d 364 (6th Cir. 2007).
  5. Id. at 857–58.
  6. Id. at 858
  7. 485 F.3d at 376.
  8. The federal courts in Ege treated the answer to the 3.5 million people as "probability testimony" without questioning Michigan's general rule that bitemark identifications are admissible. A true (and wrongly decided) case of bitemark probability evidence is State v. Garrison, 585 P.2d 563 (Ariz. 1978).
  9. The false-positive probability is P(+|O), where + is a positive statement ("the defendant left the mark") and O is the fact that some other person left the mark. Even if this probability is small, a disturbing number of false convictions could involve this evidence. Suppose that P(+|O) = 0.02, that 1,000 tests are performed in a set of cases with marks, and guilty defendants left the marks in 60% of these cases. The expected number of false positives is (0.02)(400) = 8. Assume that the probability of a true positive is P(+|S) = 0.96, where S means that the defendant is the source of the mark. Then the expected number of true positives is (600)(0.96) = 576. If defendants are convicted in all these cases, 8 convictions will be false (assuming that the culprit left the mark), and the many true positives will not be seen in the cases of exonerations of the innocent defendants. As indicated at the outset of these remarks, other data than exonerations are required to judge whether the test is reliable and valid.

Sunday, July 19, 2015

What the FBI Hair Examiner Said About Race in State v. Manning

Two previous postings discussed the meaning and import of FBI testimony about hair in a trial of Willie Manning. I questioned Justice Breyer's suggestion that the exoneration in one of these cases came about in part because "the evidence against him, including flawed testimony from an FBI hair examiner, was severely undermined." 1/ The evidence in the case that led to the overturned conviction did not include any hair evidence at all.

The verdict in the other case -- the one with hair evidence -- is still is question. Cellmark Forensics is trying to find and analyze DNA from a rape kit and fingernail scrapings. At a later point, the laboratory may turn to "other items of evidence," including, presumably, the hair fragments from the car of one of the victims. 2/

Testimony about this hair figured prominently in the trial. In closing, the prosecutor asked "how many [people] could leave hair fragments in the car, hair fragments that came from a member of the African-American race because that's what they find when they vacuum the sweepings of the car, that's what they find in both significantly the passenger's seat and the driver's seat, just like it would be if the man rode out there as a passenger and came back as a driver . . . ." 3/

The prosecutor was able to make this argument because of testimony from an FBI examiner about the perceived racial characteristics of the hair. In my posting of July 16, I wrote that the Department of Justice's letter of May 4 confessing scientific error
is hardly a repudiation of the testimony that the hairs from the car "exhibited characteristics associated with the black race." To the contrary, it endorses this testimony as a permissible "scientific analysis." ... What "would be error," in the DOJ's view, is "any statement of probability whether the hair is from a particular racial group." But it is impossible to tell from the letter whether the FBI agent gave any such testimony. The federal district court's opinion denying Manning's habeas corpus petition makes it sound like the testimony was not of the sort later deprecated by the FBI. 4/
Today, I obtained the motion that persuaded the Mississippi Supreme Court to stay the execution and to order DNA testing. 5/ The FBI agent's testimony is appended to it. The examiner not only presented the characteristics of the hairs as associated with race ("valid" according to the DOJ), but added that "these hairs were hairs from an individual of the black race" 6/ -- not "valid."

  1. David H. Kaye, Justice Breyer in Glossip v. Gross on "flawed testimony from an FBI hair examiner", July 16, 2015, Forensic Science, Statistics & the Law
  2. Letter from Cellmark Forensics to Robert L. Mink, June 5, 2015, available at
  3. Motion to Stay Execution and Set Aside Convictions, Second Motion for Leave to File Successive Petition for Post-Conviction Relief, and Motion in the Alternative for Other Forms of Relief, at 5, Manning v. Mississippi, No. 2013-DR-00491-SCT (May 6, 2013).
  4. Kaye, supra note 1.
  5. Motion, supra note 3.
  6. Transcript at 1048.

Friday, July 17, 2015

Validity, Overclaiming, and Error: More on Willie Manning's Exoneration

Yesterday I questioned reliance on the exoneration of Willie Manning as a case in which hair evidence was severely undermined, prompting his exoneration. The problem with this understanding of the Manning exoneration is that the crimes for which his conviction was overturned did not involve any hair evidence. In addition, I tried to probe what Department of Justice reported was erroneous or invalid about the hair evidence against him in a separate case. I concluded with the observation that
It is entirely possible that DNA testing will soon exonerate Manning in the Miller-Steckler murder case (to the extent of showing that the hair in the car was not his). As of May, 2015, this had not happened. If it does, it would mean that either the original determination of the racial characteristics of the hair was wrong -- something that the FBI has not conceded -- or that the determination was correct but that Manning was not the person whose hairs were in the car -- something that the criminalist never purported to resolve.
Scott Henson, "a former journalist turned opposition researcher/political consultant, public policy researcher and blogger who is presently Executive Director of the Innocence Project of Texas" 1/ and whose Grits for Breakfast blog is a great source of information with a Texas flavor, made an important comment:
Perhaps that's "something that the criminalist never purported to resolve," but posing that implication is of course the reason the prosecutor presented his or her testimony. When they're reviewing these cases, the issue isn't just whether the forensic examiner erred but also if prosecutors overstated or misused the evidence to ask the jury to draw invalid conclusions. (In Texas' review of state-level examiners, with which I'm more familiar, they examine both the forensic testimony AND the prosecutor's closing arguments.)

If the DNA testing comes back excluding Manning, IMO it'd be fair to say that the use of hair testimony in his case has been "severely undermined," since it would have treated exculpatory evidence as inculpatory.
Overstating the scientific findings by experts or parties is a major problem. The Santae Tribble case is a glaring example of prosecutorial overclaiming. The Assistant US Attorney told the jury that "[t]here is one chance, perhaps for all we know, in 10 million that it could [be] someone else’s hair." But published estimates of this probability range from 1/4500 to 1/10 (with the latter being much more plausible). Closing arguments like this are therefore unacceptable. If courts permit hair evidence to be used, then a prosecutor who does so has a duty to use it only for what what it is worth. In Tribble, the prosecutor claimed that the hairs had matching features and then overclaimed that the probability of this match was negligible if it was someone else's hair.

The odd thing about the Manning case in which hair evidence played a role, however, is that the overclaiming did not relate to the claim. The overclaiming that the FBI spotted in its hair evidence review project pertained to the ability to identify individuals by a combination of many physical characteristics of hair. But the expert expressly stated that that could not be done in this case and only made a claim about race.

Could the prosecutor have gone beyond the expert's testimony about race and "overstated or misused the evidence to ask the jury to draw invalid conclusions"? Yes, that could be. The FBI's hair review project is limited to the reports and testimony of FBI hair examiners. It will be interesting to see whether states implement hair evidence review projects and what their scope will be.

Whether or not there was any relevant overclaiming in Manning, it is true that a DNA test showing that the hair is not Manning's would severely undermine the state's case. But it would not necessarily undermine the validity of the examiner's testimony that the hair had certain "racial characteristics." (Whether, as the DOJ letter indicated, racial classifications are valid to begin with is, of course, a different question.) 2/

To see that disproof of an inference from scientific evidence that arises from better scientific evidence does not make the original evidence invalid, let's switch from hair to serology for a moment. Sometimes hair can be analyzed for ABO blood groups. As a thought experiment, suppose that the testimony had not been about race but was only that both Willie Manning and the hair in the car were blood group B, and that Manning was among the 10% or so of the U.S. population whose ABO type is consistent with this finding. Would a later mitochondrial DNA test that excluded Manning severely undermine this ABO testimony? It certainly would contradict the conclusion that Manning left hairs in the car. It would change the hair evidence from inculpatory to exculpatory.

But it would not show that ABO testing is invalid in general or that the value of the test was overstated in this case. It would be a reason to exonerate Manning but not a reason to denominate the original evidence as scientifically invalid or as having exceeded the limits of science. In this respect, it would be no different than incontrovertible evidence that Manning was incarcerated when the murders were committed -- powerful evidence of innocence but no evidence of invalidity or overclaiming.

  1. Scott Henson, Who Is This Guy?, Grits for Breakfast, Jan. 6, 2004,
  2. For a short, sympathetic review of the theory of racial classification of hairs, see Edward J. Imwinkelried, Forensic Hair Analysis: The Case Against the Underemployment of Scientific Evidence, 39 Wash. & Lee L. Rev. 41, 49-50 (1982).

Thursday, July 16, 2015

Justice Breyer in Glossip v. Gross on "flawed testimony from an FBI hair examiner"

Justice Stephen Breyer recently proposed that the Supreme Court reconsider the constitutionality of the death penalty. One of his concerns is that despite the level of scrutiny capital sentences are supposed to receive, innocent defendants may be executed. Justice Breyer's worry on this score cannot be dismissed as historically unfounded. Neither can it be written off as the predictable moaning of a bleeding-heart liberal, coming as it does from the "high court's raging pragmatist."

A related concern articulated by Justice Breyer is the "uncertainty as to whether a death sentence will in fact be carried out" and when that day will arrive. "Willie Manning," for example, "was four hours from his scheduled execution before the Mississippi Supreme Court stayed the execution. See Robertson, With Hours to Go, Execution is Postponed, N.Y. Times, Apr. 8, 2015, p. A17. Two years later, Manning was exonerated after the evidence against him, including flawed testimony from an FBI hair examiner, was severely undermined. Nave, Why Does the State Still Want to Kill Willie Jerome Manning? Jackson Free Press, Apr. 29, 2015."

Without questioning the main point about the uncertainty of confinement, I want to probe the use of the Manning case -- or rather cases -- as an exoneration in which "flawed testimony from an FBI hair examiner was severely undermined."  Justice Breyer has been called a "technocrat," and it is not surprising that he would call attention to the hair evidence against Manning and to "the more general problem of flawed forensic testimony" as illustrated by reports that "FBI Testimony on Microscopic Hair Analysis Contained Errors in at Least 90 Percent of Cases in Ongoing Review," including capital ones like Manning's. But what was the erroneous hair evidence against Manning, and in what sense was it erroneous?

The answers are complicated, and it turns out that hair evidence played no role in the case in which Manning was exonerated. Manning was convicted in not one, but two, capital cases. The hair evidence applies to a different pair of murders for which Manning has yet to be exonerated, and it is hard to see how the hair testimony there has been "severely undermined." For clarity, I discuss the separate cases separately. 

I. The Jimmerson-Jordan Murders

According to the article cited by Justice Breyer,
An Oktibbeha County jury convicted Manning for killing nonagenarian Emmoline Jimmerson and her daughter, Alberta Jordan, in the winter of 1992. The women were beaten and their throats slashed during an apparent robbery attempt at their Brookville Gardens apartment in Starkville. Manning was convicted of the crime at age 26 and sentenced to death.

The state's star witness, a man named Kevin Lucious, told police and later testified in court, that he saw Manning enter the victims' apartment from his own apartment, but police found the apartment where Lucious claimed to live was vacant at the time of the crime. The apartment manager also had no record of Lucious being a tenant.

Presiding Justice Michael K. Randolph, on behalf of the supreme court's majority, ordered the case back to circuit court for a new trial, agreeing with Manning's attorneys that "there is no question that defense counsel would have had the opportunity to meaningfully impeach Lucious' testimony that he lived in the apartment at the time of the crime and saw Manning enter the victims' apartment." ...

[Later], Luscious recanted most of his statements, saying he only testified because he feared being charged with the crime himself. ... Luscious claimed that he told Sheriff Bryan that another man, Tyrone Smith, had confessed to the murders. With the state's material witness now changing material parts of his story, the case had to be thrown out [rather than retried].
Neither this article nor the entry in the University of Michigan's National Registry of Exonerations refers to any forensic-science evidence in the case. The Registry lists the factors that contributed to the conviction as follows: "perjury or false accusation, official misconduct."

II. The Miller-Steckler Murders

In 1992, two Mississippi State University students, Tiffany Miller and Jon Steckler, who were dating, were shot and killed near the fraternity house in which one of them lived. Steckler's fraternity brother had a car that was burglarized at around the same time. The state produced evidence that Manning, who had a record of convictions for theft and other crimes, was distributing items stolen from the car. Manning conceded that he was selling stolen goods but said that he did not know who stole them. Among the other evidence introduced against Manning was hair found in Miller's car. The car was found near campus, and it had been used to run over Steckler.

An FBI criminalist testified that he could "microscopically determine if the hairs look alike and determine with some degree of certainty, although not absolutely, but with some degree of certainty if hairs, for example, found in vacuum sweepings from an automobile originated from a particularly named individual." Manning v. State, 726 So.2d 1152, 1180 (Miss. 1998). He also testified that hairs from the car "exhibited characteristics associated with the black race." Id. The examiner "went on to testify that as the hairs were only fragments, he could not compare the hairs to a known sample, and that he was limited to a determination as to the racial characteristics of the hair." Manning v. Epps, 695 F.Supp.2d 323, 380 (N.D. Miss. 2009). This determination was significant because the two victims were white, and Manning is black. Knowing that the hair had the latter "racial characteristics" made it more probable that it was Manning's and thus that he was in the car.

The DOJ issued not one, but two letters about the 1994 testimony of its agent. The first letter, dated May 2, 2013, reported that
the microscopic hair comparison analysis testimony or laboratory report presented in this case included statements that exceeded the limits of science and was, therefore, invalid. While this case did not involve a positive association of an evidentiary hair to an individual, the examiner stated or implied in a general explanation of microscopic hair comparison analysis that a questioned hair could be associated with a specific individual to the exclusion of all others -- this type of testimony exceeded the limits of the science. The examiner also assigned a statistical weight or probability or provided a likelihood that, through microscopic hair comparison analysis, the examiner could determine that a questioned hair originated from a particular source, or an opinion as to the likelihood or rareness of a positive association that could lead the jury to believe that valid statistical weight can be assigned to n microscopic hair association -- this type of testimony exceeded the limits of the science. (A copy of the documents upon which this determination was based is enclosed.)
I have not seen a copy of the testimony, but the letter does not suggest that the testimony used to link Manning (and all other African Americans) to the hair in the car was erroneous or undermined. Apparently, the criminalist overstated the power of hair features as the basis for a probabilistic or categorical statement that an individual was in fact the source of a hair, but as we saw, the expert here shied away from either of those statements. He testified only to the race of the unknown individual whose hair was in the car.

Two days later, the DOJ distributed a second letter referring to an "additional error." This letter addressed the racial identification. One might well wonder about the "racial characteristics" of hair. How definitive of race are these features? Can criminalists really zero in on African Americans this way?The letter had this to say:
We have determined that the microscopic hair comparison analysis testimony or laboratory report presented in this case included additional statements that exceeded the limits of science and was, therefore, invalid. In response to inquiries regarding whether the errors identified in the notification letter had any bearing on the examiner’s opinion regarding the racial classification of the hair, the FBI states the following: The scientific analysis of hair evidence permits an examiner to offer an opinion that a questioned hair possesses certain traits that are associated with a particular racial group. However, since a statistical probability cannot be determined for classification of hair into a particular racial group, it would be error for an examiner to testify that he can determine that the questioned hairs were from an individual of a particular racial group. Thus, an examiner cannot testify with any statement of probability whether the hair is from a particular racial group, but can testify that a hair exhibits traits associated with a particular racial group. (A copy of the FBI Microscopic Hair Analysis Report, dated May 4, 2013, is attached.)
This paragraph is hardly a repudiation of the testimony that the hairs from the car "exhibited characteristics associated with the black race." To the contrary, it endorses this testimony as a permissible "scientific analysis." (It is not clear to me that this assessment of the science is correct, but I have not researched the scientific literature.) What "would be error," in the DOJ's view, is "any statement of probability whether the hair is from a particular racial group." But it is impossible to tell from the letter whether the FBI agent gave any such testimony. The federal district court's opinion denying Manning's habeas corpus petition makes it sound like the testimony was not of the sort later deprecated by the FBI. Manning v. Epps, 695 F.Supp.2d 323, 380 (N.D. Miss. 2009).

The letter added that the FBI was prepared to perform DNA tests on the hairs or other biological material if desired. The Mississippi Supreme Court called off the execution to allow Manning "to proceed in the circuit court with his request for DNA testing ... ." Manning v. State, 119 So.3d 293, 293 (Miss. 2013). In 2015, Manning's lawyer "said several items have been sent to a lab in Houston, Texas, for analysis" and that "the timing of the testing and issuing of results is up to the lab and the FBI." 1/.

It is entirely possible that DNA testing will soon exonerate Manning in the Miller-Steckler murder case (to the extent of showing that the hair in the car was not his). As of May, 2015, this had not happened. 2/ If it does, it would mean that either the original determination of the racial characteristics of the hair was wrong -- something that the FBI has not conceded -- or that the determination was correct but that Manning was not the person whose hairs were in the car -- something that the criminalist never purported to resolve.

  1. R. I. Nave, Why Does the State Still Want to Kill Willie Jerome Manning? Jackson Free Press, Apr. 29, 2015, available at
  2. Maurice Possley, Willie Manning, Nat'l Registry of Exonerations, ("Manning remains on Mississippi’s Death Row for the Miller-Steckler murders as the physical evidence in that case was still undergoing DNA testing as of April 2015.").

Tuesday, July 14, 2015

In His Own Words: Justice Scalia's Poetry Slam

In discussing the exchange in Glossip v. Gross between Justices Breyer and Scalia on the deterrent effect of capital punishment, I alluded to Andrew Lang (1844-1912), Scottish poet, novelist, and literary critic. Although any further connection to science or statistics is nonexistent, I can't resist a poetic tangent. Fred Moss pointed me to an animated "poetry slam" of Justice Scalia's "snarky ... poetic gems" at the cartoonist, Mark Fiore's website. Rearranging bits of the concurrence in Glossip yields this additional free verse:

they have discovered the lost folios of Shakespeare
they ask us for clemency, as though clemency were ours to give
If only Aristotle, Aquinas, and Hume knew
that moral philosophy could be so neatly distilled
into a pocket-sized, vade mecum
full of internal contradictions and (it must be said)
gobbledy-gook Welcome to Groundhog Day.

Related posting:
Two Scientific Issues in Glossip v. Gross, July 14, 2015, Forensic Science, Statistics, and the Law