The problem is that, even though fingerprints have been used in criminal courts for more than 100 years, no properly controlled experiments on fingerprint examiners' accuracy in identifying perpetrators had been conducted. Some experts have even claimed to be infallible, but mistakes made to date have resulted in innocent people being wrongly accused.
We wanted to find out whether these experts were any more accurate than the average person, and to get an idea of how many criminals are being wrongly set free and how many innocents are being wrongly convicted. Well, it took two years of planning and a visit to every major state police department in Australia, but we finally managed to conduct the critical experiment.
We gave 37 qualified fingerprint experts and 37 UQ students pairs of fingerprints to examine and decide whether a simulated crime-scene print matched a potential suspect or not. Some of the print pairs belonged to the "criminal" while others were highly similar but actually belonged to an "innocent" person.
The experts correctly matched just over 92 percent of the prints to the criminal. But, they mistakenly matched 0.68 percent of the prints to the innocent person. That they made so few errors means pretty impressive human performance, in my opinion.
We concluded that qualified court-practicing fingerprint experts are exceedingly accurate compared to novices, but are not infallible. Our experts tended to err on the side of caution by making errors that would free the guilty rather than convict the innocent. Even so, they made the kind of error that may lead to false convictions.
So, was my initial skepticism unwarranted?
Well, I think healthy skepticism is the cornerstone of science and rational enquiry. Fingerprint examiners make important decisions that can put lives and livelihoods at risk and the burden of proof is on the profession to demonstrate the scientific validity and reliability of its claims. We’ve worked closely with examiners, who were were eager to demonstrate their abilities, and we’ve shown that expertise with prints provides a real benefit.
Even though this one experiment consumed my existence for two years, it was totally worth it!
Matt
mbthompson.com
Tangen, J. M., Thompson, M. B., & McCarthy D. J. (2011). Identifying Fingerprint Expertise. Psychological Science. [PDF] [Press Release]
"But, they mistakenly matched 0.68 percent of the prints to the innocent person."
ReplyDeleteAs a cognitive scientist, I think an expert false positive error rate of .68% in our experiment is very low.
ReplyDeleteBut is this false positive rate high or low in the context of the tens of thousands of fingerprint examinations made per day? And what of the 8% errors of the kind that may allow a criminal to walk free?
I'm leaving the broader interpretation of the significance of this evidence to legal scholars and the fingerprint profession itself.
Matt, I'm wondering if you could comment on how finger print identification is used in the judicial process.
ReplyDeleteMy personal opinion is that these error rates demonstrate that we absolutely should be using finger print expert testimony in courts. They err on the side of caution, and seem to make only a tiny number of errors under the most difficult conditions. Even given the large number of examinations per day, I think this false positive error rate is more than acceptable, provided it's not the only piece of evidence used to convict someone of a crime.
A question regarding statistical certainty: is it possible to determine if the low false positive error rate is significant different from zero?
Good job, Matt. Your post reminds me of one of my favourite xkcd comics: http://xkcd.com/54/
ReplyDelete@Will re statistical certainty - yes, it's statistically possible to construct such a test, but I'm not sure I understand what it would demonstrate? I think the reverse may be more important - i.e. they're NOT infalliable even if they're right 99.999% of the time.
The anecdotes I've heard from friends who have given expert testimony is that the legal system is not good at dealing with rare probability events - which is a concern because no real world test is ever perfectly sensitive and specific.
@Will Just to make myself useful... something like the Dunnet method for multiple comparisons of all treatments with a control (i.e. 0%) would be appropriate (I think Statistica has something like t-test against a constant).
ReplyDeleteYou basically repeat this experiment n times, generate a 95% confidence interval for the false positive error rate, and see if 0% lies within the confidence interval.
(There are few statistical technicalities here, but I don't think they're an issue.)
That's the statistical method that I suggested at lunch today :)
ReplyDeleteAs for the point, I was mostly curious. (I focus the following on false alarms only.) Matt posed the question, "But is this false positive rate high or low in the context of the tens of thousands of fingerprint examinations made per day?" There is an unstated premise here that presumes there is a real false alarm rate.
I find it interesting to think about - if the false alarms are statistically indistinguishable from zero, does that mean they are infallible? Probably not: they made mistakes, and that's still potentially three out of 440 people who might be wrongfully convicted, to use an extreme outcome. However, can we distinguish these false alarms from measurement error? From this study, my guess is that we cannot.
I think this topic was debated on Matt's Facebook page when the article was published.
ReplyDeleteIn short, since there's a selection bias (statistically speaking in a non-pejorative sense) it's not possible to generalise the numbers in this experiment to the population false positive rates - which is what you're saying, I believe?
However, since Matt et al. specifically targeted difficult-to-match fingerprints, it would be reasonable to assume that the population rate is lower (i.e. the 0.68% is an upper bound for the true false-positive rate), which I think is Matt's point?
I've not read the article but could someone comment on the accuracy rate of the novices(UQ students)?
ReplyDeleteI read 'experts are exceedingly accurate compared to novices'.
Any idea if there were other mediating factors such as participant motivation (a.k.a 'the 1st yr psyc student dozing off')?