Establishing a National Firearms and Toolmark Statistical Background Population for Calculation of Error Rates

The 2009 report by the National Academies on forensic science in the United States expressed concern that “... the decision of the toolmark examiner remains a subjective decision based on unarticulated standards and no statistical foundation for estimation of error rates.” The report recommended development of precisely specified, and scientifically justified, protocols that yield an objective determination of a match or non-match with well-characterized confidence limits. In the previous five years, the Forensic Toolmark Analysis Project (FTAP) at NIST, with support from NIJ, has been making significant progress to address the technological and validation gaps within firearm and toolmark examinations. These include objective similarity algorithm development, open access 3D and 2D database of firearm toolmark measurements (2013-R2R-4843), development of calibration artifacts for 3D instrumental and statistical modeling of known matching and known non-matching datasets (2016-DNR-6257). These NIST research projects form the foundation required to ultimately achieve the ability to report a statistical confidence for firearm toolmark comparison conclusions. This research proposal aims to define and implement background reference population distributions for firearm toolmarks arising from different machining methods and the firearm-ammunition pairing. A population distribution describes the frequency distributions of a similarity score for, respectively, same-source comparisons and different-source comparisons. Like DNA analysis, these distributions are required to establish a statistical foundation for the estimation of identification confidence limits and false positive error rates. The framework for this research was developed in a previously funded NIJ grant (2016-DNR-6257) which targeted five specific firearm manufacturers with 100 firearms from each. To build upon this framework, the proposed research will dramatically expand the coverage of firearms and ammunition types encountered as evidence in case work. An important component of the proposed research is to systematically evaluate, for different firearm populations, the effects of key processing parameters on the comparison score distributions and associated error rate estimates. Conducting this research will play an important role in determining the extent to which the comparison protocol, population statistics, and error rate estimation need to be tailored to a firearm sub-population. Another aim is the development of a tool and supporting protocols for implementation within a forensic laboratory that will enable firearm examiners to calculate a cumulative false positive (CFP) and false negative error rate (CFN) for the comparison results of any two firearm toolmarks. This will greatly improve the objectivity of an examiner’s testimonies in court, add a quality metric to the laboratory quality management system, as well as advance the science of firearms and toolmark examinations.