Firearm Toolmark Population Statistics for Objective Identification and Error Rate Estimation

The 2009 report by the National Academies [1] on forensic science in the United States expressed concern that “..the decision of the toolmark examiner remains a subjective decision based on unarticulated standards and no statistical foundation for estimation of error rates.” The report recommended development of precisely specified, and scientifically justified, protocols that yield an objective determination of a match or non-match with well-characterized confidence limits [1, 2]. The development of such protocols is hampered by insufficient knowledge on the degree of similarity that can be found between marks made by different firearms and of the variability between marks made by an individual firearm [1, 2]. The proposal addresses this challenge through development and evaluation of a framework for generating and updating statistical frequency distributions of toolmark similarity scores for various firearm populations

Part one of this proposal will data mine the NBTRD and collect targeted test fires to generate population statistics for breech face identifications. Population statistics describe the frequency distributions of a similarity score for, respectively, same-source comparisons and different-source comparisons. Similar to DNA analysis, these distributions are needed to establish a statistical foundation for the estimation of identification confidence limits and error rates. An important component of the proposed research is to systematically evaluate, for different firearm populations, the effects of key processing parameters on the comparison score distributions and associated error rate estimates. Conducting this research will play an important role in determining the extent to which the comparison protocol, population statistics, and error rate estimation need to be tailored to a particular firearm sub-population. Five firearm models will be chosen for the initial population tests. 100 firearms of each manufacturer will generate at least two test fires for a minimum of 1000 test fires. Known match and known non-match distributions will be generated for the global population as well as by firearm manufacturer. The distributions will be applied to estimate the respective error rates. Initially, two different similarity metrics/scores will be evaluated: the normalized areal cross correlation coefficient [5] and the number of congruent matching cells (CMCs) [30-35]. These similarity metrics provide indications of sample topography similarity at both the global and local level. The envisioned framework will enable evaluation of other similarity metrics and the dynamic update of distributions when new data becomes available.

Part two of this proposal addresses improvements to the public NBTRD to facilitate its maintenance, expansion, and usage. New features include: 1) batch-upload and reusable entry tables to facilitate submission of large datasets, 2) usage statistics to identify areas of interest, 3) refinement of search results, and 4) migration to a commercial cloud server for improved bandwidth and storage capacity.