Comparison of Examiner Conclusions with Score-based Likelihood Ratios for Cartridge Case Evidence

The project goal is to evaluate the parity between subjective firearm toolmark comparison results obtained by examiners and objective statistical measures for the strength of the evidence, as expressed by algorithmically generated score-based likelihood ratios (SLRs).

The project addresses concerns about the lack of objective statistical measures for the strength of the evidence of a specific toolmark comparison result, as highlighted in e.g., the 2009 report by the National Academies and the 2016 report by the President's Council of Advisors on Science and Technology (PCAST). In response, the FBI Laboratory, NIST, and the Netherlands Forensic Institute (NFI) have developed the Reference Database of Firearm Toolmarks (RPDFT) framework. This system is designed to provide firearm examiners with a statistical measure for the weight of evidence. The system uses a reference database of toolmark images obtained for representative samples of different firearm and ammunition brands. The images are used to automatically generate known matching and known non-matching distributions of comparison scores that are relevant to a particular case. The comparison score obtained during casework is then compared with these distributions to generate a score-based likelihood ratio for the strength of the evidence that the examiner can use to confirm and strengthen their findings. The project builds on the recent FBI/AMES II Decision Analysis Study (black box study), where 173 firearm examiners provided comparisons results for 4320 comparisons of fired cartridge cases. Comparison results were expressed using the AFTE range of conclusions, which includes three degrees of inconclusive. In this project, we will apply the RPDFT system to generate a score-based likelihood ratio for every comparison challenge in the study. These estimates for the strength of the evidence are then compared with the examiner conclusions.

The main objectives of the project are: 1) Evaluation of the range of likelihood ratios for each of the AFTE conclusions, 2) Evaluation as to whether examiner results of inconclusive are supported by the respective likelihood ratios, 3) Evaluation as to whether an algorithmic likelihood ratio estimate could have prevented false positive or false negative results, and 4) Identification of limitations of score-based likelihood ratios compared to examiner results. Overall, the study seeks to provide objective and foundational knowledge on the ability of score-based likelihood ratios to strengthen results obtained by an examiner. Project results provide a key milestone for the inclusion of objective statistical measures for the strength of the evidence in examiner testimony.