A Homogeneity Test for Ordinal Receiver Operating Characteristic Regression With Application to Facial Recognition Accuracy Assessment

• NCJ Number: 310449
• Journal: Statistical Analysis and Data Mining Volume: 18 Issue: 2 Dated: April 2025 Pages: e70021
• Author(s): Ngoc-Ty Nguyen; Larry Tang; Lulu Chen; P. Jonathon Phillips
• Date Published: April 2025
• Grant Number(s): 2019-DU-BX-4011
• Agencies: NIJ-Sponsored
• Publication Type: Research (Applied/Empirical)
• Award: Accounting for Covariates in Forensic Error Rate Assessment and Evidence Interpretation

Annotation

This article presents a research study that proposed a new covariate-adjusted homogeneity test for ordinal receiver operating characteristic curves to determine the differences in accuracy among radiologists’ medical imaging studies and forensic examiners’ biometric studies.

Abstract

Ordinal scores occur commonly in medical imaging studies and more recently in black-box studies on forensic identification accuracy. To assess the accuracy of radiologists in medical imaging studies or the accuracy of forensic examiners in biometric studies, one needs to estimate the accuracy measures such as the receiver operating characteristic (ROC) curves and also account for the covariates related to the radiologists or forensic examiners. The novelty of the paper is twofold. First, the authors propose a new covariate-adjusted homogeneity test for ordinal ROC curves to determine differences in accuracy among multiple rater groups. Second, since the covariance structure among the ROC regression estimators is not available, they obtained the asymptotic covariance matrix of the ROC estimators and derived theoretical results of the proposed test. The authors conducted extensive simulation studies to evaluate the finite sample performance of the proposed test. The simulation results show that estimated ROC curves are consistent, and the empirical coverage of the confidence intervals is close to the nominal level. The authors’ proposed test is applied to a large-scale face recognition study in which participants include facial examiners, facial reviewers, super-recognizers, fingerprint examiners, and students. The results show differences in accuracy among five rater groups. Ad-hoc pairwise comparison tests are then conducted by establishing confidence bands of differences among ROC curves. Those pairwise tests identify statistically significant differences in ROC curves among five participant groups. (Published Abstract Provided)