NIJ Multisite Impact and Cost-Efficiency Evaluation of Veterans Treatment Courts

The proposed project is a collaboration led by American University (AU) with Western Carolina University (WCU), the U.S. Department of Veterans Affairs (VA), Abt Associates, and M‐W Consulting. AU proposes process, impact, and cost‐efficiency evaluations to empirically examine a variety of legal and extra‐legal outcomes
using rigorous experimental or quasi‐experimental methods to address gaps in research on VTCs.

Process questions include: 1) What are the structures, policies, procedures, and participant populations of the VTCs? 2) How are the selected sites comparable and different? 3) Are they operating with fidelity/adherence to best practices?

Impact questions include: 1) Do justice involved veterans (JIVs) who participate in VTCs experience better outcomes compared to those otherwise processed (business‐as‐usual) through the criminal justice system (CJS)? 2) Do certain types of VTC participants experience better outcomes than others? 3) Are certain VTC program elements more effective than others?

Cost‐efficiency questions focus on determining if it is more cost effective over the short‐ and/or long‐term to treat JIVs through VTCs rather than through the traditional CJS (business‐as‐usual).

Process Evaluation
The process evaluation will identify variability in the six sites’ structure, policies, and procedures. Process data will come from: site selection assessment, site selection interviews with VTC team members, a Treatment Courts Best Practices Inventory (BPI), program manuals, participant handbooks, and court and staffing observation. Practices related to appearance frequency, peer mentorship, military culture, use of technology, service access, and COVID‐19 will be areas of particular focus. Interim dissemination items include logic models, process models, process report, a scholarly manuscript, and one fact sheet or infographic.

Impact Evaluation
Secondary data are collected on an ongoing basis for the impact evaluation from the VTC program, as well as from the general court, jail, probation, and VA records. The Onsite Research Liaisons will work with the Site Liaisons and Agency Research Personnel to obtain data. Abt will be responsible for conducting program fidelity checks; site data collection, cleaning, standardization, security, and transfer; and ongoing site communications. The Project Management Plan references Abt as Onsite Research Liaison Lead and Six Onsite Research Liaisons TBN.

The Onsite Research Liaisons will collect primary data following the IRB‐approved research protocol across the six sites. An Abt Data Processor will work with Site Liaisons and Agency Research Personnel to assess, obtain, and process administrative data from the six sites to ensure that it includes the needed variables. The VA co‐PI will lead VA data analysis and work with a VA statistician to create and analyze electronic health record datasets merged with other datafiles. They plan to obtain separate VA IRB approval for VA electronic health records in Year 4 (Footnote 7).

Primary data will be collected via interviews and oral drug test swabs of JIVs determined to be VTC eligible. Interviews will be administered via Qualtrics by the Onsite Research Liaisons, beginning with Baseline at program eligibility screening; follow‐up interviews are at 6, 12, 18, 24, and 30 months post‐baseline. Site Liaisons will inform potential subjects of the study when they schedule eligibility assessments with eligible JIVs. The Onsite Research Liaison will meet the VTC‐eligible JIVs afterwards, using
the study consent process. The Baseline interview occurs after VTC eligibility determination, and before randomization and offers of program admission. Incentive gift cards will be offered. The incentive schedule starts at $40 per baseline interview and $25 per swab and increases every 6 months to $65 and $30 by the 30‐month follow‐up. In addition to the variables described above, interviews will include Ohio Risk Assessment System Pretrial Assessment Tool (ORAS‐PAT) for a standard risk measure across sites.

Power analyses estimates indicate 89 participants in each treatment/ control group per site. An additional 10% to account for potential attrition, provides a total target sample of 1,188. They will generate random allocation sequences for each program which the VTC will not see. The Site Liaison informs the Onsite Research Liaison someone is eligible, and they apply their allocation sequence which assigns the JIV to a treatment or control condition. The Site Liaison is informed and offers participation to the treatment group and processes those in the control group through the regular CJS.

Alternatively, propensity score matching requires a pool of eligible JIVs who do not experience the treatment and identifying and accounting for confounders that contribute to the selection process. Options include opt‐outs and exclusions for arbitrary administrative reasons (defense attorney), with information on larger JIV sample using VA data reports on jail census.

Statistical models are driven by the nature of the outcome in question (logistic models for binary outcomes), as well as the structure of the data (multilevel models with person‐level random effects for time points nested within individuals). Interaction terms will be added to the model to explore differential VTCs effects on the basis of JIV (risk/need, non/violent offenses, pre/post‐plea, diagnoses) and program characteristics (mentorship, use of technology, treatment access).

Cost‐Efficiency Evaluation
M‐W Consulting will conduct the cost‐efficiency evaluation. Planning activities include reviewing cost data availability and providing input on site selection and instrumentation. Local budget data will be gathered to estimate the costs for a VTC compared to CJS business‐as‐usual during the Research phase. Data will be obtained to estimate how JIVs move through each site’s CJS and how much of each resource a typical JIV uses. Marginal costs of case processing will be estimated by combining budget data on
employee wages and benefits with local professional expertise on time spent on case types in a typical week, for an estimate of the case processing cost differential between VTC and business as usual. Analyses will differentiate between actual budgetary savings and time savings (opportunity costs); they will confirm VTC costs and funding sources, whether fees paid by participants or taxpayer
budgets.

Dissemination
Dissemination plans include required NIJ final data and written deliverables, a Project Webpage, a process report with logic models and process models, three scholarly manuscripts, four fact sheets and infographics, three conference presentations (American Society of Criminology, NADCP, and American Probation & Parole Association), two National Drug Court Resource Center podcasts, a research brief for VA National Center on Homelessness Among Veterans, and a VA Veteran Justice Program Briefing. CA/NCF