NIJ Multisite Impact and Cost-Efficiency Evaluation of Veterans Treatment Courts, Fiscal Year 2022

LINDA TRUITT: Welcome to the pre-recorded webinar for the FY22 solicitation, NIJ Multisite Impact and Cost-Efficiency Evaluation of Veterans Treatment Courts. In collaboration with the Bureau of Justice Assistance and other federal agencies, the National Institute of Justice has developed plans for a cross-site evaluation of veterans treatment courts. I’m Linda Truitt and I manage NIJ’s Courts Research Portfolio, as well as coordinate NIJ’s Drugs and Crime Research Portfolio. Our plans are ambitious and the evaluation requires rigorous research, featuring an impact study and a cost study, in partnership with up to six VTC programs. On October 29, NIJ released a research solicitation for a proposal to conduct a $3 million cross-site evaluation over five years. That period of study is necessary to examine long-term outcomes between participants and others who did not enter the program, using a sample large enough to also examine the effects of differences among the individuals and their varied program experiences. Research applications are due by January 27, and registration two weeks before then. This webinar is in collaboration with two BJA colleagues, Greg Torain and Courtney Stewart. They will contribute information from BJA’s perspective, administering the Department’s Veterans Treatment Court program. Greg?

GREGORY TORAIN: Thank you, Linda. My name is Gregory Torain. I am the Policy Advisor at BJA, and I oversee BJA’s adult drug court grant, discretionary grant program, and that is composed of our adult drug court site-based awards, as well as our training and technical assistance.

LINDA TRUITT: And Courtney?

COURTNEY STEWART: Hello. My name is Courtney Stewart, BJA Grants Management Specialist, and I manage BJA’s Veteran Treatment Courts and Community Courts grant portfolios.

LINDA TRUITT: Thank you. We’re inspired to record this webinar during the week of Veteran’s Day and very pleased to move forward on this important research. This webinar provides an overview of our plans for the cross-site evaluation and its components, and confirms expectations regarding research applications and research partner sites as detailed in the NIJ solicitation. We begin with a deep--brief background on NIJ’s past evaluation research on problem-solving courts under the Courts Research Portfolio. Those past studies laid the groundwork for a cross-site evaluation plan, as does the information from the field on VTC program needs and priorities for the Department of Justice and for BJA. We will highlight key information for solicitation applicants on the basic research tasks--process, impact, and cost-efficiency evaluation, and key outcomes of interest for VTC programs.

Based on past experience, we determined that, one, research partner sites will be identified and recruited separately, apart from the research application review process, and two, eligibility to participate as a research partner site is not limited to VTCs that have received BJA program funds or other federal funding. We hope this webinar is informative, not only to potential research solicitation applicants, but also to VTC programs interested in participating as research partner sites. Our presentation is prerecorded, so we include basic guidance on the research application process, and where both applicants and potential sites can find more information and submit any questions.

NIJ has a rich catalog for courts research funded over the agency’s 50 years, as DOJ’s primary source of independent research and development as it relates to the criminal justice system. Topics range from court operations and case processing, like pretrial and remote hearing concerns, to prosecution and indigent defense services, criminal adjudication, and sentencing. We recently released an article highlighting NIJ’s courts research over the past decade that examine policies and practices designed to promote alternatives to incarceration, which is referenced in the solicitation. The article profiles the most recent research on problem-solving courts in multisite process and implementation evaluation of eight veterans treatment courts in three southern states. The purpose of that project was to gather basic information from a range of programs to better understand policies and procedures, participant backgrounds, and other important information before launching more rigorous research. In the process, concerns regarding program capacity to provide reliable information and fully participate in the research were reinforced.

Prior problem-solving court research includes NIJ’s Multisite Adult Drug Court Evaluation and NIJ’s Evaluation of Adult Reentry Courts funded under the Second Chance Act. In contrast, those studies entail impact and cost studies. Those research teams collected information on participants and comparison groups to understand not only short and long-term outcomes, but for whom those programs were effective, and what were the net benefits applying rigorous, statistical analysis and cost estimation techniques. These projects and our current research plans align with NIJ’s courts strategic research plan. That supports four priorities: to develop the court’s workforce and enhance court workgroups, advance court practice, support the fair and impartial administration of justice, and to promote data and research capacity building. Greg will next provide some background from BJA’s perspective. Greg?

GREGORY TORAIN: BJA is proud to have supported the drug court field from the very beginning of the drug court movement in the early 1990s. We established a drug court discretionary grant program in 1995, we funded one of the first treatment courts specific to the needs of veterans in Buffalo, New York in 2009, started by the Honorable Judge Robert Russell. And in 2013, BJA began receiving a separate appropriation to support veteran treatment courts, and in 2020, Congress passed the Veteran Treatment Court Coordination Act of 2019, which directed the Attorney General to establish and carry out the Veterans Treatment Court program, which is currently being managed by BJA. Today, BJA funds approximately 73 of veteran treatment courts across the country. This year, we’ll receive an appropriation of $25 million for veteran treatment courts. This funding will be used to support new and operational veteran treatment courts, as well as offer free training, technical assistance to those veteran treatment courts.

BJA will continue to engage our federal and national partners in strategies to assess the needs of the field and to ensure the ongoing planning, development, and sustain veteran treatment courts continue. The Attorney General’s priorities are clear: to increase access to justice and to treatment, increase the use of alternatives to incarceration, combat the misuse of stimulants and opioids, and prevent overdoses, and to ensure equitable inclusion in the management of drug courts and veteran treatment courts. Also to support strategy and expand access to services for veterans, support key needs in areas of high poverty, including tribal and rural communities, and advance the Department’s commitment to advance the work that promote civil rights, increase access to justice, support crime victims, protect the public from crime and evolving threats, and build trust between law enforcement and the community. Our goal for this project is pretty simple. We want to use this information or lessons learned from this project to help us guide around our policies around the Veterans Treatment Court program. Return back over to Linda.

LINDA TRUITT: Thank you, Greg. Next, let’s turn to the NIJ research solicitation. I’m going to cover a lot of information and all of the details in this solicitation, but I think it bears pointing out some highlights. So first, just an overview, NIJ seeks applications for funding to conduct a multisite process, impact, and cost-efficiency evaluation of veterans treatment courts, and we’ll refer to those as VTCs. NIJ will make one $3 million award for a five-year cooperative agreement that will start in July. The awardee will lead a cross-site evaluation in partnership with up to six large VTCs across the U.S. The research protocol will incorporate experimental design and other rigorous research methods. The project begins with a six-month planning phase to confirm the research protocols, including site selection. So in preparation for the cross-site evaluation that will begin in 2022, NIJ and BJA are now identifying potential research partner sites. The final six will be confirmed by the grantee during an initial six-month planning phase. As noted, research partner sites will be identified and recruited separately, apart from the research application review process. Therefore, we request that NIJ research applicants do not nominate VTC sites or include program letters of support with their application.

Next, I’m going to highlight the research questions, which shown here, are pretty straightforward. Program impact on relapse, recidivism, and other outcomes, for whom are programs effective, which program elements are effective, whether programs are cost-efficient. Also, topics of interest include veterans with violent offense history, mental health and comorbidity, leveraging technology, peer mentoring, and equitable service access. And just a caveat, we learned in the preliminary research that some programs admit active military service members.

There are two project phases, starting with the planning phase, that is six months. In that time, the researcher team will finalize protocol, including site selection, site agreements, and IRB or Internal Review Board approval. Second phase, going 54 months, is to complete site recruitment, and as footnoted here, our expectation is that recruitment of all six sites will be completed within the first year of the project, and that there’s replacement available up until the second year, so as needed. And then after that, no more replacement. They will complete site recruitments, lead the cross-site evaluation, and those include research tasks, analysis, data and written deliverables, and dissemination.

A rigorous impact and cost-efficiency evaluation requires equivalent information for both the participants admitted to the program and the comparison group. Please review the reports and other publications based on NIJ’s prior problem-solving court research. In this scenario, the comparison group should be similar to the program participants--persons arrested or convicted from the program eligibility regarding military service, charge type restrictions, extralegal issues such as underlying causes of current behavior like mental health issues, and a nexus or connection between their issues and military service. The difference should not be that one group is higher in risk or needs or that standard community service and health services are available to program participants, but not the comparison group. The test is, how do similar populations fare as a result of VTC program participation, all other things held equal? Research analysis plans should not compare graduates to nongraduates of the program. That would be like assessing a school based only on the students who graduate. That is, all participants admitted should be examined in measuring impact and cost-efficiency.

The objectives of this research are three tasks: the process evaluation, the impact evaluation, and the cost-efficiency evaluation. As noted here, and again in the solicitation, the process includes logic models, caseflow, services provided, procedures to examine retention and compliance, leveraging technologies, peer mentoring, and equitable service access. The impact evaluation examines the effects on public safety we’re looking for arrest, violation, conviction, and incarceration, as well as other outcomes, alcohol and other drug use, mental health, and housing stability. Third is the cost-efficiency evaluation. As you know, examine public investment and net benefits using primary information collections, secondary data resources, multiple outcomes of interest, and findings from the process and the impact evaluation. And again, as underscored here, equivalent information on both the comparison groups and the programs.

These appear in the solicitation, I’m just going to highlight them. It’s what the application panel will be looking for at NIJ, and should be reflected or must be reflected in your application. State research questions that correspond to the methods proposed. Detail the qualitative and the quantitative methods, how will they be implemented, how will the information be compiled and analyzed. Detail the interview protocol, including the baseline and follow-up interviews. The expectation is at least 75% response rate. Also bioassay, which would be an objective alcohol and other drug tests are expected. Describe the protocol and budget options should field research plans need to be modified due to public health concerns. As we well know, different agencies, prosecutor offices, courts, and others sometimes need to modify access. So what are the options? Describe potential archival data resources, and how validity and reliability concerns will be addressed. Do not assume, what information lies there or its quality, tell us how you’re going to address it and what is the backup plan. Important here, demonstrate cultural competence in both the protocols and the team capabilities. Address geographic, racial/ethnic, language, and other diversity issues.

The purpose of this solicitation is to create a rigorous cross-site evaluation led by the research team in partnership with up to six large VTCs across the U.S. Large means caseloads of 40 or more participants. We acknowledge that many VTCs are smaller, but the programs must be stable and able to participate fully in the research tasks, which require large samples of participants to conduct comparisons and other analyses. NIJ is compiling information on potential sites in collaboration with BJA and other federal agencies, and contacting the program to gauge their interest and resources. Note the six research partner sites may have received program funds from BJA, but that’s not a criterion for selection into this study. The VTC sites will be selected by the awardee during the planning phase in coordination with NIJ. NIJ will provide a list of potential VTC sites for consideration, but the awardee may propose other sites for discussion at the project startup meeting. Each site will name, one, a coordinator or other designated VTC program representative, and two, a designated research staff representing the state or local agency responsible for court administration. Sites will be required to participate in cross-site meetings organized by the awardee, and facilitate local planning and research tasks.

Activities include contributing to and reviewing research protocol, data analyses, written deliverables, and dissemination. Site information collected for research purposes will remain confidential, and that’s ensured with a privacy certificate that stipulates. Sites must follow the final cross-site evaluation protocol. They’re encouraged to collaborate with the awardee and other sites on complementary research tasks. So there’s room in addition to the core tasks for other to be proposed. The initial NIJ grant award budget includes federal support to each site for their participation in the research project, including $25,000 in each of the first two years. The awardee will be responsible for administering those funds. This may be supplemented subject to availability of funds during the research phase, such as to support complementary research as proposed. Next, Courtney will discuss some of the questions we anticipate from the sites. Courtney.

COURTNEY STEWART: We have developed questions and answers regarding participation in the study similar to FAQs. Question one, are there any restrictions on the federal support funds that the research partner sites will receive for participating in the study? VTCs that volunteer to participate in the study will receive--will receive federal support funds in the amount of $25,000 per year for the first two years of the study. The funding is authorized to support research support and services including data collection and technologies, and administered under the NIJ grant. Question two, how will the research partner sites be selected? The NIJ grantee will confirm the final six research partner sites during the planning phase in coordination with NIJ and its federal partners. Eligibility will be based on variables including total number of program participants, 40 or more, past and current stability of program operations, and the program’s avail--availability to fully participate in the study. Question three, what happens if a research partner site decides to drop out of the study? Participation in the study is voluntary. There is no penalty if a research partner site chooses to drop out of the study, or is unable to maintain the partnership for reasons beyond their control. NIJ, BJA, and the cross evaluation research team will make every effort to address any concerns as they arise. Question four, how long does the IRB process take? IRBs generally meet monthly, so the basic research protocols, including consent and instruments, may be approved relatively quickly once those are finalized by the team. This may vary depending upon agencies involved at each research partner site and their review requirements as VTC programs are recruited or replaced. And question five, will technical assistance be available to research partner sites during the cross-site evaluation? Research partner sites can request technical assistance at any time from the BJA National Drug Court Training and Technical Assistance Program. For more details, please note the website on the presentation slide. Next, Linda will add some final notes about research application guidance and resources.

LINDA TRUITT: Thank you, Courtney. This is the last segment. And I’m just going to cover some of the basic information to pay attention to in your application and also as a potential site. Regarding the budget, as we described, there’s two phases, planning for six months, research for 54 months. NIJ will award it as $100,000 to the research grant for planning. And then the rest will be released upon submission of a revised evaluation plan. Of the 2.9, for the research phase, each site will be paid the $25K in each of the first two years of the NIJ grant. The application should have a budget worksheet and narrative that describes each phase. So in JustGrants, the application has online information and that’s standard. In addition, we just need a breakdown of the first six months for $100k versus 54 months research for $2.9. The research budget may be supplemented in future fiscal years as we alluded to, but please do not include any new tasks proposed in your application budget, that is, do not include tasks beyond the $3 million total.

I mentioned deliverables, and that’s another term we use for NIJ requirements or work products. The solicitation itemizes these, I’m just going to list them here. These are required, as described, the monthly project update calls, that is with NIJ, will include BJA. Project startup meeting, that’s a convening with our federal colleagues to review the plans, to review the potential research sites, and so on. The revised evaluation plan is submitted upon everything being set up from the planning phase, and launching the research phase. The draft abstract, executive summary, and report are NIJ requirements by their standard. There’s an NIJ briefing, that’s NIJ leadership with other feds invited representing the different federal colleagues. And then a final abstract, executive summary, and research reports that are publicly archived. There’s also professional conference presentations, the expectation is a minimum of two that will be coordinated with NIJ and BJA during the project life. And then also NIJ requirements include all desensitized data resulting from all data collection, and all the associated instruments and other information. And finally, we encourage the nomination of dissemination suggestions. So other than these NIJ work products being publicly archived, what other opportunities are there to get information out on the project during its lifetime?

The application should include the project management plan, that’s for each research team member, the staff, the contractor, the consultant, etc. to be specified clearly including their projected level of effort, regardless of their compensation, and also a timeline. So we have a 60-month project period, beginning in July. You need to indicate the key milestones including the planning and research phases, and necessary steps to produce the grant deliverables as outlined here.

As the research teams prepare their applications, it’s important to put yourself in the shoes of the application review panel and NIJ in making funding recommendations. So the guidance I offer is the application review criteria. Here’s a breakdown of four scored sections. There’s a consensus score process and consensus comments. The problem statement section, you’ll see described in detail in the solicitation, has a value of 15% versus the other two sections on research design and capabilities are 50% and 25%. Highlighted here because that is where your attention should be, and the pages dedicated in your application to those sections relatively. Potential impact includes dissemination. So again, we encourage proposals for innovative and other effective dissemination techniques. And then finally, budget is something that is assessed, but it’s not scored.

Please register appropriately, correctly by the timeline shown here. There’s two-stage process, registration at Grants.Gov by January 13, and then your application via JustGrants by January 27. My suggestion is to apply early and be able to do any troubleshooting necessary. Please label files and upload in the correct sections. So I referenced the project management plan, there’s a document that should be called that and a place for it. Itemize your appendices, like a table of contents would be very welcome, and then combine similar documents. If you have multiple resumes, which you should, in this scenario, it’s very helpful to concatenate them, include them in a single file as opposed to many resume files. And then the last here is please document your application submission, contents, and the timestamp. So if there’s any question, you’re able to share that information and we can go from there.

So to assure a fair and open competition, NIJ and other federal staff are not permitted to have individual conversations with prospective applicants. Veterans treatment courts who are interested in the cross-site evaluation, and potentially participating as a research partner site, are encouraged to contact us using a special email account dedicated to this project. And that is shown here, NIJVTC, one word, @usdoj.gov. NIJ research applicants can contact NIJ contractor at NCJRS, shown here. But any questions about the solicitation and any new information will be posted on the solicitation web page, such as the link to this webinar when it’s archived for access. We hope that the solicitation and webinar provide enough basic information for potential research partner sites to contact us by email at the address here, [email protected]. We’re interested to move forward on this important research and look forward to discussions with VTC programs who are interested in participating. We anticipate the NIJ research award will be announced in Spring 2022 for a start date of July. Research partner site recruitment will continue through the planning phase with the new research team. So let me please allow my colleagues, Greg and Courtney, if you’d like to join me in any closing remarks.

GREGORY TORAIN: Yes. Thanks, Linda. First, I want to say that I’m looking forward to the applications and good luck in the process. BJA is really excited about this project. Again, our goal is to be able to use the lessons learned from this research project to again, guide the policies around our Veterans Treatment Court program. So again, I want to say good luck. And again, we look forward to your applications.

COURTNEY STEWART: We’d just like to thank everybody for their time and attention in this project. And hopefully, you will do well with your applications. Thank you.

LINDA TRUITT: Thank you, Greg and Courtney. Thank you for your ongoing contributions. I really appreciate working with you on this project. Thanks to BJA and our colleagues at the U.S. Department of Veterans Affairs and the Substance Abuse and Mental Health Services Administration for their ongoing support of this project. Good luck, everyone, and take care.