Webinar Transcript: NIJ FY 24 Evaluation on Desistance

NIJ hosted a webinar providing an overview of the NIJ FY 24 Evaluation on Desistance Solicitation, in which NIJ seeks proposals for rigorous evaluations of desistance-focused interventions to advance understanding of strategies that might aid in the desistance process and identify innovative approaches to measure the desistance process.

Transcript

STACY LEE: Welcome to the Fiscal Year 2024 National Institute of Justice Evaluation on Desistance Webinar. It is my pleasure to introduce Dr. Kyleigh Clark-Moorman, Social Science Research Analyst at the National Institute of Justice.

DR. KYLEIGH CLARK-MOORMAN: Thank you so much, Stacy. Good afternoon, everyone. Thank you so much for joining our webinar today, focused on the Evaluation on Desistance for Fiscal Year 2024. I'm really excited to see this many people interested in this solicitation and I'm excited to get into more specifics about this solicitation, as well as the application process today. Before I do that though, I'd like to hit it over to our Executive Science Advisor, Dr. Janine Zweig, who will provide some comments about the solicitation to open our webinar today. 

DR. JANINE ZWEIG: Good afternoon. Thank you, Kyleigh. Understanding and promoting desistance, the process of individuals ceasing engagement and criminal activities, is a crucial aspect of ensuring public safety. NIJ have a long-standing history of advancing desistance research, including the 2021 publication of a six-chapter volume entitled, Desistance from Crime: Implications for Research Policy and Practice. 

For this solicitation, we're seeking proposals for rigorous evaluations of desistance-focused interventions to advance understanding of strategies that might aid in the desistance process and identify innovative approaches to measure the desistance process. NIJ made the decision to focus specifically on evaluations for this solicitation because the effectiveness of desistance-focused interventions is not well understood due to limitations on existing evaluations. Specifically, few evaluations exist that demonstrate a strong causal link between programs and policies in desistance. Existing evaluations have inconsistently defined and operationalized relevant concepts, resulting in variations in follow-up periods and outcome indicators. Innovative approaches to measure the desistance process in the context of program evaluation are not well understood. The evidence base for the effectiveness of desistance-focused interventions and the mechanisms by which they work is also limited in part because of a focus on measuring desistance primarily through antisocial behaviors. 

Before I turn it back over to Kyleigh to further elaborate on the specifics of this solicitation, I also wanted to mention some of NIJ Director Nancy La Vigne's priorities. One priority is inclusive research. We think it is important for proposals to include in its methodologies and its dissemination activities engagement with the people that are closest to the issue or problem under study. For this solicitation, that could be professionals working with justice-involved individuals, such as probation officers or law enforcement, or it could also be people who have lived experience and expertise. 

We also like to see the consideration of a racial equity lens to identify and mitigate biases in both the topic of study and the research process itself. We are interested in studies that consider biases according to race, gender, ethnicity, sexual orientation, and so forth, which are more pronounced in the justice system. Strong proposals who acknowledge the possibility of biases and proposed methodologies that will enable the detection of those biases and account for them in analysis and the findings as they are presented back. 

For all evaluations, we are looking for applications that infuse research with strong implementation science to ensure documentation of implementation fidelity in addition to intended impacts and unintended consequences. We also encourage interdisciplinary research that includes both quantitative and qualitative analysis. Research practitioner partnerships provide opportunities to include varied perspectives and disciplines in the design and the execution of the research. 

Robust dissemination plans are also encouraged. While scholarly articles and presentations are important, we encourage creative dissemination strategies and approaches to engage with practitioners, impacted communities, and the general public. This works to uphold the priority of ensuring that research evidence is translated into actionable information to promote change in the field. That's just a broad overview of the Director's priorities that cover all of NIJ's solicitations. I'm now going to turn it back to Kyleigh to walk you through the specifics of this particular solicitation. Thank you.

DR. KYLEIGH CLARK-MOORMAN: Thank you so much, Janine. First, I'm going to just talk about the specific information for the solicitation, then the expected deliverables, and then talk a little bit about what we do in the application and review process, give you some resources, and then we'll move to the Q&A. 

As far as the goal and objective for this specific solicitation, the goal, as Janine mentioned, is to advance understanding of strategies that might aid in the desistance process and identify innovative approaches to measure the desistance process. The primary objective of this solicitation is to support rigorous evaluation of desistance-focused interventions to generate and advance knowledge regarding strategies to effectively promote desistance from crime. Rigorous evaluation is what we are looking for this solicitation. In terms of this specific information, which you will find in the solicitation text as well, based on this goal and this objective, we're seeking applications for rigorous evaluations of desistance-focused interventions to advance understanding of strategies that might aid in the desistance process and identify innovative approaches to measure the desistance process. In line with that, applications that do not propose to conduct rigorous impact or outcome evaluations of desistance-focused strategies will not be considered responsive to the solicitation and therefore will not be considered for funding. That means that proposals for formative or process/implementation evaluations alone will not be considered for funding. 

For this solicitation, there's also some specific information that we want you to follow that's specific to this solicitation. First, definitions must conceptualize desistance as a process that occurs over time and may entail continued criminal behavior. Applicants must establish that the intervention to be evaluated is desistance-focused by demonstrating that it affects one or more of the key causal mechanisms of desistance. Applicants may propose to conduct an outcome or impact evaluation of an intervention whose aims extend beyond promoting or accelerating the desistance process, but it still needs to be connected to those theoretical mechanisms of desistance. 

Applicants must also propose to measure both criminal and non-criminal desistance outcomes. For proposed criminal measures like reoffending, applicants must demonstrate that the outcomes to be measured contribute to an understanding of the long-term process of desistance. In line with that, applications proposing to measure criminality using only recidivism will not be funded. And applicants must also specify their definition of desistance in the application, the alignment with the proposed measures, and the feasibility of obtaining and working with proposed data sources that are correspondent to these measures. 

I also want to talk about some specific considerations for evaluation research that are highlighted in the solicitation. If an application includes evaluation research, we expect you to propose the most rigorous evaluation design appropriate for the research questions to be addressed. The most rigorous evaluation design may include random selection and assignment of participants. In cases where randomization is not feasible or is not appropriate for the research questions, then applicants should propose a strong quasi-experimental design that can address the risk of selection bias. 

Applicants proposing evaluation research should also include an implementation evaluation component, and that's to ensure that implementation fidelity metrics are collected and analyzed. Applicants are also encouraged to propose research methods that enable interim feedback on implementation fidelity. 

Finally, NIJ also encourages applicants to consider the feasibility of including a cost/benefit analysis. Again, this is outlined in the solicitation. 

Now I want to move on to some of the other information that you'll see in the solicitation that you would be expected to talk about and plan for. One is that we have several required deliverables. We have standard grant reporting requirements, those are semi-annual progress reports so two progress reports a year, as well as quarterly financial reports. We also require a final research report that must be submitted at least 90 days ahead of the end of the award period. We also expect grantees to submit their datasets or files to the National Archive of Criminal Justice Data or NACJD, our data repository. 

In addition to these deliverables, NIJ also expects scholarly products to result from each award under this solicitation. With that, scholarly products can be the obvious things like journal articles, book chapters, et cetera. But we also expect there will be an equal effort made to make the research findings accessible to practitioner and policymaker audiences. So again, we would expect to see scholarly products geared towards scholarly audiences, as well as products geared toward practitioner and policymaker audiences.

In terms of the application submission, there's a number of elements that you have to submit, which you will see in the solicitation. But there's also critical application elements that must be submitted in order for an application to move on to peer review and be considered for funding. They are the program narrative, the web-based budget form, and that actually includes an opportunity for you to provide both budget details, which are the math of your budget as well as the narrative that explains those budget details, explains the math behind your budget, and then also CVs or resumes for key personnel. So those are your critical application elements. We also have some required forms and those are standard for federal assistance, and that's the SF-424 and SF-LLL. 

We now have a two-step application submission protocol. This entails first registering on Grants.gov, that is where you will submit your SF-424 and your SF-LLL. You must do that by April 30, 2024. If you do not complete step one within the deadline, you cannot move on to step two, so that's very important. Step two is submitting the full application including all of your attachments into the JustGrants system, and that deadline is May 7, 2024. Again, really important dates to remember. If you don’t complete both of those steps, your application will not be advanced and will not be considered for funding. 

In terms of the application review process, we get all the applications and then we determine whether applications meet Basic Minimum Requirements, or what we call BMR. That is just making sure that the program narrative, the budget detail and narrative, and the CVs for key personnel have been submitted and have content in them. From there, all the applications that meet BMR are moved to external expert peer review, where peer reviewers provide scores and comment on the application. Applications are also reviewed by NIJ science staff and leadership, as well as other federal subject matter experts as appropriate. And then finally, all funding decisions are made at the discretion of the NIJ Director. 

Next, I want to go over some common critiques that are raised during the peer review process. As you saw on that last slide, our peer review process is very rigorous. We have a lot of expertise that are looking at these applications. Each application is judged on several categories that hold different weights in the overall scoring. The solicitation has detailed information on what should be included in each section. But I'd like to talk about some common critiques that we see raised during the peer review process. First for the statement of the problem, that's going to account for 10% of the overall score. Some common critiques that we see for that are that it fails to identify gaps in the literature or understanding of current research, the literature review is insufficient, or the scope of the proposed research is either too limited or overly ambitious. 

For project design and implementation, where a lot of the meat of the application comes from, it accounts for the most weight of any section at 45%. Some common critiques are that the overall strategy was not well articulated, the design or methods or the approach does not logically flow or there's a lack of clarity, the proposed sample size is not supported by a power analysis, or that the project is not feasible. In the peer review, we have practitioners as well as researchers or research practitioners so we do consider the feasibility of the project as well. 

Next we have capabilities and competencies and that's worth 20%. We often hear that proposed staff do not demonstrate familiarity or proficiency with the proposed methods or that the project team does not include necessary and appropriate subject matter experts. For potential impact, which is also worth 20%, we also see some common critiques that the dissemination plan lacks specificity or is not innovative or there's no plan for reaching non-academic audiences. So again, going back to what Janine outlined for the important things in dissemination of what we like to see. Then for the budget at 5% of the total score, we often see critiques that there's an insufficient budget or there's insufficient staff time dedicated to proposed activities or that staff duties are not well-articulated. 

In terms of just some general tips about how to write successful applications or what to include in a successful proposal, we encourage you to design proposals that are well-written but also innovative, timely, rigorous and well-designed, feasible, and impactful. We encourage you to consider all of these different components when thinking about your proposal and writing your application. And I know that's really hard. I just outlined a lot of things that each one of them is hard to do on their own, let alone together. But there's a number of resources that you can access online with FAQs and resource guides from OJP that I'll go over at the end of this webinar. I do encourage you to access those resources as well. And then, just another note is that if an application proposes research or evaluation, and all applications for this solicitation should propose evaluations, you must also demonstrate research and evaluation independence and integrity. This can be submitted as an attachment in JustGrants. 

Generally, letters of support are really important for applications. If you have a supporting entity for your project, the application should include a signed letter of support from each supporting entity that outlines the reasons for supporting the project as well as the scope of work that they are coming to. And again, this information is provided in the solicitation as well. 

There are some things that will not be funded and I want to go over those as well. It's very important. Just keep in mind that NIJ will not fund applications that are unresponsive to the solicitation. For this solicitation, that means applications that do not propose rigorous evaluations of desistance-focused strategies or applications that propose measuring criminality using only recidivism. We will also not fund duplicative or similar research. NIJ does not fund applications that are primarily to purchase equipment, materials, or supplies or to provide direct services. However, a budget can include these items if they're necessary to conduct research, development, et cetera but they can't be primarily what the proposal is for. We will also not fund training or direct services that are unrelated to the project. 

Now to go over some common questions that we get. First, the award amount and the period of performance. The total anticipated amount to be awarded under the solicitation is $2 million. We anticipate awarding one to three awards which will obviously be determined by the number of applications and their merit. The period of performance will start at 1/1/2025. The end of the period of performance, so how long it goes, is to be determined by the applicant. But successful applicants are generally expected to complete the work within 60 months or a five-year period of performance. 

For the timing of award and non-award notifications, award notifications are generally made in the fall. New awards are posted to the website upon clearances. So as things are cleared, they will get posted on a rolling basis. At least that's how it was last year, so pay attention to that website. 

Another thing to note is that foreign governments or organizations, foreign colleges and universities are not eligible to apply as a primary institution. Where appropriate, however, a US grantee may sub-contract with a non-US institution or a non-US individual for work that's necessary to complete project task, but they can't be the primary institution on the application. 

Additionally, you can't actually start work until all forms are submitted and approved by the IRB, and also by our Human Subjects Protection Officer. So that's just something to keep in mind as you're preparing everything for your application and then if you get funded, for the eventual funding. 

Let's talk about some of the resources that we have for you (Slide 17). First, we have the OJP Response Center. If you have any questions on the solicitation itself, please contact them either by phone or email. If they do not have the answer for you, they will find someone that does have the answer for you. But that is who you should talk to with all your solicitation-related inquiries. If you have issues or problems or questions about Grants.gov or JustGrants or submitting your documentation, you can contact them as well. Grants.gov's phone number is here as well as their email, same with JustGrants. If there's any technical difficulties, do not hesitate to reach out to them as they are happening because those deadlines that we went over are very firm. Just make sure to reach out if you're experiencing any difficulties. 

I also have several recommended resources here (Slide 18) for you. One of my biggest tips in this process is make sure to pay attention to the solicitation and make sure to look at these resources and think about them as you're crafting your application. One is the OJP Funding Resource Center. The DOJ Grants Financial Guide, that's very important as you're doing your budget and your Budget Narrative and thinking about what can actually be funded. And then, there's also an NIJ Funding FAQ page and that has a number of sections and dropdowns and it's very comprehensive and can answer a lot of questions as well. But again, if you have questions on the solicitation, please contact the OJP Response Center. 

And with that, I am done with the actual presentation and I'm happy to turn it over to questions. I did have a question that already came in, so as my colleagues who will be assisting me in this process are getting their cameras on and getting set up, I'm going to go ahead and read that and answer it. 

The first question that we got via email prior to the webinar was, “The application states that NIJ will only fund projects that evaluate well-developed programs for the solicitation. Please describe what is meant by well-developed.” We are looking for impact or outcome evaluations in this solicitation. We're not going to fund formative or process/implementation evaluations alone as we've noted in the solicitation. Applicants must demonstrate that a program can support rigorous outcome evaluation. 

DR. MARIE GARCIA: Thank you, Kyleigh. Hi, everyone. Good afternoon to those of you on the East Coast and good morning to those of you on the West. My name is Marie Garcia and I'm here to assist Kyleigh with the question-and-answer session of the webinar. We have two questions so far in the Q&A, one that I have answered. We will provide additional information about the webinar, the recording of the webinar at the end of today's session but just to very quickly, our webinar is being recorded. We will make all materials available to you at a later date. 

But the first question is, “The solicitation includes an in quotes ‘identifying innovative measurement approaches’ component, yet the solicitation seems to indicate that proposals are required to involve interventions. Does this mean that a proposal that aims to solely develop and/or validate measures of desistance processes would not be appropriate? Such a measurement-focused proposal would include approaches that would be tailored to measure desistance processes in the context of future desistance-focused interventions.”

DR. KYLEIGH CLARK-MOORMAN: Thank you so much and thank you for the question. In the design of this, we were going for evaluations of desistance-focused interventions. I don't want to preclude anyone from applying. I think that we are looking for applications that fit the solicitation as we've described it. And if you feel like your project could do that, then I encourage you to apply. However, I will say that the focus for this is on the evaluation of desistance-focused interventions. That is a component that should be included.

DR. MARIE GARCIA: Thank you. Our next question, “Does this solicitation fund evaluations implemented outside the US, for example, Latin America?”

DR. KYLEIGH CLARK-MOORMAN: Another great question. There's nothing in the solicitation that would preclude us from evaluating something in a different country. There should be information about how that could apply to the US system that there's specific language in the solicitation for that. However, there's nothing that says that we would not accept applications or fund a project that was evaluating desistance-focused intervention in a different country.

DR. MARIE GARCIA: Great. Thank you. Our next question, “Is there interest in juveniles and desistance?”

DR. KYLEIGH CLARK-MOORMAN: Yes. I think there's an interest in all populations in the system. We are interested in all justice-involved populations.

DR. MARIE GARCIA: ‘Our next question, “Would NIJ consider evaluations of state policy changes on the desistance process so long as the methods the definition of rigorous?”

DR. KYLEIGH CLARK-MOORMAN: As I said, I don't want to preclude anyone from applying. If you feel as though your evaluation fits the solicitation requirements, then I encourage you to apply.

DR. MARIE GARCIA: The next question, “Page 11 discusses the new investigators/early career opportunity. I understand all bullet points must be met. For example, non-tenure terminal degree prior to September 30, '24 never received funding. However, when you say postgrad must have been completed prior to September 30 or postgrad clinical training, does this include if you have a postgrad postdoc? For example, if you finish your postdoctoral fellowship in 2019, would you meet the criteria even though a terminal degree was received in 2017?”

DR. KYLEIGH CLARK-MOORMAN: That is an absolutely wonderful question. And Marie, I'm going to ask you to help me with this one because I'm not a hundred percent sure.

DR. MARIE GARCIA: Not a problem. We need more information about the actual timeline. The postgrad fellowship complicates it just a little bit because you're not in a tenured track position. But you could make the case, for example, that because it is six years, I believe, and it's by 2024. We do have those parameters. I don't want to answer that incorrectly. So if you just direct chat me, Tamara, I will get an answer for you. My initial thought is no, because the degree was completed in 2017, which would not meet our early investigator qualifications, but I don't want to be wrong about this, so please just send me your email and we'll get an answer for you.

**As a follow-up, a key question for potential applicants to consider is whether post-doctoral fellowships qualify as clinical training. If they don’t, the date of the terminal degree should be used to determine if the applicant would qualify as a new investigator. 

DR. KYLEIGH CLARK-MOORMAN: I'm glad it wasn't just me who wasn't sure about that one. 

DR. MARIE GARCIA: It's complicated.

DR. KYLEIGH CLARK-MOORMAN: Yes, it is. I think the most important thing for us is the documentation.

DR. MARIE GARCIA: My next question, “The solicitation states that if we are proposing a partnership with a justice agency, we need a strong letter of support that includes a commitment to provide de-identified data that would be uploaded to NACJD. If we are proposing an evaluation of a community-based intervention that does not include a direct partnership with a justice agency, can we use public records to assess criminal outcomes of desistance? In this instance, we would not need such an MOU, is that correct?”

DR. KYLEIGH CLARK-MOORMAN: We like to see letters of support if you have a supporting entity. So if you do not have a supporting entity, then we would not be looking for a letter of support. As far as the specific measures and the data, the way that you would be getting them, that is all at the applicant's discretion with the goal of designing your project in a way that fits the solicitation requirement. I'm not going to comment specifically on whether those would fit or not, but just making sure that in the application, you're demonstrating how they do fit.

DR. MARIE GARCIA: And Kyleigh, I'll also add for data that are publicly available, you are encouraged to provide information in your application and data archiving agreement stating that the data are accessible. If they are available and anyone in the public can access them, we won't ask you to archive them. But we will ask you to submit the materials and links to the data. And if they are public and you somehow have access and others don't, we would still require the data to be archived. So you might need permission to archive data, even if they're public. That's really important to the archiving process and the data accessibility.

DR. MARIE GARCIA: Okay. “In the analysis of the rigor of the proposal, are there preferences for quantitative research methodologies over qualitative?”

DR. KYLEIGH CLARK-MOORMAN: There is no preference. I encourage you to look at the solicitation, read the requirements for evaluations and go off of what's in there for designing your proposal. But no, I wouldn't say there's a preference.

DR. MARIE GARCIA: Next question. "As you discussed, random assignment is preferred, and if not, a rigorous quasi-experimental design” would be okay. I'm adding those words. “Will proposals be considered if they do not have a control group and if all participants participate in the intervention? Also, what is the preferred follow up period?”

DR. KYLEIGH CLARK-MOORMAN: This again is subjective for the applicants. We encourage you to design the most rigorous evaluation possible and to support your proposed methodology to describe why that is the most rigorous method possible and how that can work toward our goal and our objective in evaluating these desistance focused interventions. It really is up to the person who is designing the research and submitting the application. But I encourage you to pay attention to what is in the solicitation as far as what we're looking for. We leave it up to the applicant to determine those things, but also to justify why they determined those things.

DR. MARIE GARCIA: I would also say when it comes to the follow up period, and it's implied in the solicitation, please make sure that your budget and timeline actually allow for those follow up period timeframes, so that everything is lining up. 

We had a question about the desistance volume that was referenced. I provided a link in the Q&A for that response, so hopefully that helps out. (https://nij.ojp.gov/topics/articles/desistance-crime-implications-research-policy-and-practice)

We have another question for you about data. “Would it be required that qualitative data be made available publicly? In certain instances, there could be ethical concerns with the possibility to identify individuals.”

DR. KYLEIGH CLARK-MOORMAN: Ethics is always something that we're considering when we're doing these data agreements. We have a data officer who is in charge of all the specifics of this that we can interface with in terms of data requirements. If you have something that you feel should not be or cannot be shared online, we can work with those specifics with you. But I'll say that there are exceptions that can be made when data cannot be shared publicly.

DR. MARIE GARCIA: Another data question, “For data that is public but may be timestamped or updated as time goes by, is there a need to mark a specific point in time when that public data was used and/or must we archive a copy of that data from a specific moment in time?”

DR. KYLEIGH CLARK-MOORMAN: That's also a really good question and I think something that could be worked out with the data officer. If you get funding and then you are ramping up to figure out how you're going to be collecting this data and how you need to archive that data and things like that or keep that data and what kind of records you need to keep, all of the specific peculiarities with specific projects can be worked out with the data officer. I don't have an exact answer on that, but I would just encourage you to raise that point in your application. I would raise that point and then plan for that as you're doing your data collection. But again, these are all things that could be worked out with the data officer.

DR. MARIE GARCIA: Also the point of the data archiving process that we requested as part of the application is for you to tell us the specific nuances of your data collection. If something is timestamped or something can't be made public, those are things we want to know at the start so there aren’t surprised years later when you say, "Oh, sorry, I can't archive that," or "My agreement with this agency says that I can't." Those are things we want to know upfront. And it's not going to be a problem for us. All of our research is publicly funded, so we want to make sure some level of information is made accessible to the public. However, if there are nuances and things for us to consider for example confidentiality and privacy, we will work with you on that. So just let us know at the start so that we can work with you to make sure we have the right plan in place for your data. 

Okay. Next question. “If an outcome measure could include self-report of criminal behavior, is it possible to obtain a writ of confidentiality to ensure they don't incriminate themselves and won't be prosecuted?”

DR. KYLEIGH CLARK-MOORMAN: That's a really great question. In my experience that is generally an IRB issue, when you're doing these projects you have to go through your own IRB and then we also have a Human Subjects Protection Officer that reviews all of the IRB materials. I am not actually sure if we would provide something like that or if that's something that you work through with the IRB. Marie, I don't know if you have any experience with something like this.

DR. MARIE GARCIA: So something to consider because we are the Justice Department, we do have really high levels of security and confidentiality for our research, more so than some other agencies and cabinets in the government. We do require a privacy certificate as part of your package. It's not just IRB approval from your university, it's also a privacy certificate with NIJ. Those are issues that we could address in the privacy certificate. We do of course have other mandated reporting issues, but that's a different question. You would not be able to move forward until that agreement and privacy certificate were in place. We want to protect you as well, your data and your subjects. I hope that's helpful. 

To a question that was posed about the desistance volume. It is on NIJ's website. (https://nij.ojp.gov/topics/articles/desistance-crime-implications-research-policy-and-practice) It's available, you can download it chapter by chapter or the full volume, the link is in the chat and solicitation if you're interested. And one quick look at the chat and I think we're all done. Kyleigh, back to you.

DR. KYLEIGH CLARK-MOORMAN: Great. Thank you so much everyone. On behalf of NIJ, I really appreciate you joining today and thank you so much to my colleagues. And Marie, thank you for helping me answer those questions. We really look forward to seeing all your applications. I'm really excited about everything that I'm hearing today and all these questions. And that will conclude today's presentation. Thank you again.