Webinar Transcript: NIJ FY 2024 Research and Evaluation on School Safety

This webinar provided information on the NIJ FY 2024 Research and Evaluation on School Safety solicitation. In collaboration with BJA, this solicitation will seek applications for rigorous research and evaluation projects to fill knowledge gaps in two topical areas:

1. Studies on the root causes and consequences of school violence.
2. Examinations of the impact and effectiveness of school safety approaches implemented for purposes authorized under the STOP School Violence Act.

Transcript

STACY LEE: Good afternoon, and thanks for joining us today for the “Fiscal Year 2024 Research and Evaluation on School Safety Solicitation Webinar.” It's my pleasure to introduce Dr. Jennifer Grotpeter, the Social Science Research Analyst at the National Institute of Justice.

DR. JENNIFER GROTPETER: Thank you, Stacy. Welcome, everyone, and thank you all for joining today's webinar. My name is Jen Grotpeter and I'm one of the Social Science Research Analysts here at NIJ and I'm working specifically on the School Safety Portfolio. For those of you who are not familiar with NIJ, we are the research, development and evaluation agency of the Department of Justice. We issued a solicitation with two main topical areas for School Safety and it was released on February 20 of this year. The solicitation closes on May 21 with an interim May 7 Grants.gov web deadline. We'll go over what those two deadlines mean shortly. 

For the purpose of today's webinar, we're going to first go over the Fiscal Year 24 Solicitation for Research and Evaluation on School Safety, and the application and review process. We will also refer to two other NIJ solicitations that are related. We'll go over some common issues and critiques that we see during the peer review process, we'll provide some tips, and answer some frequently asked questions. Finally, we will identify places to access additional resources before moving on to the question and answers space, including answers to the questions that we received in advance. And now, I'm going to hand over the microphone to Dr. Nancy La Vigne, Director of the National Institute of Justice. Nancy has been incredibly supportive of this portfolio of research. And I'm so pleased to welcome her for opening remarks.

DR. NANCY LA VIGNE: Thank you so much for that nice introduction and welcome everyone to the webinar on our FY '24 Research and Evaluation on School Safety solicitation. If you've joined in on this webinar, clearly you're interested in or have conducted research on this topic in the past. It's one of NIJ's biggest all-time investments, thanks to Congress who, following the tragedy in Newtown, Connecticut, initiated the Comprehensive School Safety Initiative, which included a robust source of funding for research that continued for several years...and actually continues on today through the Student, Teachers, and Officers Preventing School Violence Act, the STOP Act of 2018. 

We also benefit from a close partnership with the Bureau of Justice Assistance, our sister agency, as well as the Office of Juvenile Justice Delinquency and Prevention who, as you can imagine, has a strong interest in school safety. 

This solicitation really represents a lot of what we've already learned on the topic of school safety, input from our partners throughout the Department of Justice, and listening to the field. As much as we've invested a lot and learned a lot, it's amazing to really contemplate how much we still need to know. So Jennifer is going to be describing in more detail the components of the solicitation, but suffice it to say, the big takeaway here is that we're eager for actionable information. Even for the first category that's looking more at causes and correlates, we're seeking inquiries that really can lead us to understanding what the levers are that can be pulled to prevent school violence and to promote safer school communities. Of course, evaluation comes into play as well. We have a strong focus on evaluating interventions that can help inform the field about what the best measures are in promoting school safety. 

As you're listening to the webinar, you're going to have some questions. Hopefully, you'll get a better sense of what makes for a winning proposal. I just wanted to emphasize some of the priorities that we bring to all of NIJ's solicitations, Priorities that aren't topic-specific, per se, but are more about how we'd like to see the research get done. One key priority is what I call inclusive research, and that's that we really want to invite proposals that include at least some component that engages with the people who are closest to the issue or the problem on their study. So in the case of school safety, if you are looking at an intervention in the K-12 school system, learning from administrators and teachers and students and parents could be a very important component of the proposal. 

The second priority is that we feel that all research that has to do with issues of safety and justice be conducted through what I call a racial equity lens, or an equity lens writ large. There are a lot of ways to think about equity, but certainly in the context of the criminal justice system and issues of, say, the school to prison pipeline, there are a lot of reasons why we need to be considering issues around police presence, and safety, justice in the context of whether they're perpetuating or imposing biases. An equity lens is also about how our own research and data we use, and our research methodologies, may have biases baked into them, so that we can be more intentional about that and mindful as we develop and conduct our research. 

I already mentioned our focus on evaluation research. When it comes to evaluation, I want to be very clear and draw out one component of this solicitation and all others that pertain to evaluation, and that is that we don't want to fund impact evaluations that don't have an implementation evaluation component to it. Some people call them implementation evaluations, others refer to them as process evaluations. What I'm getting at here is to ensure that you are measuring not just whether an intervention worked, but how it was implemented and the fidelity of that implementation. We think that that's vital information that should accompany any evaluation research. 

Now, I recognize that not everybody sets out to conduct research having an array of both qualitative and quantitative experiences, and that's where we really encourage you to team up. We're big proponents at NIJ of interdisciplinary research teams. We think that drawing on the strengths of different disciplines and different methodologies really makes for strong research. So we hope that you'll consider ways to team up so that you can both conduct the research as you wish but also ensure that the research is inclusive, ensure that your evaluation has both that strong design from a quantitative perspective as well as a qualitative perspective on the implementation and process piece to make for more rigorous and more nuanced research. It's my belief that the combination of rigor and nuance is what makes findings better positioned to be communicated to the people who can make changes on the ground. We're talking about the educators, the school administrators, the local law enforcement, the school resource officers - fill in the blank, to get the research into the hands of the people who can best use it to make change and promote safety. That requires taking this kind of mixed methods approach. 

Toward that end, another priority of ours is a dissemination priority. We're inviting proposals that think creatively about how to get the word out there and to reach those audiences, going beyond the academic journal article writing, which of course we understand is a priority for many of you and absolutely could be included in your budget. We're encouraging people to think about ways to disseminate to practitioner audiences and to do things that are even more novel than getting in a practitioner journal, maybe it's doing a webinar or a podcast, or being more intentional at the conferences that you go to and how you communicate your findings. For that reason, we are prioritizing proposals that dedicate at least 15 percent of their budget towards dissemination activities.

Now, I know I shared a lot, but it's all right there in the solicitation. Jen's going to provide you a lot more detail on the specifics of what we're looking for topically as well as some tips and tricks on how to put your best foot forward. I want to thank you, again, for your interest in this program and we look forward to seeing your application. Now, I'm going to turn the mic back over to Jen. Thank you.

DR. JENNIFER GROTPETER: Thank you so much, Nancy. I so appreciate you taking the time to be on this today. The Fiscal Year 2024 Research and Evaluation on School Safety has two topical areas, one is on the study of root causes and consequences of various forms of school violence in K-12 schools. More specifically, we're interested in receiving proposals that use rigorous methodology to identify root causes and consequences in a variety of forms of school perpetration and victimization of violence. But particularly, those we are interested in are those stemming from structural inequality and systemic, and institutional biases. The proposed studies should demonstrate how they will build on complement existing knowledge in this topic and avoid duplication with ongoing research efforts and aid in the development of interventions and strategies to prevent school violence. So those, you can search on the NIJ website and you can find what is currently under study and what has been studied and reported on in the solicitation for this.

We're particularly, as Nancy just noted, interested in identifying malleable factors, mediators and/or moderators of school violence, and give preference to proposals that have a high likelihood of identifying actionable result for programs, policies, and practice. So if you're proposing to examine a risk or protective factor that doesn’t seem obviously malleable, it's incumbent upon the applicant to make the case for how results can be used to become actionable. Applicants should provide a clear explanation of the specific forms of school violence to be studied and they may include studies of more serious school violence such as those involving weapons, but they may also include non-physical forms of aggression and bullying in school. 

The second topical area references purposes under the STOP School Violence Act. It's conducting a rigorous evaluation of the effectiveness of school safety approaches implemented for purposes authorized under STOP School Violence Act, that is a mouthful. But the STOP School Violence Act of 2018 calls for projects that would support a variety of school safety activities and resources. This solicitation complements that act by seeking applications for rigorous evaluation of the effectiveness of school safety approaches addressing those purpose areas. We strongly encourage you to access to the links to STOP and programs funded by BJA, the Bureau of Justice Assistance and the Office of Juvenile Justice and Delinquency Prevention, OJJDP, as well as the Office of Community Oriented Policing services, the COPS Office. There are links also to these in the solicitation. 

Within the range of approaches and activities supported by STOP School Violence Act, NIJ is particularly interested in examining several understudied approaches and they are enumerated in the solicitation. And these include programs and practices involving school police or SROs, it involves new technology, virtual reality enhanced prevention program, comprehensive or multicomponent school safety approaches, and then, we've just had a kind of a catch all category, other common and understudied approaches. That is not an exhaustive list. That fourth bullet allows you to propose other strategies as long as you make the case that they are implemented for the purposes authorized under the STOP School Violence Act and as long as the case is made that is an understudied approach. This could be in sheer quantity of prior studies or in rigor of prior studies, or something else. 

To be clear, the applications do not need to be evaluations of school safety approaches that are receiving funding under the STOP School Violence Act from BJA, OJJDP, or the COPS Office. You may, but you can also propose evaluation of school safety programs that received funding from other sources, but nonetheless align with the activity in category as defines STOP School Violence Act. Anyway, I also want to mention what Nancy said as well already and that is an outcome impact evaluation must also include a process or implementation evaluation component. 

Now I'm going to mention a couple more specific things about the scope of this solicitation. Under the solicitation proposals can include incidents outside of the school setting. For example, offsite school sponsored extracurricular activities, travel to and from the school and home, online, but also must demonstrate the relevance of the incidents to a school setting. It can be something like travel to and from school and home. It can be online like cyberbullying, but it must demonstrate the relevance of the instance to a school setting. What will not be funded under the solicitation includes applications to purchase equipment or material supplies, et cetera, to provide direct services and applications that are not responsive to the solicitations, but that probably makes sense. 

There are two related solicitations that I'd like to point out to you as well, so that you can disambiguate and figure out where your idea fits best. One is under the NIJ FY24 Research and Evaluation on Firearms, Violence and Mass Shootings. Topic area two is evaluation of school-based campaigns for safe storage of firearms. That is one, and there's a research topic there as well. so schools are in scope in terms of researching mass shootings in schools. Then under the NIJ FY24 Research on School-Based Hate Crimes, that's fairly self-explanatory. This addresses research on school-based bias motivated incidents. So if your topic area better fits there, then that's a place to—at least read it and consider whether it's a good fit for your idea. 

If your award is funded, there are several required deliverables. You'll have the standard grant reporting requirements, such as the semi-annual progress reports and the quarterly financial reports. Additionally any recipient of the award under this solicitation will be expected to submit a final research report at the end of the award period and provide a draft 90 days before the end of the award project period. Award recipients will also be expected to submit to the National Archive of Criminal Justice data. All data that result in whole or in part from the work funded by this award, along with associated files, any documentation necessary for future efforts by others to reproduce the project's findings and to extend the scientific value of the data set through secondary analyses. And in addition to these deliverables, we expect scholarly products that result from the award under the solicitation, taking the form of peer reviewed scientific journal articles or other appropriate products as are listed in the solicitations. NIJ expects that there will be an equal effort made or more to make their research findings accessible to practitioner and policy maker audiences. 

Next, there are several elements of an application that must be submitted for the application to even be considered. And if even one of those elements is missing, the application will not move on to the peer review process. So the following elements must be included in the application to meet the Basic Minimum Requirements in advance to peer review. And those are the forms SF-424 and SF-LLL. Those are in Grants.gov. You have to have a Program Narrative. It has to include a Budget Web-Based Form, budget details and narrative, and actually the solicitation for this one does not include Financial Management and Study of Internal Control questionnaires. Just double check the solicitation to make sure that that's a requirement. 

Another one that's really critical that sometimes people miss is curricula vitae or resumes for all key personnel. That includes investigators, co-investigators, principal investigators, co-principal investigators, whether or not you're at the prime site of the applicant or you're a subcontractor, if there is somebody you have named as key personnel, they need to have a resume or CV included at the time of application; that can't come later. 

Additional requirements are many others that accompany an application for full consideration. And so I've listed a few here, Project Abstract, Letters of Support tools, instruments, questionnaires, tables, but more importantly, I think on another slide, I have flagged the pages where the full explanation and the applicant checklist can be found. 

Next, I want to go through the two-part application process that NIJ has moved to. It requires submitting information in two different systems, each of which have their own deadlines. So first you need to submit the SF-424 and the SF-LLL, about lobbying, in Grants.gov. And the deadline for this is May 7 at 11:59 p.m. Eastern Time. Second, you'll submit your full application, including all application attachments in JustGrants and the deadline is May 21 at 8:59 p.m. Eastern Time, not 11:59, 8:59. Also note not just the end of the day, 8:59. A couple of additional important notes. Applicants must register with Grants.gov and JustGrants prior to submitting. Processing delays up to several weeks can sometimes occur when registering. So this is why we strongly encourage applicants to register several weeks before the application deadline. Even if you're not ready to submit your application yet, please register early. If you do not submit your forms in Grants.gov of by the deadline, your application in JustGrants will not be accepted. 

When submitting your application, we urge applicants to submit at least 72 hours prior to the deadline to allow time for the applicant to receive validation messages or rejection notifications from Grants.gov. And in this way, you'll be able to correct any problems in a timely fashion. And it also will allow you time to verify all materials were uploaded correctly to JustGrants, and I realize that there's probably many people on this call laughing at 72 hours prior to the deadline being finished. But things happen and so the earlier you are ready, the sooner you can get everything submitted, the greater chance that your application will be able to receive full consideration. 

We recommend also that you label your documents and attachments appropriately with the file name indicating what the document is and upload that document and that corresponding section of JustGrants. 

I want to emphasize the deadlines are hard. Third party software to submit has been used and applications have sometimes not made it in on time. These applications cannot move forward for consideration no matter what. We often get questions about whether an applicant can submit an additional piece of the application that was missing from the initial submission or not received, such as a letter of support, but that cannot be done after the deadline. Finally, we also often get requests for extensions or exceptions. NIJ is unable to grant these, unless an application can prove—and let me stress "prove"—that there was a technical error in the NIJ system or software while the applicant was submitting the application that caused the missed deadline, NIJ cannot accept applications after the deadline. Because this is a competitive process, we are unable to grant applicants additional time as it would be unfair to those applicants that submit their application in full and on time. 

In order to prevent some of these issues that we just discussed from arising, applicants are also strongly encouraged to contact the OJP Response Center, Grants.gov or JustGrants support. We've listed the full numbers and email addresses here. These can also be found in the application itself. 

After a solicitation closes, all applications are screened for Basic Minimum Requirements, or BMR. During that process, we assess the application to make sure it's responsive to the solicitation. If you're proposing a research project that responds to at least one of the two areas of focus, and if it's being submitted by an eligible applicant. During BMR, we also check to make sure the application includes those critical elements. Again, those being the project narrative, the budget detail worksheet and narrative, and the CVs and resumes of all key personnel. 

As a reminder, there are other application material that would be included, such as human subjects and privacy documentation, data archiving plan, letters of support project timelines. While these aren't critical elements that would cause an application to fail BMR, failure to include these items may result in a less favorable review or in a delay of releasing funds if awarded. 

Once your application passes BMR, it moves on to the External Peer Review process. The external peer review panels are made up of researchers and practitioners in the field relevant to the solicitation. They score the applications and they discuss them providing NIJ with both the merits and concerns of the proposed research. The review criteria can be found in the solicitation, and that indicates the page number 30 on there. NIJ then uses those scores and comments to conduct its own internal review of the applications. The internal review comprises a team of social science and leadership at NIJ, in addition to any other relevant experts who may be able to weigh in on the merits of the application. 

After reviewing the applications, taking into consideration the scores and comments provided by the external peer review panels, as well as the budget, NIJ science staff then make funding recommendations to present to the NIJ director. The NIJ Director then decides which applicants will be awarded funding. It should be noted that all final funding decisions are made at the discretion of the NIJ Director. And in addition to all of the priorities listed in the solicitation, I cannot strongly enough encourage you to read and take to heart the insights of NIJ Director Nancy La Vigne, that she shared in her Dear Colleague letter that was posted last month. I will walk through a few, but not all, of the points made in that letter. 

All things seem equal, NIJ favors proposals that meet our funding priorities, which are that they propose an inclusive research design, that they address issues of race, ethnicity, gender, and other potential disparities in data sources, research methods, outcomes, and that they are well-justified as well in the Statement of the Problem. We also favor proposals that propose a multidisciplinary research team that ensure rigorous measurement of implementation fidelity and that allocate ample resources for the translation and development of research findings. These cannot just be an afterthought. 

To be successful, an application must also achieve two objectives. First, it must propose rigorous research that develops needed knowledge or tools to address the major challenges of safety and justice in the U.S. And second, it equally important, an application must demonstrate that the resulting products have a potential to address those challenges, either directly through policy and practice improvements or indirectly by advancing the body of knowledge. 

Over the years, we've noticed a variety of commonly missed opportunities. This is also specifically noted in the director's Dear Colleague letter. Failure to describe how the applicant will ensure the research integrity, independence, and integrity of evaluation findings when the project team includes project staff. The optics are not good when you are evaluating your own program, so if you are proposing an independent evaluation, then you need to provide measures to mitigate that if someone from the project team is the evaluation team is also part of the project staff. Another commonly missed opportunity is insufficient discussion of the potential risks and harms to individuals or groups associated with the use or misuse of a proposed research practice or technology. Underdeveloped dissemination strategies that failed to make the case that the resulting research information would lead to actionable changes in the field, timelines, staffing plans, and/or budgets that are not aligned with the proposed work. And we're not just talking about, "Oh, you've asked for too much." But maybe you've asked for too little. Is the project feasible? Is the project as proposed with the staff that is listed, actually feasible to be conducted in within that timeline? Inadequate specifications of research questions and/or failure to connect research questions to the research design and evaluation plan is another commonly missed opportunity. And I can't tell you how many times we're sitting in peer review and the reviewers note, "They did not list research questions or hypothesis." Make sure that you list explicit research questions and hypothesis. Another missed opportunity is failure to demonstrate that the most rigorous, feasible research and analysis methods have been proposed. In short, make sure your science is fully buttoned-up. 

Now we're going to move on to common critiques raised during peer review. Within the Statement of the Problem, we see that sometimes the Statement of the Problem fails to identify gaps in the current literature or understanding of current research. The lit review is sometimes insufficient or dated. And the scope of the proposed research may be extremely limited or, conversely, it may be too ambitious. We also see applicants dedicate too much space, too much real estate, to argue for the necessity of the proposed research at the expense of clearly articulating the innovation, rigor, and feasibility of the project design. 

Moving on to the Research Design. Some other common critiques that have been raised. Sometimes we see that the research questions are not derived from the lit review or inadequately specified or identified at all. We see problems with the research design where it's not well articulated or clearly laid out, where research designs or methods don't flow logically from the problem statement. The research questions and lit review are too ambitious, too complex. Alternatively, we'll see applicants fail to demonstrate the proposed research plan is the most rigorous it can be and that it is feasible. So under this solicitation, we're seeking applications for projects with clear potential for producing findings that have practical benefits for improving school safety in order to produce findings with a high potential for influencing school safety and practices. Researchers are strongly encouraged to work closely with educators and other stakeholders within the schools and in the community to develop the right questions, prioritize challenges, identify solutions, collect data, and make sense of the findings. 

Next is more research design critiques raised during peer review. We sometimes see issues with samples, such as the sample size not being supported by a power analysis, or the sampling strategy being too flawed or ambitious. We also see problems with quantitative analysis being vague and unclear or the data collection analysis plan being confusing. If complex predictors, mediators, moderators, and/or outcomes are proposed for the study, so, for example, variables are based on subgroups, such as race, ethnicity, sexual orientation, or gender identity, their measurement should be well-described. If there are limitations in available secondary data, the application must acknowledge those limitations and describe specific mitigation approaches and potential impact on the interpretation of their results. And this one comes up more often than you might think. If multiple secondary datasets are proposed for compilation for analysis, the application must describe how those datasets will be matched, what will be the unit of analysis, specific variables common to each dataset. Don't leave it to us to trust that you know how to put these datasets together. Please explain how they will be linked. 

Other common weaknesses include failure to identify potential research challenges and strategies to mitigate them, and failure to demonstrate access and the ability to use proposed data sources, such as the ability to merge multiple administrative sources, whether or not you actually have the letters of support that indicate you actually can receive those data. That is absolutely critical and something that we oftentimes find missing. Applications also suffer when the timeline, staffing plans, and/or budgets don't support the work proposed. And again, this is not necessarily thinking that the budget is too big, but it could be too small. And you need to request, propose, the budget that really aligns with the work that needs to be done. Another thing to include, applicants proposing to evaluate one or more school safety projects must demonstrate in their application that the projects are sufficiently developed and well-positioned to support the proposed evaluation. So, for example, before a rigorous outcome evaluation can be conducted, the school safety program or activities must have a clearly defined conceptual framework or logic model. NIJ wants to build understanding not only of whether a program is effective, but also what elements within or externals to the program designer implementation contribute to its success or its failure. Bottom line: the innovation; rigor, and feasibility of the research design, must clearly articulate research questions that need to be fully demonstrated, including consideration of risks and appropriate mitigation strategies. 

I should say at this point, I feel like all we're saying is negative things about applications we've received before. A lot of times we receive a lot, we've received some amazing ideas that just maybe fell down in one or more of those research design components. And so that's why we're taking the time to talk about all of these today because you can have a really great idea, but you really have to have that research design be solid. 

Moving on to other categories of critiques raised during peer review: that the PI doesn't demonstrate familiarity or proficiency with the proposed quantitative analysis or with their mixed methods, or that the PI has direct involvement in the program that they're proposing to evaluate. Under potential impact, we've run into situations where the dissemination plans lack specificity or they're just not innovative or there's no plan to reach non-academic audiences. It's important to make a case for how the research results will meaningfully contribute to safety and justice and how that dissemination plan will make the resulting research information actionable. 

So, specifically, Dr. La Vigne via her Dear Colleague letter notes underdeveloped dissemination strategies that fail to make the case that the resulting research information will lead to actionable changes in the field. So you've probably heard some consistent themes today and that is make sure that your science is sound and that the results have a reasonable likelihood of leading to actionable changes in the field. 

Finally, this is really just a quick summary of tips. Proposals should be well-written, innovative, timely, rigorous, well-designed, feasible, impactful, include all your letters of support and consider research and evaluation independence issues. There are a few frequently asked questions that we received and I'm just going to go over a couple of these really quickly. 

Amount of awards and periods of performance. Again, this depends in large part on the quantity and quality of applications we receive. Each solicitation has a maximum amount to be awarded under the whole solicitation. But we have $6 million available for the whole solicitation. But there is no set number of awards that will be made. And there is a variety of ranges that they can fall into, so you just need to make sure that your idea and your budget are consistent and fully justified. 

Another frequently asked question involves notifications of award and non-award. Award announcements are usually made by October 1. Non-award announcements are usually made by the end of the year. And both awarded and non-awarded applicants will be sent their peer review comments as well. There have been some years in the recent past where the award and non-award announcements were delayed, but we don’t anticipate this being an issue moving forward. 

Another often asked question is, “Are U.S. citizens working outside the U.S. eligible either as co-PI or PI? And can any grant funds be paid to institutions outside the U.S.?” Our solicitation states that foreign governments, foreign organizations and foreign colleagues and universities are not eligible to apply. However, the situation with subcontracts and sub-awards may be different. We can get to sub-awards in a minute, but first, for the initial question, we encourage everyone to review the Code of Federal Regulations, particularly to CFR Part 200, which covers various matters regarding foreign organizations or entities, costs incurred in foreign countries, and several other matters that may arise in connection with non-U.S. based entities or individuals that may be funded under a federal grant award. The DOJ Grants Financial Guide also covers policies with regard to foreign travel costs under the award. And generally speaking, a U.S. citizen who’s not otherwise disbarred or suspended from participating under a federal award speaking as a general matter could be funded under the award even if working outside the U.S. However, more information about the work being done, by whom, in what capacity, is unique to every situation. If you have a specific question, please submit it through the NCJRS system so it can be routed to the appropriate person. 

And another question is, “Can grant funds be paid to institutions outside the U.S.?” NIJ solicitation language barring applications from foreign entities would not mean that a U.S.-based applicant could not propose a foreign sub-recipient or submit a proposal under which a U.S. citizen would propose work outside the US. 

All in all, we encourage applicants to familiarize yourselves with the Code of Federal Regulations mentioned and exercise the appropriate cautions when partnering with foreign entities to conduct research. Data archiving, human subjects, and privacy considerations are all major contributing factors as to whether a partnership with foreign entity would be feasible with an NIJ award. 

Again, if you have specific questions, you can please send them in through the OJP Response Center. And the last frequently asked question I'm going to mention is we also receive a lot more general questions around submitting forms, what constitutes a new investigator, and other questions related to privacy and human subjects concerns. For all of these, I encourage you to please submit the questions through the OJP Response Center mentioned on the solicitation and down at the bottom of this slide (Slide 19). Your question will be routed to the most appropriate person who can answer the question. 

And here are more specific websites for the OJP Funding Resource Center, DOJ Grants Financial Guide, and NIJ Funding Frequently Asked Questions. We are now at the end of my long presentation with slides and we welcome your questions. If you can put them in the Q&A box, that would be great. Ben Adams is also here to help wade through questions that have been submitted through Q&A and any that you continue to submit through Q&A. And he also is going to talk about some we received in advance. So I'm going to turn the microphone over to Ben.

BEN ADAMS: Thank you so much, Jen. We will begin with a few questions that we received in advance of this webinar and then proceed through questions in the Q&A box. 

The first question is regarding the data archiving plan. “We're planning to use secondary data that has been collected by an external organization. Based on our data use agreement, the data is restricted and we're not allowed to publicly post it. Would it be possible to note in our plan and mention that we'll post our code files only?” 

I won't speak to this specific proposal, but I will say generally, NIJ asks prospective applicants proposing research involving partnerships to include as part of their letters of support an acknowledgement that de-identified data from the partner will be archived at the National Archive of Criminal Justice Data. If you're involved with a project partner that's providing data, now's a really good time to have that discussion about data archiving for the specific research that you're proposing. I would also recommend sharing with any partners the NIJ data archiving webpage (https://nij.ojp.gov/funding/data-archiving), as well as the NACJD data submission guidelines (https://www.icpsr.umich.edu/web/pages/NACJD/archiving/deposit-nij-ojjdp.html) so your partners can understand how archived data are curated for release and protected. 

Secondly, NIJ has indeed funded secondary data analysis projects that do not involve original data collection. And when the existing data used for analysis is not publicly available, we encourage researchers to deposit that data with permission from the original producer. If justifiable restrictions preclude the deposit of that data, the grantee would still be required to provide what we refer to as a data roadmap to identify the existing data sources, to provide a description of the steps taken to acquire the data, and to submit any other documentation, the code, syntax, and/or set up files, so that other analysts would be able to replicate that process. And again, I'd point you to the NIJ data archiving page for more details. 

A second question that came in, “The solicitation notes that proposed studies must examine data over time. Can proposed studies use repeated cross-sections of data versus data collected on the same participants that are tracked over time?” I think for that one, I'll pass to Jen.

JENNIFER GROTPETER: We have funded panel studies in addition as well as funding longitudinal studies that track data over time. It's better if you can, if you're trying to demonstrate that change, pre and post, to have to follow the same people over time to see if there's change, particularly if you're looking for changes, not just overall in school climate, but particular individual acting and behaving in certain ways in terms of engagement and aggression or violent behaviors. But we have funded some panel designs as well. Ben, do you have a more specific answer than that?

BEN ADAMS: No. That's incredibly helpful, Jen. And again, I would just point back again to the Director’s guidance to present a rigorous and feasible project design appropriate to address the questions of interest. Make sure that you're able to justify why that approach is the most rigorous for supporting those questions. 

One other question we received before the call, and Jen addressed it at the outset of the webinar, deals with evaluations of programs for purposes authorized under the STOP School Violence Act. And the question is, “Do proposals that include program partners that are not funded directly through a STOP School Violence grant award, are those eligible to be evaluated?” Jen shared that yes, they are as long as they align with the purposes authorized under the Act. Jen, I don't know if you want to add anything else to that response.

JENNIFER GROTPETER: No. I think you can look online to see what is being already funded by BJA, by OJJDP or by the COPS office to try to find a partner that's already going. But also, as long as the project aligns with the same purposes as the STOP School Violence Act, you're fine.

BEN ADAMS: Great. Thanks so much, Jen.

BEN ADAMS: A few more questions here. “Is a proper grant writer needed to be considered for review for funding?”

No. To be considered for review, applicants have to meet the eligibility and other requirements outlined today during the webinar. If you're a new prospective applicant to NIJ, I would strongly encourage you to go to the nij.gov website and review the guidance for applicants and awardees. There you can see we've posted a selection of some sample application exemplars. There's the Dear Colleague letter that we mentioned from Director La Vigne that provides insights about what makes a successful research grant application. And of course there is also the OJP Funding Resource Center, and specifically the OJP Grant Application Resource Guide. So there's a wealth of toolkits and resources on nij.gov about how to write a successful application to NIJ. 

Another question. “Is there a listing of potential teams allowing a new member?” 

There is no listing of prospective applicants, but as Jen just mentioned, we recommend potentially reviewing program awards made by BJA, OJJDP, COPS, getting in contact with your networks whether they be community-based organizations, schools, academics, and others. 

Another question. “We have some specific districts and schools with whom we'd like to partner for topic two. Do these schools and districts with whom we want to partner need to be a specific project that's funded under STOP?” 

The answer is no.  

A multi-part question. “First, does NIJ permit multiple PI applications as NIH does?” I will take a shot at that. At the beginning of the solicitation, it states that an applicant may submit more than one application if the application proposes a different project in response to the solicitation. Also, an entity may be a proposed sub-recipient in more than one application. So the answer there is yes. The second part of that question was, “Is there an award ceiling on a per application basis? 

I think Jen answered this before, but the maximum dollar amount for each award is determined by the requirements of the research proposed and the grant selected for award.” So, again, proposed budgets that match the research activities proposed up to the full dollar amount anticipated awarded under the solicitation. And Jen mentioned we have $6 million available. 

JENNIFER GROTPETER: Let me just add something to that, and that we do allow multiple PIs, oftentimes people will propose one PI and a co-PI or multiple co-PIs or multiple investigators. Make sure that you have included CVs for everyone for whom you label them some sort of an investigator. Also that you make clear in the management plan who is going to be ultimately in charge of what in that the overall day-to-day of the project or specific subject matter expertise provided, that shall be very specific, especially if you're proposing two PIs versus a PI and a co-PI. Thanks.

BEN ADAMS: All right. Thank you, Jen, for that clarification. The next question asks, “What are letters of support?” 

This is right in the solicitation. Essentially, applicants should include for each named supporting entity a signed letter of support that outlines the supporting entity's reason for supporting the project, and the scope of work they're committing to. It should include a discussion of the relationship between the applicant and the entity, a description of the need and what benefits would be gained from the project, and a description of the applicant's capacity to complete the project. And, where applicable, where the entity is providing data, an acknowledgement that the final de-identified project data is archived at NACJD. So those are the letters of support that would accompany the research proposal. 

Next question. “If a large proportion of the budget goes to data collection. In this example, survey vendors collect large longitudinal data, would that be appropriate for an application?” 

Here applicants are encouraged to propose budgets that match the research activities proposed. The key is justifying your timeline, your staffing plan, your budget, and making sure that they are aligned with the proposed work. 

An easy one. “Will the recorded webinar and associated documents be shared with the participants?” The solicitation, slides, and transcripts will be posted on nij.gov. 

A question about new investigators. “Does a new investigator include someone who might have received funding from other DOJ agencies but not NIJ?” 

I'd point you to page 12 of the solicitation which specifies the criteria for being considered an early career investigator. The one dealing with NIJ is specifically having never previously received NIJ funding as a PI on a research project with the exception of graduate research fellows. But there's also degree criteria, completing a terminal degree within the past six years and also being a non-tenured assistant professor at an accredited institution of higher education, or an equivalent full-time staff scientist position at a research institution in the United States. So please take a look at those criteria. 

 “Can you clarify what you mean by the PI cannot have direct involvement in a program they proposed to evaluate? Does this mean the PI cannot be the one that created the program?” Jen, do you want to take a first shot at discussing research independence and integrity a bit?

JENNIFER GROTPETER: Sure. I understand that with community-based participatory action research that a lot of times you're wanting to make sure that the program developer is involved in the process and receiving implementation fidelity data back and working with them. And you'll want to have a relationship with the program staff because you're going to be measuring implementation ability. But we don't want the evaluation PI to be the same person as the one who created the program because that does not support independent research that is independently conducted, that the PI of the evaluation doesn't have a vested interest in how the results come out. Even though we trust you, but we need to actually see that distinction between the evaluator and the program. 

BEN ADAMS: I would just add that applications that propose research and/or evaluation require a research evaluation, independence and integrity statement. They have to demonstrate independence and integrity, including all appropriate safeguards before an award receives funds. That has to do with all applications that come in where any actual or perceived risk to independence and integrity exists. So please review those parts in the solicitation. 

Let's see. There are some questions about letters of support from government entities. There are many different entities could potentially be partners under the solicitation. Again, they could be governmental or non-governmental depending on who the project is working with. 

 “Can teams with new investigators be multi-PI teams or does the junior scholar need to be a sole PI?” 

There can be co-PIs on applications. The early career investigator should be the primary principal investigator for the entity submitting the application.  

“We're a small business who does training and research on this topic, would it be best to have separate project proposals or within one submission?” 

Again, I will suggest that proposals include the most rigorous approaches to answer the research questions that are being articulated in the study. So the answer really depends if you have two distinct ideas. Multiple applications are allowable, but for the purposes of this solicitation, we would point you back to the guidance that we provided in this presentation. I think unless I missed any, those are the questions that have come in so far. 

We have a minute or two left, if there's anything last minute. If we did overlook any questions, we will go back through at the conclusion of this webinar, and certainly can add responses to any questions we missed, so that they'll be included in the transcript. With that, I think I will turn it back to Stacy to close out the webinar.

STACY LEE: Thank you. On behalf of the National Institute of Justice, thank you for joining today's webinar. This will end our presentation.