NIJ FY25 Research and Evaluation of Emerging Technology Implementation and Impact for Law Enforcement Purposes and Research and Evaluation of Artificial Intelligence for Criminal Justice Purposes NOFOs

Albert Irion: Good afternoon and thank you for joining us today. As you may already know, the National Institute of Justice is a component of the Office of Justice Programs and is the research, evaluation and technology agency of the US Department of Justice. NIJ is dedicated to using science to improve knowledge and understanding of criminal justice issues that affect communities throughout the country.

Today's webinar, NIJ FY25 Research and Evaluation of Emerging Technology Implementation and Impact for Law Enforcement Purposes NOFO and Research and Evaluation of Artificial Intelligence for Criminal Justice Purposes NOFO will be presented by Jill Barnas, Mark Greene and Steve Schuetz. They will discuss the NOFOs and answer any questions about the application process. Before getting started, we would like to go through a few housekeeping items.

Please note that today's webinar will be recorded. If you are using a computer and need to adjust your audio setting, at the bottom of your screen, click on the down arrow unmute button to adjust as needed. If you are still having an issue, within this icon, you can request that WebEx call your phone directly. 

You can submit your questions at any time, which will be answered at the end during a Q&A session. To submit your questions, at the bottom right of your screen, click the three dots and select Q&A. Be sure to send your question to all panelists. Closed captioning is available. The closed caption button will be visible in the bottom left corner. Now, let's turn things over to Mark. 

Mark Greene: Thanks, AJ, and welcome to all of our attendees to this webinar today. Looks like we have a pretty good attendance, and so I'm very delighted to be able to talk about these two notice of funding opportunities that NIJ has published. So, today's webinar, AJ covered the welcome. I'm going to give an overview of the NOFO. Jill and Steve are going to talk about the goals and objectives of the two specific NOFOs.

And then after they talk about the artificial intelligence and the technology implementation NOFOs, I'm going to come back and talk a little bit about the application submission process review and some of the other things that you'll find within the NOFO document. And then at the end, we'll be doing a Q&A session online.

In the meantime, you can type in any questions you have into the Q&A box over on your right. And we can start reviewing those as they come in and hopefully be prepared to answer them. So, AJ, if you could just go to the next slide and I'll give some overviews of the two.

We typically, when we do webinars for our NOFOs, we do sort of one webinar per NOFO. This one, we combined these two NOFOs into one for this year. One, because they are sort of thematically very similar, in many respects, in terms of what it is, the research questions we're asking.

Secondly, we're on a much quicker timeline than we typically are for those of you who may be familiar with NIJ and our grant processes or may have been, current or prior, previous grantee. Typically, our NOFOs are out on the street for a little bit longer. So, both of these were published on May 14th and they're going to close on June 15th. So, we're only out for 30 days. And if you have not taken a look, you can find links to both of these NOFOs on Grants.gov. You can also find them on NIJ.gov, which is our home page, our website.

There's a tab, a bar at the top, and there's a funding tab. If you go to current opportunities and scroll down, you'll find both of those. And you can download the NOFOs both as a PDF document and read through them. Both of the NOFOs are about, you know, there's, there's about 38, 39 pages of content, and the table of contents of both of these, I just want to, just hopefully address maybe some questions ahead of time.

You'll find some basic information about the NOFOs overall concept, and the dates at the front of the NOFO. We have a section on eligibility. Typically, the eligibility to apply to our grant opportunities is pretty broad. Generally speaking, the only type of entity that cannot apply to our grants would be foreign organizations, and typically individuals.

Then we talk about the program description. In both of those, talking about kind of this is the sort of the research questions that both Jill and Steve will talk about. And then we have a section on what needs to be in the application itself, what the submission requirements are and the deadlines.

I'll talk a little bit more about this kind of later in the webinar. Both of these NOFOs are a little unique for us this year. Many of the NOFOs that you'll see on NIJ's website request application or submission of full applications. And that's how we have operated in years past.

Both of these NOFOs are actually sort of a two-phase process. We start off with a concept paper, which is described in the NOFO, which is a very short document that will I think open up the opportunity to many, many folks who may not typically be NIJ applicants.

And then the best of those concept papers will be invited for a full application later on in the year. We do talk about the application review process in the NOFO, what happens if you do get an award and in some of the post-award requirements in the administration of a grant, once you get a grant. There's a lot of great online resources I'll come back to you and kind of point out as well, as we get through the NOFO that you can read about, what to do during the sort of the application process. And in addition, if you do end up becoming a grant recipient of NIJ’s, sort of how to administer your grant if you're not familiar with how to do that.

And then there's some other information at the end and a checklist of important things to make sure that you provide with your application. So, without further ado, I am joined on this webinar by two of my team members. We are both, we are all from NIJ's Office of Technology and Standards.

NIJ has five science offices and my office, which I'm the office director of, we cover areas of criminal justice and law enforcement technology, performance standards for equipment, conformity assessment of protective equipment, and anything that involves sort of technology and standards within the criminal justice system. So, AJ, if you could move to the next slide and I will pass it over to Jill Barnas on my team and she can talk about the artificial intelligence NOFO.

Jill Barnas: Hi, everyone. My name is Jill Barnas, and like Mark said, I'm going to walk us through the artificial intelligence NOFO. Recent changes and AI capabilities are finding their way into many products and services, including those designed for, marketed to and used by criminal justice agencies. This program seeks to build on the prior NIJ efforts to improve the effectiveness of criminal justice processes through AI applications to enhance crime prevention, public safety, and decision making. NIJ expects to fund studies that explore and examine the suitability of using AI to solve operational problems, as well as evaluate the impacts and outcomes when integrated into criminal justice practice. AJ, please go to the next slide please. 

So, there's one goal and objective for this NOFO. The goal is to generate new knowledge that enhances the effectiveness of AI in criminal justice, and addresses potential risks by supporting research that advances the understanding and application of AI in criminal justice, including projects that explore innovative uses of AI, evaluate existing systems, or develop methodologies for testing and improving AI technologies.

And the objective of this NOFO is to support the development of innovative research projects that explore new applications or methodologies for AI in combatting violent crime, immigration enforcement, and other law enforcement activities, including the evaluation of existing AI technologies in practice, advancing knowledge and practices in the field. And with that, I will turn it over to Steve to talk about the technology NOFO.

Steve Schuetz: Next slide, AJ. Hi, I'm Steve Schuetz. I'm a senior physical scientist at NIJ. And I'm going to talk about the 25 research and evaluation of emerging technology implementation and impact for law enforcement purposes. Many law enforcement agencies continue to grapple with fundamental questions of whether new technologies have led to better outcomes, what the actual returns on investments are and whether the new capabilities have been deployed in the best manner.

NIJ expects to fund research and evaluation of technology implementations to guide the law enforcement community on the best ways to implement technology to solve contemporary problems in policing and understand the factors that contribute to successful technology implementations. Next slide, AJ.

NIJ welcomes research and evaluation on the growing use of small-unmanned aircraft systems or drones, such as drone as first responders capabilities. NIJ also welcomes research and evaluation of the use of artificial intelligence, such as the effectiveness of integrating advanced threat detection technology with existing security cameras in public areas and other efforts to retrofit existing security infrastructure.

Next slide, AJ. So, there's one goal with two objectives in this NOFO. The goal is to create rigorous evaluations of technology implementations for law enforcement purposes and generate new knowledge that provides practical evidence to law enforcement practitioner audiences to inform policy and practices. First objective is to support the development of innovative research projects that rigorously evaluates technology implementations for law enforcement purposes.

The second is to generate studies that examine technology as a solution to other operational challenges, such as recruitment, retention, and the training of sworn and nonsworn personnel or other staffing issues.

And I guess I will give it back to Mark. Please advance to the next slide.

Mark Greene: All right. Thank you Steve and Jill. All right. So now I'm going to talk a little bit for the remainder of the webinar more about process. So, this is sort of the application process. And I think, I think many potential applicants or those interested in our funding opportunities probably come to the table with varying degrees of understanding about sort of the grant process.

One of the things you'll want to do, ahead of time, is register on SAM.gov. And to activate that registration status. We do have information, in the NOFO about how to do that. What is required. If you are attending this webinar and you are from a large organization like a research university, for example, you probably have an Office of Sponsored Research or something akin to that, that handles that and has probably already done that.

If you're a small organization that has not received grants from the federal government, I would recommend reviewing that section and registering as early as you can. Go ahead and advance to the next slide AJ.

All right. So, I want to talk a little bit about the application submission process. Both of these NOFOs are a little bit different than our other NOFOs. We have divided it up into two phases. The first phase is a concept paper, and the second phase is submission of a full application upon invitation.

Those of you who may have applied for funding with other federal departments, for example, the Department of Defense or Department of Homeland Security, Department of Energy, some others. You may be familiar with multi-phase application process, starting with something called a quad chart and then a white paper and then a full proposal.

Our concept paper kind of sits in the middle between a quad chart and a and a white paper. What we want to do is allow, I think a greater applicant pool to be able to access the funds available on both of these NOFOs. We've described in the NOFO an NIJ internal merit review process. We will be reviewing the concept papers internally with federal staff, in accordance with a rubric we describe in the NOFO. And then, the best scoring applications from that step will be invited to submit a full proposal, which will be reviewed by an external, be merit reviewed by external subject matter experts, followed by award decisions later on in the year.

Go ahead and advance it to the next one AJ.

All right. So, the submission of the concept paper. First step is to get into Grants.gov, and get in there at least 48 hours before the deadline of the NOFO. Our deadline for both of these is June 15th at 11:59 p.m., so get in a little bit earlier than both of those.

I think that's a Monday. So, I would probably recommend the week before at the latest in case you run into any problems. Register in Grants.gov if you have not already, confirm your roles. And then, of course, very important, in the world of grants is to complete your application for federal assistance, which is a form called the SF-424.

That's the only real paperwork you have to complete at the concept paper stage. And you can complete those really any time starting right now, if you haven't already. And you can also go to Grants.gov. They have a quick start guide for more tips and resources on how to use Grants.gov. So go ahead and advance to the next slide AJ. 

All right. So, we have a system that we use here at the Department of Justice called JustGrants. Applications come from Grants.gov, the 424 and then applicants upload their documents to our system called JustGrants. JustGrants is a system that is used by the Office of Justice Programs, which is a big grant-making component of the Department of Justice as well as the Community Oriented Policing Services component, and also the Office of Violence Against Women (OVW). So, these are the big grant-making components of DOJ and we all use JustGrants. So, this is where you will be able to register and then upload your concept paper package. It's pretty straightforward. It's pretty simple. You can upload it as one document. And I don't think we restricted to either be, they can either be a Word or PDF. We, you know, obviously PDF is very easy, but a Word is fine. We have a cover page with your project title and contact information. Four-page narrative describing the project. We do have some guidelines in the NOFO about that: 12- point font, double spaced, one-inch margins.

Your typical kind of, you know, kind of format. Then one page outlining the applicant's capabilities that includes sort of a biographical sketches of the principal investigators or key personnel, institutional resources. And then a one-page budget sketch. I've been kind of tracking the Q&A and just to sort of address this, we've kept the, you know, the format here pretty open and a little bit agnostic.

You know, so, you know, in terms of a budget sketch, you know, one page, we want you to be able to capture what you think is the important information. One thing that we typically do, well not typically do, but we always do with our grant application process is we go through a process called basic minimum requirements, or BMR, and that is just to make sure that each application has all the required materials.

So, the narrative, the one-page capability statement and then one page budget sketch are our sort of required materials. We do have the one page and the four page and the one- page limit on the other two. If you were to submit something that had, that went beyond those page limits, we would not reject the concept paper for that reason, but we would essentially limit it to those one pages, so we would stop reading after one page for the budget sketch and four pages for the narrative. So just be aware of that. We do have those. We are going to sort of adhere to those strict limits for the concept paper. Okay. Go ahead, AJ. Next slide.

And what happens next? We're talking about that just already. After you submit your application into JustGrants, we will start our review process starting with the basic minimum requirements, responsiveness to the NOFO, you know, whether you're writing a project, a sketch for something that is not responsive to what we're asking.

So AI or technology implementation. And we have an NIJ internal merit review process. We're going to be doing merit review with federal staff inside of the Office of Justice Programs. We kind of outline what that review process looks like. We'll be reviewing each one of the three major components, the narrative, the bio sketch and the budget sketch using a scoring rubric of 0 to 4.

So each one of those will get a score. Each one of those categories will get a score of 0 to 4. So each application will get a total score of 0 to 12 and each one will be reviewed by at least two staff members. Once we have reviewed all the concept papers, the highest scoring ones will be invited for full proposal.

Go ahead to the next slide.

All right, now submission for a full application. So, if you do receive an invitational letter, we do specify that in the NOFO that a letter will be sent inviting you to submit a full application.

That one we have a lot of information in the NOFO. This is typically like our other NOFOs. There's a lot of information you've got to provide, some standard applicant information, abstract proposal, the narrative statement, which is up to 30 pages long. A more detailed budget and a budget worksheet, any memorandum of understanding, other supportive documents, letters of support, tribal authorizing agreements, if that's applicable, and a number of other things that you need to apply. Research integrity, human subjects and privacy. A lot of disclosures and assurances. 

So, a lot of information to keep track of. We do anticipate, hopefully, depending on the number of concept papers we do receive to be able to get invitational letters out by the middle of July. We are hoping, and then we'll have more information in those letters, but all of that is detailed in the NOFO and I think you'll have about 30 days upon receiving an invitation to submit a full application, to get that into JustGrants. You do not need to go back into Grants.gov at that point.

You just go right back into JustGrants and upload the full application. So, unless you are invited, please do not submit a full application. Go ahead and go to the next slide.

So, what happens after that? Again, we'll do basic minimum requirements. And what documents are our basic minimum required documents are outlined in the NOFO. But that's project narrative, budget narrative, CV's of key personnel, updated SF-424, things like that. It's all specified in there. We will do merit review with our external subject matter experts.

Typically, that's 3 or 4 SMEs. I typically like to do four because we are an applied research agency. We typically like to do sort of two researchers, two academics, and two practitioners. There's a programmatic and financial review part, risk review as a part of the review of the full applications. Go ahead to the next slide.

And then, award notices will be made, on the OJP website. There's a section in the NOFO about that. And then funded applications, notifications will be sent through JustGrants. There's a, the system sends out an email to the applicants and if you're a university that receives grants, typically those, those notifications go to an authorized representative.

That's getting a little bit into the weeds down the road. But, that's how the notifications will be sent. You'll have 45 days to accept or decline the awards. And then there's, you'll get a packet of information about that. And then there's a lot of great resources about now what? You know, I have grant, what do I do? We have a lot of great resources about how to manage grants. DOJ financial guide. But again, if you're at a large organization that receives grants, you'll probably have a sponsored research office that will help you with that. All the funding decisions are made at the discretion of the NIJ director.

That is in our statutory authority. That is how it's been since NIJ has been an agency. So that is, NIJ director makes final decisions. AJ, go ahead. Next slide.

And a lot of great resources here to ask questions, we have our response center here. You can also reach out to Grants.gov if you have specific issues with Grants.gov. If there's an outage with Grants.gov, I'm not sure that we can be any of any help, so you'll have to reach out to them directly. But if you do have an issue with JustGrants, you can call us.

We have a phone number and email there about JustGrants if you have technical issues with that one. If you do have questions about the NOFO itself, like what's in the NOFO, those should be directed to the OJP Response Center at the 800 number or the email. And AJ, if you want to go to the next slide.

All right. And we have some great resources again about the funding. Again, the DOJ financial guide, that's more about what you should expect when you get a grant and how to manage that. And then we have some NIJ specific FAQs about our funding. And we have a pretty good page about funding, current funding, expired funding, projects that we've funded.

A lot of good information on the NIJ website. So definitely recommend you guys check that out.

And AJ, I think that probably brings us to the end.

All right. Well, we will start with the Q&A and see if we can get through some of the Q&A here.

Okay. So, question here. How detailed does the budget have to be in the concept paper stage, the first step. Can it be in narrative form with general funding categories anticipated? For example, most funds will be used for personnel travel or does NIJ want a rough estimate with a budget table and a brief justification?

We keep it sort of open for folks to provide what they think is important. But certainly, I think we would want to have kind of a rough order of magnitude, and what the funds would be used for. So if you think that it's going to be, you know, certain amount of money and it's going to be mostly for personnel, kind of let us know that and sort of what a staffing might look like I would say, but we kind of keep it kind of open for you guys to tell us what you think would work for a project. And I hope that that answers the question. 

Next question. If the proposal covers both topics, should we submit an application for both, or how do you suggest we select which to apply for?

It's a great question. I think that, you know, both of the NOFOs kind of have their own distinct research questions. So, both concept papers would need to be responsive to those. I know that we are requesting or including AI into the technology implementation one. So I think that as long as you are responsive to both of those NOFOs with your application, and of course, you can modify it, you know. 

Certainly, you could apply it to both. You know, at the end, of course, we always de-conflict and we don't want to fund a project twice. So, I would not expect at the end of this process that should you be invited for a full application that you would get duplicative funding. So, if they're very, very similar you know, you certainly could apply to both.

But, you know, we have to sort of de-conflict those, kind of, as we move forward.

All right. Next question. Are the AI applications you want audited machine learning based or are they expert systems? Machine learning based applications are non-deterministic replicating output that can sometimes be challenging. Well, we kind of keep it open, open to interpretation and not want to limit any idea. So, I think again, we're an applied research agency and very much interested in kind of research to practice so long as it is responsive to the topic in the NOFO and I think, I think that it would be, you know, we welcome those applications, but we didn't want to be too overly descriptive in terms of what we want.

This is a research NOFO. These are grants. This is not a sort of contract for services type of activity, so it's not an acquisition. So, we want to just kind of keep it open. All right, moving on. These are great questions. I appreciate it. Keep them coming. For the concept paper, how detailed does the budget need to be?

Do we submit a budget table and justification or just a budget narrative for the rough estimate of the anticipated cost? Hopefully I answered that one before. You know, I think, you know, some discussion of what the budget, you know, what the sort of overall level of efforts, you know, and what funds would be spent and why, and trying to capture that in one page.

Do you have to submit an SF-424 for each grant? Yes.

All right, I’m Dr. Ellis Zornek, White House National Security, United States Director of Civil Division, and my nomination is. Okay. I'll have to come back to that question. I don't know if that's responsive to the NOFO, but I would have to come back to that question. 

What is the expected timeline for the review of concept papers prior to the issuance of invitations for proposal submission? Great question. We're hoping that by the time that phase one closes on June 15th that we can have invitations out by July 15th.

That that is our anticipated deadline. We hope. Should we get, receive many concept papers, it may take us a little more time, so we'll see.

Okay. For university applicants, how detailed should the concept paper budget sketch be? Should it include graduate research assistance stipend, tuition, fringe, indirect cost? Or is a higher-level summary sufficient? You know, I would say that the spirit of this is that it's probably a higher-level summary. I mean, you can certainly don't want to suggest you don't include information.

But it's not necessarily a detailed, detailed budget that you would need to have, you know, all the costs, articulated. We understand that a fuller budget would take a little bit more time. So, the idea is to just get a sense of, again, level of effort, what the funds would be used for.

You know, cost, general cost categories. You know, that that I think is, is, you know, and that it supports, you know, that the budget itself is supportive of what you're describing in the narrative. You know, certainly you don’t want to talk about something in the narrative and then, you know, describe something in the budget that doesn't sort of comport.

You want to make sure that those two things kind of match each other, that the level of effort is commensurate with what you're proposing in the narrative. Great question.

Could NIJ clarify how the required cost benefit and cost effectiveness analysis be weighted relative to the core emerging technology implementation objectives? Because rigorous economic analyzes can significantly increase project scope and cost. Additional guidance on the expected level of rigor would be helpful. Great question. Not sure I have a straightforward answer for you. I think that if you are going to propose a cost benefit or cost effectiveness analysis and focus heavily on that, I think that you should describe that the budget should also describe accurately what that will cost. The budget sketch. I don't know, Steve. What do you, what are your thoughts on that question? It's a great question. 

Steve Schuetz: Well, on the concept, it's not broken apart. That would come into play more in the full proposal if we've asked for that, for that applicant.

And then in there they obviously have a lot more pages to go into the more detail, right, of what that would require. 

Mark Greene: I think that's a really good answer. I mean, I think that if you can sketch it out in the concept paper and then if you're invited for a full application, you've got a lot more real estate to kind of, you know, really, if you like, unpack that and then and kind of itemize some of the cost associated with that.

It's a great question.

Do you have an idea of approximately how many invitees for full submissions will be sent? That's a great question. And let me just by answering that question, let me talk to you, talk a little bit about sort of, you know, why I think we're exploring the concept paper in these two. You know, if you look at the number of NOFOs that NIJ has out right now, how many we have, and if you're familiar with NIJ and our grants in the past, you know, we published 15 to 25 NOFOs a year or so, something like that. Depending on the year, you know, we will often have NOFOs that have maybe $2 million available, $3 million available. 

You know, the reality is, with a lot or $1 million available, 1.5, you know, the reality is, is that we end up having, you know, just because of how much a research grant costs, you know, we may fund 2 or 3 for each NOFO and we receive 30 or 50 or 100 or in some cases, 150 applications, full applications that applicants need to put together. And when you've only got enough funds to fund two or 3 or 4 grants, you guys can do the math. That's not great odds in terms of getting an award. 

So, you know, the idea is to offer the field, you guys, the potential applicants, an opportunity to submit a concept paper with sort of a smaller, less level of effort, so that we can review that and then invite I think a healthy number of invitations, so that we can explore sort of the best concepts and not burden everybody with, you know, having to put out, put together a full proposal, knowing that, you know, the odds aren’t great to get a grant to begin with. So, you know, we would typically probably have, you know, maybe something like three times the number of applications, or three times the number of grants that we would offer.

So, you know, that that kind of gives you a sense of, of how many we would invite maybe 3 to 4 times that.

So, so that's, you know, that's kind of the idea. You know, it's also a considerable amount of effort to do full external peer review. You know of full applications. And if we're trying to, you know, review dozens and dozens and dozens of full applications, you know, that gets time consuming. It's expensive and it can delay things. So, we're trying to find this kind of happy medium in terms of process about, you know, allowing as much accessibility while kind of reducing the overall sort of effort at the front end. And then when we get to the full proposals, you know, then I think, you know, those invitees will know, hey, this is serious, and they're very interested.

You know, and this is something that we have a better shot of getting. So that's the idea. I hope that answers that question. Once registered in SAM, do we register with Grants.gov or only after concept papers has been submitted? An invitation for full response has been received. So, register in SAM, like, first, early and often.

Register with Grants.gov before the concept paper. Hopefully that information is, we can we'll have this webinar information online afterwards at some point shortly. But it's also in the NOFO. You'll be going through Grants.gov to submit your concept paper in JustGrants. So, make sure you register in both ahead of time.

Okay. Good question. What distinguishes a strong concept paper from a merely eligible one? What do reviewers weigh most heavily? Well, that's a great question. And I don't know that there's an easy answer to that. You know, we have the content in the programmatic program specific content in the NOFO. Again, we're, you know, an applied research agency and our goal is to conduct and fund research and evaluation to support the criminal justice system and it's functioning. So, I think that it's up to the submitters to make the strong case that this is an important thing, this is an important concept to explore, this is needed, and why. So, I think in some ways it's up to the applicant community to put together a strong concept and convince us, I think. Are the subjects. 

Okay. Next question. Are the subjects in which we propose audits open ended? For instance, can I suggest an audit of your systems for tracking crypto finances? Current solution is inadequate. 

I'm not really sure that I know what you mean by open ended. I guess I could interpret that as we have a very sort of, we're not drilling down I think super specifically. I mean, I think we've highlighted a couple of things we're interested in, but, you know, those are just, you know, for examples.

I think if you have a compelling argument, if you have a topic that is compelling, that has a criminal justice nexus and it is very important, and you can describe why, I think that, you know, submit a concept paper and let's see what happens. I hope that answers the question.

I like all the questions. As the questions come in, it kind of pushes my window a little bit. So just give me a moment. What is the maximum time frame for the project? That's a great question. Saw Jill answered that question. Yep. We typically have a five-year period of performance. Overall, we usually start with a shorter period of performance.

But, you know, we can go out to five years.

If applying with a local state implementation site partner, would you encourage they apply for the BJA awards as well for implementation or personnel staffing? I can't really answer that question. I certainly would encourage anybody to apply for any grants that they feel that they, you know, would make sense for their organization. You know, I think that if you were, if the implication is that they should apply for a BJA grant, and that that would be a part of the NIJ project, you know, I think, you know, that would be something that you could explore. But, you know, there's a lot of contingencies. Like, if I do this, then this happens. So, I can't really answer that question adequately. I would say just, you know, you'd have to, you know, put together a strong concept paper, you know, and we'll see where it goes.

Is there a cap on indirect costs in the budget? That's a good question. Indirect costs to address for a grant organization. And I am just going to, don't hold me to this. The DOJ financial guide talks about indirect cost rates. So, if you're talking about sort of finalizing a budget on a grant, that's a really great resource to go check.

I will tell you, just generally speaking, universities, research organizations will have a federally negotiated indirect cost rate agreement that is recognized by all grant-making organizations or other departments if you're applying for a contract or something like that. That's typically recognized and you'll have a cognizant agency does that. Sometimes it's Department of Energy, sometimes it's Department of Homeland Security, DoD, and that will have a indirect cost rate that is kind of set in stone, so to speak. I think in the grant regulations, they're all, if you don't have that you can use a de minimis rate, which is like 10%. But that's again specified in the DOJ financial guide and also articulated in 2 CFR 200.
We call that part 200. That's the grant-making regulations. So, I think you can either have an indirect cost rate agreement or use de minimis. And if neither of those work out, our financial team, you know, you can ask questions of them as well, but I would check the DOJ financial guide for clear guidance on indirect cost rate agreements.

All right. We are a police department and we’d like to partner with university’s researcher. We are looking into regional DFR, we committed to purchase DFR components? Yes. Those are eligible things to do. In fact, one of the things that we are encouraging, is, if you're going to do a technology implementation and I assume the question is for the technology implementation, is that we would like, we very much welcome concept papers that take technology and ultimately get them into the hands of the practitioner agencies at the end of the project. So implement an evaluation, you know, implement, evaluate and then transition.

Are there any examples of previous concepts online? More for clarification of how detailed the information should be or to remain within the page restrictions? Unfortunately, there aren't. NIJ used to do concept papers a long, long time ago. I think, Steve, it's been 20 years. 20 years. So I would say we don't really have a good example of that online.

We're sort of trying this out again. Trying it again for the first time after many, many years. So unfortunately, I don't, but we do keep it kind of open, and agnostic so that, you know, you guys can, be creative. What is the anticipated budget range for funded research projects?

That's a great question. We do have some language in the NOFO about that. We do have a cap on the total amount and the total awards. We do not have necessarily like we are looking for this specific amount. I think we want to have projects and budgets that are congruent with another one. Well, what does that mean? I mean, if you have, something small that's like 250k that you're like, this would be great. We can just knock this out in a year. I mean, we welcome that. If you're like, I want the whole enchilada and this is, I want multi-site. All this, you know, to do a bunch of things, you know, there's nothing that prohibits you from trying both.

So, we're keeping it as open as possible. So, I don't want to narrow this to like well, our average grant award is, our median award is. Like, we're not. We're just like, we want projects and budgets that are commensurate with each other. How many proposals do you anticipate being funded?

Like I said, you know, typically, with any NOFO we, you guys can, even, you know, you can confirm this on our website. If you go look at expired funding and past NOFOs, you will see, you know, grants that have been funded, you know, we'll typically fund 2 or 3 or 4 per NOFO.

So, you guys can get a sense of what we've funded, how much they are. You know, not to say that that's exactly how it goes. You know, your mileage may vary going forward, but that can give you at least a sense. All right. We're making good. We're making good progress. And the questions are great.

Next question. The research period is 60 months. Does the budget sketch include a year over year budget? Is the 2.5 million budget for all five years, i.e. 500 per year total. So that's a great question. The total, yeah. I'd say we you know, if you in the budget sketch, you know, you want to, you know, propose well it'll take me three years to do this or it'll take four years or five years. You know, talking about how much per year and what that's. You know and what the money goes towards. I think would be welcome to see that. To the best of your ability to kind of sketch that out. But that total amount is sort of the total amount, you know.

So, we would not, although on some occasions we do supplement grants, we would not expect to be supplementing funds on any of these. These would be like, this is my grant proposal, I proposed X amount of dollars, and it's going to take three and a half years to do it. Tell us why. And that's what it would be.

Why have two NOFOs? What is the biggest distinction between the two? Well, the one is very much focused on artificial intelligence. The other one is more broadly, technology implementation, for new and emerging technologies in the criminal justice and law enforcement areas. So, those are the two distinctions.

Okay. Next question. In our concept paper, can we propose alternative solutions to problem sets such as a custom solution or alternative technology, or is this limited to purely to an audit? I don't think it's limited purely to an audit, I think that you could propose something. I think that the difference here, just give like a little bit of guidance is this is a grant we anticipate awarding through grants and not a contract.

So, this would not, there should not be any expectation that NIJ will take a solution and that NIJ at the end of the day will be sort of like an app store and be like a delivery mechanism for something. So I think if you wanted to propose a custom solution or alternative technology, you know, working with a law enforcement agency, you know, I think an important part of that would be sort of explaining, you know, what happens once the grant is over.

You know, talk about the durability of that solution. I think would be, you know, probably in order getting Steve in that kind of. Yes. Yeah. Sounds like with both practitioners and researchers on the expert panel, would you say that the proposal should be written for a broader audience understanding non-technical. Very technical. No, I mean, the practitioners that we get are our experts. These are people that have been doing this for years. Some of them are very. Pracademics.

Yeah. Like Steve mentioned, many of them are pracademics because somewhere they straddle the world of practice and academia. Many have PhDs of their own, so I don't think that you should pull punches at all. I think what is important is to explain. Explain what you're trying to do, why it's important. Do that clearly, technical or not, and I think that's the most important thing. But I wouldn't, I would say, you know, you can be as technical as you want. Just don't be so technical that nobody knows what you're talking about.

Are there any technological innovations AI uses that funders are not interested in funding? Best I can say is that, you know, what we have in the NOFO is the best guidance you're going to get on that. Again, we're an applied agency. We're interested in sort of research to practice. So, make sure that, you know, make sure that the concept papers you submit kind of support that kind of mission.

At the end of the day, we really want to see, solutions, you know, that are useful for the criminal justice and law enforcement community and that can be fielded. So that's really so I think, you know, making sure that things have an interest, or not interest that are, they're relevant to sort of policy and practice in the US. We are a police department, and we would like to, oh same question. Yep. Which is a position to the researcher. Are we permitted to purchase the DFR for components in addition to the researcher. I think if you, I'm going to. The best I can interpret that question is, technology purchases are permitted.

We do like to see technology transitioned into the field. If you mean permitted to purchase the different components in addition to the researcher using a potential grant funds. I think you guys would just explain. You'd have to just explain that to us. But I don't see anything in that question that would say there would be any prohibition on that.

To clarify, can we go with our NICRA assuming we have one? Is that correct?

Give me one second on that question.

Yeah. I mean, I yeah, that's I mean, that would be something that would be recognized. I mean, that would be your indirect cost rate agreement. If you're trying to suggest how much indirect costs would be associated. Yeah, I think your NICRA would be fine. Oh, geez. Questions just jumped up. Yeah, they jump.

Okay. Question. How can a graduate in criminology and criminal justice position himself to contribute effectively to NIJ-related research, forensic or investigative projects?

Well, I think read the NOFOs, and see if there's any that, you know, you think you would be, you would have an idea to respond to. I think that was a pretty general question. So I think you just have to look at our NOFOs and see if there's something you're interested in, in submitting an application to.

What are some common reasons why concept papers are rejected? Well, we have, this is sort of a concept paper pilot. Both of these, I think that the concept papers would be rejected for the reasons that full proposals are rejected for the same exact reasons. Maybe not a strong argument, maybe not communicated well. No relevance to the field.

You know, perhaps, you know, an idea that has already been explored and proven not useful. And oftentimes, perhaps, you know, just, I think sort of a, you know, we want strong, strong arguments, strong, convincing arguments, to be presented to us. So I don't think there's anything special about the concept paper. I think it's all the same idea as the full proposal just shorter format.

Could NIJ clarify whether equipment and technology purchases are allowable under the solicitation, and if so, whether there are any limitations or caps on equipment related costs within the proposed budget? It's a great question, and I'll tell you why it's a great question. No, there's not a. Technology, equipment and technology purchases are allowable under the solicitation. However, those of you who may pay attention to both NIJ solicitations or NOFOs and BJA NOFOs is we are a research agency.

And so for a technology implementation NOFO, for example, we would expect there to be very serious and rigorous sort of evaluation, like a technology implementation plan and a rigorous evaluation in addition to the technology purchases, as opposed to a BJA project where that may be just like they're providing funds to purchase equipment, and there may not be much in the order of an evaluation, you know, perhaps a brief evaluation of, you know, does it work well or something like that, but nothing rigorous.

So, I would say that's a balance. You're going to have to balance the technology purchase with the ability to evaluate that technology in practice. Hopefully that answered the question.

All right. So. What opportunity skills certification professional that will gain from this project.
Will. I'm gonna have to come back to that one. I'm not sure that's, I'm not sure I can answer those questions in the context of the NOFOs. 

Okay. Next one checking understanding. If we have a single site and a more ambitious multi-site with slightly different objectives, we could consider submitting both. I think if both are responsive to the NOFO, then yes, we would not reject any. I would not recommend. We would not ever discourage anyone from submitting more than one NOFO multiple, more than one concept paper. More than one concept paper can be submitted, so long as they are distinct and different.

Could you reiterate the NIJ internal merit review again? You mentioned something like a total of zero, right? Right, right. Okay. So each, each of the three components, the narrative, the budget sketch and the bio sketch will be scored, each one will be scored on a scale from 0 to 4. And then the sum of that will give you the total score for that application. So, you will get a score of 0 to 4 for narrative, 0 to 4 for budget sketch, 0 to 4 for bio sketch or capabilities, called the bio sketch with capabilities and competencies. So, you would have a total overall score of 0 to 12, and then we'll have at least to two reviews per concept paper.

So, your overall score would be summing those. Those categories will not be weighted. They'll just be 0 to 4 for each.

Would AI as it relates to radio frequencies and/or signals intelligence be applicable? Well, I'm not going to say it's not applicable. But you should explain in your concept paper why it is and why it's important. So, any concept, you know, so long as it's responsive to the NOFO, I'm going to say is fair game.

But you'll have to explain why it is, why it's important and why we should fund it. Okay. 

Question. Right. What level of detail in AI architecture would be recommended? Well, I'm not sure I can fully answer that question.

Like I said, I think maybe for the above one, if it's germane and relevant to the argument, I will leave it to you to describe the level of detail of your AI architecture that you think is important, to convince us that it's a relevant project problem to the field and that that it’s worthy of funding.

Last question. We're coming up on time, folks. You're doing great with your questions. 

And, I get to this last couple here. Could a university submit a proposal to work with data from a police department that DOJ considers not in compliance with 8 U.S.C. 1373 if there is no pass-through funding to that police department. I would probably need to defer an answer to that question because I'm not sure of the answer to that. That's a very specific question. I appreciate the question. I, I don't want to give you an incorrect answer. 

Doing good? Hang on one second, I'll make a just a quick, let me make an attempt to try to see if I can give, you know, prohibits, to retrieve information. I think there's something in the NOFOs about it. Oh, is there? Yeah. I think it's spelled out in the NOFOs. Okay, let me. Appreciate the question. If you guys can just hang with me for one more second. It's. So, in the NOFO that is addressed. So, we talk about, so in the. Yes. In section the program description, there you'll see a section called unallowable uses of funds. This is addressing that last question. 

So, it says the following are certain unallowable costs and certain activities that are out of program scope and will not be funded. So one of those is out of program scope is any program or activity at any tier that directly or indirectly violates or promotes or facilitates the violation of federal immigration law, including 8 U.S.C. 17, section 1373, or impedes or hinders the enforcement of federal immigration law, including by failing to comply with 8 U.S.C. section 1373, give access to DHS agents, or honor DHS requests and provide requested notice to DHS agents. 

Here is what I would recommend if you, to the requester, to the question asker. Send your question to the OJP.Response Center email. It's in the NOFO. That will get routed to us, and we can address that question. I do not want to give you incorrect information about that though, but it is at least addressed in the NOFO to some extent and hopefully we'll find you a more specific answer to your question. So send that to the OJP Response Center, please.

I think we have come to the end of our questions. Yeah. I really appreciate everybody attending today. Hopefully you have gotten something out of it. I'm hopeful that we have been able to answer your questions quickly on the fly to give you some guidance and some insights as to sort of what we're looking for, what the review process will look like, and certainly we encourage everybody if you're interested, to apply. We've tried to make this with the concept papers as easy as possible for everybody to, you know, submit ideas to us. Let us know. And we appreciate everybody's interest. And look forward to reviewing some really great concepts.

So, with that, we really thank you for your time and your attention, and AJ, I'm gonna hand it back over to you. Close us out.
All right. 

Albert Irion: Thank you all for joining today's webinar. This concludes the session. Everybody, have a great day.