Expanding Research to Examine the Impacts of Forensic Science on the Criminal Justice System

ERIC MARTIN: Hello, everybody. This is Eric Martin. I'm a Social Science Analyst with the National Institute of Justice. I am joined by my colleague Jon McGrath. He is from the Office of Investigative and Forensic Sciences. And we're pleased today to be able to present to you a panel that we originally were intending to present at the 2020 meeting of the American Society of Criminology. As you know, given the current situation with the pandemic, a lot of conferences had been scaled back, but NIJ has decided to leverage technology and our resources to be able to bring you the same information. We are commenting that we think our attendance is — we're able to reach a broader audience than what was able to fit in to the panel room, so we're pretty excited about that. We're co-chairs. We're going to get some comments at the end. But we have three amazing presenters for you all today. Before I get to that, just a disclaimer, you know, we want to make sure that it's clear that the opinions and points of views expressed by all the panelists in the presentation do not reflect any official position or policy in the United States Department of Justice. And of course any errors are our own.

And now that that is out of — we've covered that, we want to get to — for those of you who may not know who we are, the National Institution of Just — National Institute of Justice is the research, development, and evaluation agency for the US Department of Justice. We support criminal justice policy and practice through research, particularly at the state, local, and tribal level. We're going to talk a little bit about NIJ at the end, but just so you know, the panelists are going to be presenting primarily research that was funded in the past by NIJ, and we're excited to be able to bring this to you.

Now, to introduce the panelists we have today, this is going to be in the order that they're going to appear in our panel. First, we have James Anderson. He is the Director of the Justice Policy Program and Institute for Civil Justice at RAND Corporation. He will followed by Kevin Strom. He is the Director of the Center of Policing Research & Investigative Science at RTI International. And then he will be follow by Donia Slack, who's a Research Forensic Scientist also at RTI. But she is the Associate Director of the NIJ Forensic Technology of Excellent — Center of Excellence. So we're pleased that all three of these panelists took time of their busy schedule to participate in today's panel. So now, I am going to turn it over to James Anderson for the first presentation.

JAMES ANDERSON: Thanks, Eric. So, good afternoon. I'm James Anderson from the RAND Corporation. And I'll present our research on forensic science and its efficacy overall. First, I want to thank NIJ for funding this research and my co-authors as well. It was a — it was a terrific multi-disciplinary team. I'm a lawyer by training. Carl Matthies is a Forensic Scientist, Sarah Greathouse is a Forensic Psychologist, and Chari is an Economist. So we had a — it was — it was fun and I think illuminating to have a variety of perspectives on the — on the research team to really try to figure out the effect of forensic science.

So I think I'll talk about a little bit of a theoretical promise of forensic science and, you know, why, you know, why it matters. Really going back to the 19th Century, folks have recognized that forensic science has its tremendous potential. There's this great quotation, "The dark in courtroom, the odd silence, the assembly, the intense mental strain, I know is most deeply interested, the awful force of the blow to the guilt — guilty man when he first beholds the evidence of his crime illuminated by the light of scientific test." This is from Percy Edwards in 1896. And for more than a hundred years, the potential of forensic science to be the aid the criminal justice system has been recognized. And it's also sort of an interesting way in which increasingly other systems, medicine, aviation, industrial processes have really focused on doing various processes as an overall system and on reducing error, making it as robust to human error as possible, and really focusing on the reliability. And James Doyle, for example, has argued that the application of set of concepts to the criminal justice system to increase its reliability. And to do that, you have to abstract a bit and really see the criminal justice system as a — almost as an industrial process that's really just a diagnostic system to really try to identify certain categories or certain steps when crime is committed by a particular person. And there are only sort of two particular types of errors that we worry about: one are wrongful convictions and the other are unsolved cases. And this is obviously a key abstraction of the system. And there are lots of other constraints going on. But it's the least important component of — and I think of any vision of the criminal justice system that we would want it to accomplish those tasks as reliably and as consistently as possible. And one of the great things about forensic science, right, is it's an additional source of information to classify these cases, and a means to make the system more robust inevitable of human errors. It's a — it's a whole new area of potential information that can be gleaned about the world. And on top of sort of more conventional evidence, eyewitness testimony alike.

So when you're thinking about trying to reduce errors in a system, what you really want are sort of uncorrelated errors. Like want — you want things that — and one of the advantages of forensic evidence, of course, is that hopefully it's more subjective and scientific as possible, hopefully it's more reliable than eyewitness testimony or confessions and the like. It's independent of another information, any other should be uncorrelated with other errors. And in the — in the sort of the accident reduction vocabulary, we're talking about loosely coupled systems, so that the process of forensic science can hopefully serve as a check, an independent check on other — on other forms of evidence. 

And it's important to recognize that there are some key assumptions here, and assumptions I'm not going to really go into today. First of all, you know, obviously the evidence has to, in fact, be scientific. And a number of folks have commented, at least in some cases, forensic science, or what's called forensic science is not actually scientific. There are issues about peer review. There are issues about junk forensic science, edited results, well — poorly understood error rates, those lab scandals of course where some people aren't doing what they're supposed to be doing. So there's a — there's a — there are reasons to quibble with without assumption. I'm not going to really talk about that today. But on the whole, at least in theory, there's considerable promise there. The other key assumption I am going to talk a little bit more about today is that the—systems have to be loosely coupled. So if — errors are more likely to occur if systems are tightly coupled than an error in one part of the system can sort of impact other parts of the system. And as — I'll talk about it a little bit, that is a risk in our — the way we currently do forensic science.

So we want to know how — enough about the theory. How is the work in progress? There's — the literature here is not — you know, it's not huge. It's a little sparse. But it's pretty consistent from Parker in 1963, found it was used in less than one percent of cases. RAND did a study in the mid-1970s and found that it was used very little despite its availability in many, many cases. And Peterson more recently has found that, again, it's rarely used. And — but we wanted to do a study of really trying to understand, well, you know, that — that was a while ago. There's been a lot of funding that has — had gone into forensic laboratories, their DNA has revolutionized forensic science. How is it working now? So what do we do? Well, we did — we did a fair amount. We did a mixed-method study in five jurisdictions, Sacramento County, California, Sedgwick County, Kansas, Alleghany County, Pennsylvania, Bear County in Texas, and King County, Washington. In each of those jurisdictions, we did interviews with detectives, forensic labs, prosecutors, and sort of various levels of each institution. We collected a random samples of a thousand reported crimes of various types, two hundred of each type in each jurisdiction, with the hope that we'd be able to really measure the effect of different kinds of forensic science. We also conducted an analysis of national crime lab data using the specific publicly funded crime laboratories from the Bureau of Justice statistics from 2002 to 2005 and 2009. And we conducted an experimental survey of prosecutors and criminal defense attorneys to understand the effect of forensic evidence on plea bargaining which is, of course, is what many of these cases can resolve.

So, the first interesting we found was really when we began our interviews are sort of — we found that the development of forensic evidence wasn't really independent of the development of other evidence. And actually the case status drove the analysis from the development of the forensic evidence. So, it wasn't a situation where we can rely on sort of exogenous creation and the availability of different kinds of forensic evidence to then draw conclusions about how the case would progress. The — we heard again and again that if there was a little other evidence, probably that there would be — that there was very little focus placed on either evidence collection or testing. But if the case is actually going on a trial, there was a lot more analysis conducted. We also collected a fair amount of information about how often forensic evidence is collected and analyzed. And, you know, and here, the — you can see the numbers are probably a little bit small, but — for folks to read. But in homicides, in all of our jurisdictions that are collected barely frequently and tested very frequently. But in rapes, aggravated assaults, robberies, and burglaries, there was less evidence collection and quite a bit less analysis and testing. And the only sort of takeaway that we thought was interesting was that there was, you know, very little — very little standardization and that there were, I mean, apparently no sort of national best practices, even in large fairly sophisticated departments. And, you know, here you see, for example, that in robbery cases in San Antonio, they collected in 15% of cases compared to Sacramento, you had 55% of cases. I mean, some fairly stark differences in the amount of evidence that was — or the frequency with which evidence is collected. We also looked at what stage of analysis did the forensic analysis occur. And here we have, for all the cases, whether it's prior to each of these figures represents the fraction of arrests that lead in trials that are proceeded by request for and completion of forensic analysis.

Again, the story emerges that when there is testing and analysis, it tends to occur relatively late in the process. And very few cases, did it occur prior to arrest, you can see by different categories of analysis, and I should say due to time, I'm really focusing more on the results today. The full workup of the papers is on the RAND website as a — as a working paper, and I urge you to look at there if you're particularly interested in these results. But anyway, but — so you can see that a very, very low numbers prior to arrest, relatively low numbers prior to plea bargaining, and even part of the trial, the number of cases where there's actually testing occur is pretty low. So — and the sort of the takeaways of this analysis is that forensic evidence is rarely analyzed prior to an arrest. So, detectives are not really using forensic evidence to identify suspects. It's often not analyzed until the eve of trial. And so, the sort of the additional information advantage of forensic science that I talked about at the beginning isn't really being utilized. And the other sort of effect to this is it reduces the independence of forensic science if the — you know, if potentially inconsistent results will — you know, will lead to a lost case. And generally when you're designing systems, you don't want a design system with incentives along those lines. 

So, you know, why are things — why are things taking too long? Why is the testing pushed back? Well, you know, one of the reasons is the analysis is just — it takes a long time. Here we have the analysis time in days for the — for the different labs. And, you know, we can see that the times are quite lengthy, right? And what we're really talking about is — and for all these labs, we're talking about — and for all these different kind of interpretations, we're talking about, you know, months, not days. So, no, it's very difficult to integrate forensic science into your investigative practice if the turnaround is going to take months. You know, you can't — you can't exactly, you know, follow-up a particular lead that way. So — you know, so, that's one of the problems. So, why is the analysis occurring so late? Well, one, it is the shortage of forensic lab resources, required triages. Yes. Partly mindset forensics were really viewed as a means of strengthening already strong cases. And one of the interesting things we learn in the — in the interviews is that, you know, is — that's disconcerting was that in a number of jurisdictions, the forensics were viewed as a — as a way of meeting unrealistic jury expectations. And so, there was no real hope that any probative evidence or information would come out of the process. It was just done to close off a defense argument from saying, "Oh, well, you didn't — you know, you didn't do the forensic testing here." And there was a real concern about juries having widely unrealistic expectations and understanding of what forensic science could even potentially yield.

So — and I guess the other part that I want to mention was that the — it sort of makes sense that if you have limited lab resources that — -and you have a limited number of cases, of course, going to — going to trial if the case is triaged up, but you want to, you know, put these resources in the case when you're going to trial. But the — you know, the problem with that system, of course, is that many of the theoretical advantages of forensic science in terms of being an independent check that is uncorrelated with any other parts of the system and a loosely coupled check, you lose many of those advantages. Yeah. So, basically what I just said that it is rarely being used to identify suspects. It's rarely being used even in charging. And in rare cases where it's used at all, it's often being used to meet unrealistic jury expectations. And here, it potentially creates incentives for misconduct of two kinds. One there is that there — and there was some evident of this, of disagreements about the importance of various pieces of forensic evidence when it didn't fit a very well-developed prosecution theory of the case. And when those — when those — when that evidence only is developed late, late, late in the process, that's gonna be problematic. Similarly, there's pressure on prosecutors not to disclose that information to the defense if it potentially undermines a case. Now — I have no doubt that in a vast majority of cases, both the forensic examiners and the prosecutors are doing the right thing and exposing the evidence appropriately. But again, in my systems design perspective, you don't want to have a system that at least creates — perverse incentives like that.

So, we also did the — an experimental survey process where we — where we tested our sort of assumptions about forensic evidence. I mean, maybe forensic evidence is just vastly overrated, and it doesn't really matter. But we found — I hope unsurprisingly, both prosecutors and defense counsel thought it mattered quite a bit. We presented various scenarios, hypothetical scenarios to a handful of prosecutors, to a handful of the defense counsel. The findings were, you know, in accord with what one's expectations would be. The stronger forensic evidence, the more likely the defense counsel would push their — or recommend to their client that they accept a more a severe sentence. Similarly the — in the category of offers made, if the forensic science was stronger, the prosecutor would make a more severe sentence offer. So, we also did — we looked at correlations between these categories of forensic science and outcomes both in terms of plea bargaining outcomes and outcomes of trial. And certainly, stronger forensic science was associated with professions. And so, we're a little hesitant about drawing a strong causal conclusion there based on our interview findings because we're a little concerned about reverse causality, because a number of people, again, said that the stronger the case completely apart from the forensic evidence, the more likely the forensic evidence was to be even tested as a result. So, the question was whether the independent strength of the — of the case outside forensic evidence was actually driving the strength of the forensic evidence rather than what I think we had assumed going into the fold of study that it will be the other way around, that the strength of the forensic evidence would be driving the results.

We also looked at forensic lab efficiency. And again, using the Bureau of Justice Statistics Census of labs, we found that fee-based laboratories generally have a higher productivity. These are laboratories that charge fees to law enforcement agencies to test. That sort of free market system might act as a sort of an effective and efficient means of triaging cases. And then similarly, the existence of a computerized laboratory management system also is associated with increased productivity. I have to say we were a little discouraged to find how rare those were, that they were far from universal. And then actually when we requested our data, it required a fair amount of manual work to extract the data that we needed. For both of these, we're a little bit reluctant to draw strong causal claims, once again, because we have some concerns about potentially reversed causality or that there's something else that's going on here. But at least from a — at least that both of those factors were associated with higher lab throughput.

So, big takeaways here are that the theoretical promise of forensic science remains unrealized. It's — and this is consistent with the lecture for the last, you know, 50-odd years. The limited and late use of forensic evidence undermines some of the key theoretical benefits of the initial information and independence. A quicker turnaround of these categories is probably a necessary condition for earlier and better use. And then, of course, the fee-based labs appear to have greater productivity. I mean, overall, forensic science I still think has enormous theoretical potential to be used to identity suspects and a wide range of offenses. But, of course, in practice, it's rare outside homicide and it's used almost always following arrest. You know, in theory, it can be a truly independent verification of a suspect's guilt or innocence. In practice, however, we found the labs typically work pretty closely with law enforcement and often knew in advance what detectives and prosecutors wanted to prove. In theory, charging decisions are affected by the availability of forensic evidence, but in practice, prosecutors often did not have the results at the time of charging. In theory, juries correctly understand the probative way of forensic evidence and its limitations and reasons for its absence, but in practice, prosecutors and law enforcement, I believe, the juries harbor unrealistic beliefs about it. And a testing is often conducted to address these unrealistic beliefs rather than to actually generate probative evidence. And rather than forensic science shaping the subsequent investigation, we often found that the forensic evidence is often tested if prosecutors and investigators believe they already have an otherwise strong case. While this form of triage may be understandable given the scarcity of limited forensic resources, the absence of the testing and the investigating stage can lead to the problems. And it can also put pressure on both forensic science personnel and prosecutors in a way that's not helpful. A bunch of other professions and industries, manufacturing, aviation, and medicine, have really focused on designing systems with these independent, decoupled processes to reduce the chances that inevitable human lapse will lead to a grave error in the outcome of the system. Decoupling reduces the interdependency of the system and the effect of a single mistake in the outcome with the system. Forensic science has the potential to be such an independent process for the criminal justice system but can reduce unsolved crimes and wrongful convictions, but the use of forensic science that we documented unfortunately, uneven and generally late in the criminal justice process, is not able to solve that problem. Thank you.

KEVIN STROM: Hello, everybody. Thank you, again, for joining us. Thank you, James, for your presentation. I think what I'm going to talk about today sort of builds on some of those themes with the focus more on sexual assault kits and sexual assault kit processing. Some of the challenges that, you know, we've seen in the field but also opportunities to increase how law enforcement and laboratory — crime laboratories work on this challenging process.

So, as many of you know, this issue of unsubmitted and untested sexual assault kits has been a major issue for multiple decades, especially over the last decade. It’s received a good amount of attention in the media and received a good amount of funding from entities like NIJ and BJA. sexual assault kits, I'm sure maybe you're familiar with what these are, but this set of items used and collected by medical personnel from the — from the victim survivor, for the preservation of victim of physical evidence following a sexual assault report. And it's also done as a source of medical care to the victim survivor. So, extremely important both for the survivor's perspective but also from the perspective of public safety. So, unsubmitted kits just in terms of terminology, I'm using this term. Unsubmitted SAKs, sexual assault kits, are those that accrue in law enforcement agencies and those are ones that makes it in evidence rooms or other places in an agency. And then untested kits are ones that are with pending analysis in crime laboratories.

So, why do — why is this issue important, especially from the issue of unsubmitted but also on [INDISTINCT] testing? Well — and how do we go about sort of arriving at this problem which is — which is — I would say there is — there are really examples of success and promise, but also it's a huge challenge especially the amount and the demand for evidence, for DNA evidence in particular, has risen across the country. Well, one is DNA testing has not existed forever, as we all know. And in some cases, colder cases obviously, DNA testing was not available. And as part of that, it's also taken — part of the systems to catch up particularly in the law enforcement side to the understanding of the benefits of DNA testing [INDISTINCT] this point testing as early in the process as possible. There was also limited policies in the past regarding Sex Assault Kit submission testing and funding was a major issue. In past work, we found that sometimes law enforcement was hesitant to submit a sexual assault kit to the crime laboratory because they were concerned about the resources the crime lab had, and maybe they saw that in past experiences. There was also, in some cases, unclear benefits in investigation. Maybe it was consensual or reported consensual crime on the part of the suspect. And so, unclear benefits in terms of what the — analyzing the sexual assault kit would provide.

Staffing was — is a major issue and continues to be — or be a big theme of what I'm talking about today, insufficient staffing to investigate and prosecute cases and also insufficient staffing on the crime laboratory side. And also there is victim-blaming and misinterpreting signs of trauma. The idea of trauma-informed training and trauma-informed understanding about how victim survivors bring back differently as a result of trauma.

So, before jumping into the NIJ research project, I did want to talk about the Sexual Assault Kit Initiative which is led by BJA and some of the opportunities that's presented since FY15 and sort of really transforming what the area of not only sexual — producing backlog onto sexual assault kits but also addressing the systemic needs to more — to more — to sort of better link the system in terms of testing those kits, investigating those crimes and prosecuted them. And also all within the context of victim engagement and support. So, SAKI's grown tremendously since FY15 and currently there's 71 SAKI grantee sites, as you can see that there 25, those are statewide sites and then associated of other types of jurisdictional setups. A large amount of funding has been prior to that, and RTI is fortunate enough to be a part of that in terms of working local agencies for address their goals. They also — every site has to inventory all their unsubmitted kits and have a testing plan for handling them.

So, what is the national impact of SAKI to have been to date? Well, as I mentioned this inventory of unsubmitted sexual assault kits, conducting investigative case reviews which update with these kit — a testing plan which sometimes means working with a private laboratory out — outsource some are all those kits, sexual assault kit tracking, and also effective communication engagement with victim survivors. So, a few — a few examples of metric process SAKI sites as of June 2020 and the few that I just want to call your attention to. One, hundred and thirty thousand previously unsubmitted sexual assault kits had been — had been inventoried to date by the state local jurisdictions. Many of these more than half have been sent for testing. And then another key metric here over 23,000 CODIS upload into the system with 11,000 CODIS hits and many of those have led to subsequent investigation. So, really it's an active way of engaging and proving staff and resource and other resources to these jurisdictions in need.

So, why is it so important to test — submit and test a sexual assault kits in a comprehensive manner? Well, one, violent offenders, what we've learned especially over the last five plus years about the nature of violent offending, sexual assault offending, cross-over offending is significant, and I'll talk a little bit more about that in a second, justice for victims holding offenders accountable, but also this issue of populating the CODIS system, it's really only as effective as there already cases in CODIS and the research that come out of that. What we know about serial offending through sources like CODIS, cross-over offending in terms of cross-over offenders that don't just target strangers, that really target multiple type victims in their path, and also learning more about the reasons why SAKs have not been submitted or tested.

So, just a few examples in terms of serial cross-over and serial offending. Serial offending is much more common I think than is previously known and this is based on a study in Cuyahoga County but also similar work has been done in places like Detroit, Kansas, and other jurisdictions. And another key point here is that most serial offenders do not appear to consistently stick to a certain type of victim. Often, they're targeting acquaintances, family members, children, strangers, sometimes, a campus victim, on a — on a campus. So, we should sort of move away from this idea that we should have policies based on strangers versus non-strangers because all these things fit together. Another important point is the idea that we should prioritize necessarily stranger relate — kits associated with stranger crimes versus non-stranger. In part, because I think we've learned a tremendous amount in terms of the value of testing those acquaintance or non-stranger kits, and also the idea of linking cases, including offender to target, those victims known to them.

So, with that, I will move into —  So, let me talk a little bit, I'm going to talk — move now to the NIJ project that was conducted several years back. So, the focus of this project was really trying to develop of more of a quantitative approach towards looking at sexual assault kit processing efficiency. And there's certainly been some great work done in this area of sort of taking economic models and applying them to criminal justice systems, including the work done in — at West Virginia by Paul Speaker and others, including the folks at RAND. So, one of the key focuses of this study is to really look at what can be — what — can we identify and contextualize the types of factors that contribute to backlogs on the law enforcement side, and also, on tested kits and forensic crime laboratories. And perhaps most importantly, can we identify research informed guidance that help agencies move kits effectively through the process. We are focused here on sort of the crime — the law enforcement crime laboratory perspective, I'll talk a little bit of the prosecution perspective, but the quantitative analysis is really focused on law enforcement and prosecution. So, again, NIJ project — funded project determines specific policies, characteristics of jurisdictions, including input, resource input, staffing, technology, equipment, and then, also — and then, also other factors in terms of the characteristics of those agencies or laboratories. And the idea, again, is to create more of a research informed framework that ideally can benefit practitioners out there for guiding what they advocate for, and how they utilize both internal and external resources.

So, this is a Mixed-Method Study, I'm really going to be focused on Phases one and two. So, just to kind of review a little bit the methods here, we surveyed all forensic state and local forensics laboratories that conduct biological analysis, DNA analysis, and a 145 responses of those, and then, a linked sample of law enforcement agencies associated with those crime laboratories. So — that was for design, they also, in a third phase, we conducted site visits. So, six of those jurisdictions, including conducting interviews with crime laboratory personnel, law enforcement, and prosecution. I'm really going to focus mainly on Phases one and two, although, I'll talk a little bit about some of — some of the things we learned from these three.

So, what are some of the key findings? And just to — I won't get too much into the analysis framework here, but we did use an economic physical process called physical frontier models, to estimate the production or cost functions of these input and outputs. And then, one of the explicit goals is to account for the existence of inefficiency in the process. A larger question, as to what extent are inefficiencies in the process affecting maybe the inability for law enforcement to submit kit process and submit kits, and for crime lab to test them versus how much of it is dependent on resources. And it might be findings that — it really is combination of those technical inefficiencies and lack of resources that is impacting efficiency. So — but one of the key findings is that, even if for law enforcement agencies, even if all agency resources, so staff, equipment, other types of policies in place were used to their fullest potential, about 75% of law enforcement agencies would still have an operating backlog.

So, this is key for several reasons, one, in that it really demonstrates that we cannot really adequately address and overcome this issue without investing in staffing, and without investing in policies, and I'll come back to that a little bit more. Put little — put a little bit differently, law enforcement would only be able to submit under about 60% of all sexual assault kits, given their maximum use of resources. So, for crime laboratories, a little bit different picture, but also demonstrating the huge need to address the lack of resources, so even if all laboratory resources were used to their fullest potential, maximizing, sort of, maximizing their efficiency, 50% is a half of state and local laboratories would still have an operating — a testing backlog.

One of the other things we've learned is that, to relate to the — my point I was just making, is that jurisdictions cannot create a sustainable sexual assault kit processing — a process or plan without investing in staff. And this is one of the first studies to really quantify staff — the impact of staffing on these sexual assault kit processing, when I'm saying processing, I'm including law enforcement and crime labs, and its outputs. So, as an example, the key of law enforcement staff — and one issue here, just so folks know, we're looking at total staff and law enforcement agencies, which I — which I recognize is not — it's a — it's a little bit of more of an aggregate measure. But still, the importance of staffing is huge, 25%. So, a 25% increase in staffing, or 25% increase in process would — from a 100% increase in the number of full-time sworn officers, and for crime laboratories, number of SAKs tested increased by about 43% for a 100% increase in analysis. And I realized, these are large increases in staffing, but also — but they demonstrate that we can't overcome this issue without addressing it.

The other — the other outcome sits, I think really important to point too, is the importance of policies. I know many agencies are actively implementing policies, this data is about five years' data, about five years dated, I should say. So, we were able to look at how many — how many of these crime labs had to test all submission policies, for example — and similarly on the laboratory side, some of the specific policies they had in place. And I think the bottom line here is policies matter, it may be is common sense, but law enforcement agencies with a hundred percent submission policies are more efficient. And — but also the higher number of formal policies they emplace, increased sufficient, affecting the outcomes for law enforcement agencies. And so, this is not to say that that is the only way to address this problem, but I think it shows, that having a structured way, that the law enforcement agency, and those operating with them, understand what's happening with regard to how they process kits, is extremely important. Ultimately for crime laboratory, evidence acceptance policies, other types of policies associated with, for example, affect only analysis where they prioritize the sexual assault kit, and not the other evidence including bedding, only if they had a negative test on SAK, those are important factors increasing efficiency outputs.

So, what are some of the recommendations coming out of this, and I should mention, we have two articles in press, and I'll get to those in the following slide where you can get a lot more detail in terms of the specific models and results, and the implications. One, of course, is we need to invest in staff hiring and retention, and part of that is not only the idea of increased staffing, but also it's the idea of qualified and trained staffing, and looking at the issue of staff sustainability. So, one of the things that came up in the interviews we did is, is health and wellness, and also the idea of making sure that burnout is addressed. And that came out particularly on the laboratory side. But equally in the law enforcement side, we've done a lot of work on the SAKI project, looking at sexual assault units, and those types of recommendations, and how those are organized, and I think there's a lot of innovation that is occurring, but they could increasingly occur, not only address staffing, but look at how we staff a sexual assault unit, including adding civilian staffs, for example, a forensic, a technician, it helps track a forensic evidence as it moves in and back to the agency. Related recommendation is developing a system wide process map, so not thinking about these things in silos, which we hear a lot about moving away from silos, we have to put that in action in terms of developing a system-wide process map that looks at how things link together, and even developing sort of jurisdictional or system-wide outcomes in metrics, they can hold each other accountable, but also demonstrate that all these — all these entities, law enforcement, the crime lab, prosecution, victim services are all linked together, including the sexual assault nurses. And finally, developing interagency multidisciplinary policies, there are studies — our study has demonstrated the importance of policy, and also showing the ones that at least we found to be the most effective. But I think moving forward, we should look at how do we create policies that are cross-disciplinary, how do we engage the sexual assault response team or the NDT as these policies are created, and also get the buy-in before they're finalized. So, with that, here are — and this might be a little small, but the two articles I mentioned. One is the Criminal Justice Policy Review, and then, the other is Journal of Forensic Science. And my contact information is there, so happy to address the questions at the end, or please contact me directly. Thank you.

DONIA SLACK: Well, thank you, Kevin, very much. I am really excited to speak to you guys today about the impact of research and technology dissemination through the FTCoE. So, the obligatory disclaimer slide. This is my own opinion and does not necessarily reflect of the U.S. government or Department of Justice opinion. So I just wanted to give a quick overview about what the Forensic Technology Center of Excellence is. It is a program, a cooperative agreement, of the National Institute of Justice. And it does fit in the Office of Investigative and Forensic Sciences. So, it does fit with Jonathan McGrath, who will discuss some other programs shortly. It was instituted in 2007, and it has been administered by RTI International since 2011. And in this time, the FTCoE since 2007, actually, has reached tens of thousands of criminal justice practitioners by providing unbiased dissemination and evaluation of forensic technology to include best practices and policies.

It is — at RTI, we have a team of scientists and multimedia specialists that help support the FTCoE, along with a robust list of partners that we work with, around 16 or 17 or so, thirteen of which are CPAC universities that also help provide a lot of our activities and analysis across the board. It is managed by Dr. Jeri Ropero-Miller, our current American Academy of Forensic Sciences president. And it is co-led by myself, Donia Slack. I'm a research forensic scientist. And also Nicole Jones, who also is a research scientist at RTI.

So, in general, the FTCoE has guides overarching goals. We provide scientific and technical support to the NIJ research portfolio. We facilitate demonstration and transfer and adoption of technologies into any pertinent stakeholder. And we also provide technology assistance beyond the crime laboratories and into other appropriate criminal justice agencies to include law enforcement agencies. And we also provide and develop access to resources for research and education, training, best practices, all across the criminal justice community. And then, the fifth one, which I'll focus on today, is that we help to develop and implement strategic plans to evaluate the impact of forensic science funding across the entire criminal justice system.

So we do this essentially in three different ways. We call them our pillars. We do this by advancing technology, sharing knowledge, and addressing challenges. And each one of those basically are just different ways that we do our outreach to the community. We work with practitioners and researchers, we help with the R&D portfolio that NIJ funds every single year. And then, we also provide webinars and a lot of outreach and dissemination products across the board in our sharing knowledge pillar. So I — obviously, this is a little bit of an eye chart there, but I just wanted to show that, you know, every year, we do look at our impact across the criminal justice system and how many people in different organizations that we're reaching. And this was some of the metrics that we pulled from last calendar year. We were due to do that again by the end of next month. But we have thousands of people that we reach, either through our podcast, Just Science, or through different webinar attendance that we host in-person, and now virtual, a lot of virtual symposia and forums. And we also host working groups across the board, whether that's for law enforcement or for crime lab practitioners. We're a community resource, really. So I wanted to go over a handful of different projects that the FTCoE has been involved in for the past several years, that show some of this return-on-investment in NIJ and government funds.

So, with that, I wanted to take the presentation kind of like from start to finish of, you know, investing in funds from the very beginning, whether that's, you know, investing in an actual building design, all the way through what does that look like for management, and being able to look at, you know, how personnel and full-time equivalents actually are functioning in a — in a productive laboratory. So the first project that I want to discuss is our Lean Facility Design project that we work with — worked on a couple of years ago. So, in 2013, NIST actually published the "White Book on Forensic Science Labs”. It's basically a handbook to help laboratories plan and design the construction of efficient laboratories. And so, the FTCoE was able to put some funding into the evaluation of the impact of being able to follow this type of process. So this is looking at a Lean Facility Design. And as you could see on the right there, the — there's a checklist that's included in that White Book. And it — we worked with the Midwest Forensics Resource Center, formerly the Ames Laboratory, and also The Brazos Group, to help implement these guides and the process map for one agency. So we work with the Broward County Sheriff's office to go work with them through the checklist, and through the process map to do an assessment of their own facility to see where some of the gaps and where could there be some process improvement.

So we went through the Lean Facility Needs Assessment Methodology, starting off with situational analysis, looking at what is their current state of practice, and working through what could an ideal state of operation be. And then, closing the loop basically, meaning, you know, where were some of the gaps, what would be the ideal way for this — the structural architecture to actually be most efficient. So, we completed this analysis, and the results indicated that the Broward Sheriff's office actually did need to expand into a larger space. And so, what was great about this project is that, through this analysis and through this investment, they were able to work with their leadership to start receiving bids to be able to start procuring architectural bids, so that they could grow out the laboratories and they could implement the plan as stated by this Lean Facility Design analysis. But in the short term, what also came about was that they showed that they needed to switch to some more cloud-based options to implement on their current programs. They also noted that there were some interagency communication gaps. So they were able to improve that. And then, they also noticed that there were some bottlenecks that what they needed to do was just actually implement a couple more management positions, and promote a couple of positions so that those bottlenecks could be reduced.

The next project I want to talk about is one on advancing DNA efficiencies through working on Lean Six Sigma principles. So, we work very closely with West Virginia University, their College of Business and Economics. And so, on this project, we actually — we knew that they had this course teaching Lean Six Sigma, and certifying forensic professionals specifically in — as Green Belt. And the — it's — the course that was taught for three years in a row, and it was taught by Tim—Kupferschmid of the NYC OCME, who he himself is a Lean Six Sigma Black Belt. So the courses that they taught included forensic professionals from all over the country, and the requirement was they were supposed to try to find a tangible project that they would be able to take from start to finish, and through that, the FTCoE decided to follow the cohorts of different classes to determine what was the return on investment of being able to implement these Lean Six areas and principles in their own laboratories.

So there was a slew of different projects. These are some examples on the left of some of the projects that were funded, or that were implemented throughout the process. And the one I wanted to talk about was the decreased turnaround time in DNA analysis. So we worked with our social client specialist, Dr. Kelly Barrett, to be able to do both qualitative and quantitative analysis on how these programs were implemented. And we asked the basic program evaluation questions of, was the project actually implemented, what were the unintended and intended outcomes, and what was the leadership response, and what kind of challenges. And then, actually looking at some quantitative data as well to see was there an — a tangible return on that investment.

So the project that was implemented for advancing DNA efficiencies was actually the West Virginia State Police crime laboratory. And so, they were interested in reducing their turnaround time for DNA analysis. So, the laboratory had two separate units, the serology unit and the DNA unit. And through the Lean Six Sigma principle of Define, Measure, Analyze, Improve, and Control, they were able to find some areas for tangible change to be able to implement some more efficient processes. So some of the changes that they met — made were they were able to direct sample transfers straight from serology to the DNA section. They implemented a regular huddle meeting so that they could improve communication across the unit. They moved to a paperless case file process as well. And then, one of the big ones that they found was that they had a lot of active cases, but they saw that there could be an efficient way to outsource certain ones, and still be able to maintain an efficient workflow for themselves. And with that, they noted that there were three additional positions that if they just added these full-time equivalents, then their processes would actually increase.

So the quantitative results from this study, after year one, they saw a 17% reduction at turnaround time. And after year two, they actually saw that that increased to 48%. And then, when they looked at all of the active cases that they needed to outsource, they decided to outsource all of their sexual assault kits, does not include the efficiency as well. They focused on some unnecessary tasks. And then, they began tracking trends, so that they could perform their own change management, and they could do their own lessons learned, so that they could implement this across the board, and for other units. And so, because of the success of the DNA efficiency program that they did through Lean Six, they were able to move on to doing similar processes in other units like their drug identification section.

The next project is another one that is also advancing a DNA unit efficiency. So this was an interesting project because this was an evaluation of the impact of the DNA Efficiency Improvement grant program. So in 2009, the NIJ released a solicitation to specifically help forensic DNA efficiency units improve their efficiency. And one of the grantees that was — that was awarded a project was the Palm Beach County Sheriff's Office. And they — what — they were able to get funding, and this is one of these really good examples where cost sharing kind of comes into play because they also put up some of their own funding. They wanted to perform — or they wanted to actually enhance their efficiency. So — and I wanted to put a little plug here that this is an open access emphasize synergy article that you guys can download for the full study. But this was a project implemented, you see the milestone there in 2009, and then, you don't see a lot of the impact realized until a little bit later on.

So, in 2009, Palm Beach, their forensic biology unit, wanted to respond to some of the concerns of their surrounding — three surrounding law enforcement agencies regarding their turnaround time. They were basically getting complaints of the turnaround time was not adequate. And so, they did apply for this funding so that they could develop, or they could — yeah, they can develop a biological processing lab. And this was a pre-screening laboratory that would be managed by a city police agency, and it would be completely independent of the crime lab itself. And so, this was interesting because it serviced the three surrounding law enforcement agencies, and allowed crime scene evidence to be pre-screened at this location before moving on to the crime lab or the forensic biology unit. And what's interesting was this was actually put in place as a fee for service type of — it was a fee for services. So, you know, each one of the three law enforcement agencies actually had to help fund this pre-screening crime lab or — yeah, department. So the project — their project actually completed in 2012, and RTI was interested in this project from a different project that RTI had where we were analyzing the impact of forensic DNA efficiency program. And so, we saw that it — when we completed our project around the same time, around 2012, that we weren't seeing a very big return on investment. However, we stayed in touch with Cecilia Crouse, the Crime Lab Director there, and we knew that there were some return on investments that were being realized.

So the FTCoE was funded to be able to do a more in-depth analysis several years later. So in 2018, we went back, and we decided to take a look at all of the efficiencies that might have happened between 2012 and 2018 when their projected ended to current status. And so, we — when we did an analysis, one — the one thing I wanted to really bring attention to is that we saw an increase in law enforcement agency and the crime lab communication. So that was really important because before there was a little bit of contention on the turnaround time, and now, because they were working as a multi-disciplinary unit, they had a better communication strategy going forward. But beyond that, we actually saw some pretty impactful quantitative results. There were 27% turnaround time reduction; a 9% decrease in analyst case load; 88% decrease in the DNA backlog that they had; 54% increase in non-suspect cases; and then, also an 8% increase in CODIS entries. So one of the things I wanted to really call your attention to here is the lesson learned for doing evaluations like this is that laboratories do require time to fully implement a process before a thorough evaluation can be completed. So even if you're looking at the end of a grant period, you might not see an immediate return on investment, but when processes are put into place, you can see a more tangible return on investment as the years go on.

And then, the last one I wanted to talk about in this DNA world here is, you know, those were crime labs, and even a partnership with law enforcement, but this one was one where we really do try to target anyone who uses the forensic technology. So in 2017, Richland County in South Carolina expressed a desire to expand their ongoing rapid DNA program, and they wanted to put it into the booking environment because they knew that that's where the FBI was heading, especially with some of their pilot programs that came a little bit later on. And so, we helped to support the validation and proof of concept study of a RapidHIT ID placed in one of their detention centers. So the real impact of this study was showing that we — or the study shows that Rapid DNA instruments could be — produce a DNA profile meeting QAS requirements for CODIS that non-scientists are able to operate these instruments. And, you know, they were able to show a small yet still powerful return on that investment in that 69 profiles were uploaded in CODIS and generated even three CODIS hits. And a publication did arise from this as well.

So another — I wanted to talk a little bit about some of the FORESIGHT work that we work on with our partner at West Virginia University, Dr. Paul Speaker. So Project FORESIGHT is a business-oriented, self-evaluation tool that allows lab managers to analyze their productivity and their performance, and then, actually compare themselves against other similar sides or similar-in-function labs. And so, this has been developed and this has been managed by Dr. Paul Speaker at West Virginia University since 2008. It's ongoing. And it allows a synthesis of operational opportunities so that lab managers understand where they might need to improve budget, personnel, lab management. And he does this by looking at seven key performance areas, including cost, productivity, turnaround time, backlog.

So one of the evaluations that the FTCoE did was to determine what was the real return on investment from processing backlogs of untested SAKs, and I think this is a nice follow-on to even the work that Kevin just presented. And so, the FTCoE and West Virginia University, we evaluated — or Paul Speaker evaluated that return on investment at a jurisdictional level, so what is the real net benefit to society relative to that investment? That includes even things like victim services and that includes even case closure, or even pulling violent offenders off of the streets. And so, previous cost benefit analyses showed, you know, would just look at maybe just numbers, that the cost range per kit might be anywhere between $23 and $980. But what's great about FORESIGHT is that it does control for an economy of this scale. So for smaller jurisdictions, a cost expenditure might be $1,842, even at $980 a kit. But regardless of those expenditures, the return on investment for even those labs, it's still above a 5,000% return on investment. And when you're looking at a perfect economy of this scale, when you have a certain-sized jurisdiction, if they're able to process somewhere around 6,200 cases per year, the return on the societal investment is somewhere around 64,000. Of course, the mean is somewhere around — or maybe the median is closer to between 9,000 and 12,000, but either way, these are really tangible returns on investment for things like reducing the backlog. Same with the opioid crisis. So we all know that, you know, current financial considerations are showing that there is a severe underestimation on what has been happening in this opioid crisis, and how it has impacted the system. So the cost of the criminal justice system at this point has been estimated around $8,000,000,000 with $270,000,000 of that actually being borne by crime labs. So FORESIGHT data actually looked at different analytical processes to see, well, what is specifically — how is this looking especially in the crisis states and comparing those to the national norm. And so, even with the projected three percent annual rate increase, which is across the border, across all forensic discipline, it was shown that in the crisis states, you're getting way more expenditures than what was ever planned for. So what's great about these types of studies is that this allows jurisdictions to go to Congress and ask for additional funds. This is a tangible return on that type of analysis. And this is again another open access FSI Synergy paper that you can read.

And again, so we worked with Paul Speaker to determine, well, what is the best way to occupy a sustainable workforce? So right now we're in year three of working on a workforce calculator. And I just wanted to put a plug out there, we would like as much data as possible to make sure that we are basing it on reality, but there's a calculator, it's housed on the FTCoE website where you can enter in the type of jurisdiction that you have, whether it's large or small, different crime rates for violent property crime, and the population served, to determine what would be the optimal FTE per unit that would be needed to address the case workload. So please check that out and if at all possible, if you're able to share your data back with Dr. Speaker, that would be ideal.

And so a couple of good, positive things that, you know, we know are of importance in the community, one of which is our workforce resiliency endeavors. We have at this point, seven archived webinars, one of which is an OVC tool kit which I think is a fantastic resource that can be used by forensic professionals as well. Also there was a recent study funded by the NIJ on a survey of what is the real impact to crime labs on things like stress, burnout, vicarious trauma. We have a couple of podcasts, one on how leadership can be applied in crime scene investigator and then also building workforce resiliency. And then this is a new paper that just came out last month. It's a literature review that I wrote on some of the trauma and coping mechanisms that have been exhibited across the forensic science community, so that just came out a couple of weeks ago and it's open access again.

And then another challenge that we know the community is facing is how are we actually presenting evidence and is it being understood and our jurors understanding a lot of this evidence. It's hard enough for a forensic scientist and even of the court system to understand a lot of the statistics that's going on now. But what is a real juror's perception on how they're understanding forensic evidence? So we have a podcast on that and we also have a really great gap analysis and lit review on the comprehension of jurors and that's again FSI Synergy.

And then just to close it out, we do have other impact evaluations coming, looking more in the drug realm. We're looking at the FIDO program, mobile drug testing to aid in substance abuse response. And that one's really interesting because this is one that just came about during the COVID-19 pandemic. And a lot of the lessons learned — we're learning that can actually be shared for smaller jurisdictions that might not have a lot of funding, so we're doing an analysis of what that looks like. We're also doing a few cost benefit analysis, one on outsourcing for medical examiners and coroners, what is the return on investment for doing a straight direct to DNA, and then also what is the return on investment for reducing turnaround time for DUID cases. And then we have a couple of reports coming out, one on lawfully owed, some of the challenges of the CODIS database not having all of the samples from convicted or arrestee offenders that should be in there, and what does that look like and presenting some used cases and lessons learned. And then of course the forensic genetic genealogy is a big thing right now, so pulling together some of the information on what are the differences in the technology, what does this mean concerning third-party doctrine when it comes to databases like GEDmatch, and then what are some of the best practices in working in collaboration between law enforcement and crime labs. And then we've got some great podcasts coming out, some case studies where forensic technologies have been of impact, actual training on using forensic radiology for autopsies, and then a couple of apps, and then I always like to give a plug out for the NIJ's R&D symposium and poster session. It's going to be a pretty great one this year. It's virtual and it will be in February as it always runs at.

And then just the last plug, we can't do this without the community. We are completely indebted to all of the researchers and practitioners that do reach out to us and try to bridge the gap. And so, you know, see our webpage where we do try to — we do try to team together researchers and practitioners also through the NCJRS website and through ASCLD elite programs. So if there are areas in which you feel like you would like an evaluation or you think that there is a way that the FTCoE — we can help with any of these impacts, just — just reach out. So with that, thank you.

ERIC MARTIN: Thank you, Donia. Now Jon and I are going to kind of do a quick wrap-up, provide some closing remarks on the state of the social science research and forensic science portfolio. Jon's going to lead that off and then I will facilitate the Q&A after that.

JONATHAN MCGRATH: So I'm Jonathan McGrath, and I want to thank each of our panelists here. So yeah, I just want to thank each of our speakers, Donia, Kevin, and James; this is really great information. I think it demonstrates the breadth of what NIJ has been able to accomplish in the last 10 years or so. And so Eric and I wanted to discuss kind of where things have been with this social science research and forensic science portfolio that NIJ has been building. In particular, Eric is going to kind of run through some of the activities that have occurred when the program launched in approximately 2005 through the activities that went to 2015. And I'll touch upon several of the current updates and trends over the past five years and current activities that NIJ is focusing on. But I kind of thought of some fun facts as Donia and James and Kevin were talking too. The — you know, I'm going to scroll to the next slide.

So I fit within OIFS or the Office of Investigative and Forensic Sciences. And so we're the lead federal agency for forensic science research and development. And so we — our office provides short and long term resources both to develop forensic science research in the physical science area and the STEM fields, but then also to translate that research into practice as Donia mentioned. So I can't thank Donia and the FTCoE enough, they've done some wonderful work transitioning these products from the research world. So one of the fun facts that Donia didn't mention but alludes to, so one of the things that — James' talking points, was the implementation of LIM systems or Laboratory Information Management systems, and the BJS surveys that the most recent one that was conducted and they're finishing up their current iteration of the survey — since the survey of crime laboratories. But based on data from 2014, nine out of every ten laboratories had implemented a LIM system and the FTCoE recently provided a landscape report on LIM systems. So I encourage you, if you're interested or want to find out more information about the text and information that's collected and developed in laboratories or are interested in understanding how to tap into that information and to develop your own LIM systems, there's a great amount of detail in that report that just came out. So I'll try to throw a link in the chat box.

And then also when we discuss about testing all for sexual assault kits, there's a number of studies that have come out that have complimented Kevin's work based out of Stanford University, being Paul Speaker and Jen Doleac who have done some work on sexual assault kit return and investments. And really, the research is showing that the testing all kits is the best policy, it really is something that NIJ has looked at for several years now and is been incorporated into the recent DNA funding opportunities over the last three years that our Office of Justice Programs has provided. So that — there's actually a secondary allotment of funding for states that have legislation on the books to do test to all kit policies within their jurisdiction. So Bureau of Justice Assistance now handles that grant program, the Capacity Enhance of Backlog Reduction Program for DNA. So I encourage you to look at these things, we're definitely taking all this information in and helping develop our policies and practices accordingly as well.

So on this note, I certainly invite you, as collaborators from social science and physical science fields, we've had a great relationship with Eric Martin and his office to develop and bridge the gap between the social science researchers and forensic science researchers, and we look forward to additional partnership. So I'll turn the slide deck over to Eric for the new few slides and he'll talk about his office as well. Give me one second, Eric, and I'll toss—presenter roll to you.

ERIC MARTIN: So before I start cycling through my slides, just — I'm going to give just a brief overview of where I fit at NIJ, what we're about, and then present just a very high level review of the social science research and forensic science portfolio. And, you know, of course everybody we talked to belong to that portfolio as well. So I'm going to pretty much summarize findings, kind of generalize across a number of different studies, just kind of give you a picture of where we're at. And I'm sure there are many other researchers who partnered with us in the portfolio, some may be on the call and, you know, I'm not trying to pick out — cherry pick certain studies just kind of tying in ones that go together.

But before I get to that, little segue, I am a Social Science Analyst the Office of Research, Evaluation, and Technology, Jon's current counterpart agent — office at NIJ. And what we do, we're kind of a combination of social scientists and engineers if — you know, and kind of broad strokes in that lens. We handle a number of program evaluations of policy and practice. We also have a very robust technology standards division and developing technological solution to current criminal justice challenges. And then we also have social scientists who look at, you know, the causes and correlates of crime offending and juvenile delinquency. So we're pretty — we have a broad stroke here at NIJ. And we also support the entire criminal justice system. We deal with policing, corrections, courts, administration of justice, certain special topics like wrongful conviction. And Jon, at the end of our mini presentation, we're going to get to how research for SSRFS has continued on in support some of these other portfolios.

So now, just again, a very high level history of what we've done and where we're going. Kind of going all the way back to 2005, one of the founding analysts from NIJ who's no longer with NIJ, she's at BJA, Kathy Browning, really started to work with others, Nancy Ritter, Jim Doyle, as well, at the agency, looking at forensic science as a discipline and forensic science, if you can imagine back in 2005 was starting to become very popular around the country. There is a realization of forensic science, what it was doing, and although James said, you know, it has its roots back into the 19th Century, DNA analysis didn't become pretty much in the scene until the late 1980s, early '90s, and then it started to proliferate at that time. And so there was the sense of, we have this — we know what this is, we know what it's supposed to do, but we haven't really — we don't have a body of knowledge and program evaluations of how forensic science contributes to the criminal justice system, primarily in arrest and prosecution. And one thing that Kathy's stewardship of the portfolio was really instrumental in — from an NIJ's perspective was kind of bridging the gap, forensic scientists and social scientists usually are totally different departments and academy, they may not speak the same language, they may not even really associate with each other. So a lot of those early years was holding, working — or holding workshops and seminars, bringing the best and brightest of both fields together so that they can learn from each other and see what are the practicing issues in forensic science, how do we study that from a social science perspective.

And one of the interesting things that Kathy imparted in the report that I have up in the slide was, you know, social scientists often brought the right method to analyze efficiencies, but they weren't policy relevant. And forensic scientists of course are policy relevant because they knew the problems, they were dealing with the issues, but they needed assistance as far as a lot of method to do the analysis. So a lot of that first, what we call Wave 1, was building that groundwork. And then we get in Wave 2 to some of the emerging issues that were becoming very big at the time, particularly DNA sexual assault backlog. NIJ can kick up a number of action research projects, notably in Detroit and Houston, and also Los Angeles. And we also started to support through the portfolio other like-minded research topics where they started to kind of blend together. There was a lot of work at NIJ at the time in eyewitness testimonies. And that's one thing where the regular policing researchers and the SSRFS analysts really complement each other as well. There was kind of this growing support at NIJ among investigation and generating evidence. And also, you know, acknowledging some of the issues with investigations and how to correct them. Although it came out in much later, this is where you started to get some of the groundwork for the Sentinel Events Initiative, kind of growing out of a lot of these questions. So one portfolio kind of went to another portfolio went to another portfolio, and we have a whole groundswell of different topics that are very much interconnected at NIJ right now.

And then Wave 3, it was building on some of those early findings. You started to see, like, one project funded in 2005-2006, a couple of maybe in 2007, and those findings were starting to come in and there was two major things that happened at this time. One, NIJ hosted a seminar of the researchers that have participated in the portfolio to date at that time to kind of take stock of what we know where we are and how to move forward. And then also the National Academy of Sciences issued their report for strengthening forensic science in the United States. And the NAS report had a number of different recommendations for how to strengthen forensic science. And NIJ, we ended up, through our collaborations with researchers, really kind of focusing on three. Although, you know, if you go back and the report is still available, if you Google it, you'll be taken to the National Criminal Justice data archive, and you can pull the report from there. And then you'll see, it's like a 300-page report, there's a number of excellent recommendations in there, but through our work with our researchers and our portfolio, we really kind of hammered on three, improving investigations and how forensics can really assist investigations, and not just come in as we already discussed, later in the criminal justice process kind of assisting plea bargains and making stronger cases for conviction: Can forensic science redo it's wrongful conviction? Can we eliminate those sources of criminal justice error? And then also how do we have jurors understand the potential and limits of a forensic analysis when it comes to providing probative evidence in a case?

And you can see, we're talking about Wave 3, a very delineated time at NIJ, we call it 2010-2013, but these topics are still being researched at NIJ. We still keep coming back to it and a lot of what was already presented in our presentation speak to that. So briefly again, I alluded to, we had this seminar, it was in early 2013, and we really wanted to kind of take back of what we've been learning, where we want to go. We had done a number of different projects on DNA analysis, how DNA is used in court and in criminal investigations, how labs process DNA evidence, how backlogs can occur, how best to rectify those. And there was kind of — what was interesting, I was at the meeting at the time, it was really interesting to hear the — from the social science researchers who were at the meeting, and there was just kind of a growing awareness that we really needed to move the portfolio beyond just DNA to look at the potential — and, you know, go through the same efficiency trials and seeing how these different forensic methods support investigations and prosecutions.

So we delineated three different types of forensic methods that we wanted to also look at besides just DNA: ballistics, cyber, and digital. And ballistics, of those three, we've really have done a number of different — we've done a lot of work on it. We have some pretty solid findings on that. Cyber and digital are definitely forthcoming. We've been talking about it for a while and Jon's going to talk about where, you know, what he's doing, especially with how forensic can support crime analysis and intelligence, criminal intelligence, you know, that's where we see Cyber and digital really informing. And also we wanted to look at, you know, how police and crime labs interact. The police departments are definitely a big player here and part of this relationship and we really need to get a better handle of that.

So I promise you I'd be — this will be a short presentation and is probably wondering like, "Oh, my goodness Eric, turning over me." So here's the meat of it and I'm going to do this very quickly. I've been kind of alluding to my whole presentation, you know, the findings could really be — across number of studies, it could really be summarized into a few key bullets and I think that's great. When we start — we've done a lot of replication studies in the portfolio looking at how forensic evidence is used in different jurisdictions across the United States. And, you know, although there are certain subtle differences in the findings, a lot of them just kind of coalesce along these major themes. And I think that's important when we're trying to communicate lessons learned to practitioners and advise policy in how to improve. So as we all know, nothing majorly shocking here but DNA can probably be the most impactful when we're talking about the different types of forensic evidence in criminal cases. But that's a big statement and there's a number of different ways to use DNA ,and we find that DNA evidence usually comes into play more often in major felony cases, particularly sexual assault and homicide. Although homicide, you may be limited, sometimes in some types of homicides in what typical evidence is there. But we were inspired early on by some of the examples from the United Kingdom, to see, you know, what can we do about property crime? Can DNA really come into play here to clear some of these cases? And lo and behold, we did a randomized controlled trial, looking at — we didn't choose which cases to test physical or collective DNA evidence, we manipulated the timing and that's how they randomize. But they found that DNA when collected at burglary scenes and analyzed, yielded twice as many suspects as regular conventional investigative methods. And also interestingly enough, those who were identified tend to be more prolific offenders in other crimes as well. And we just published a NIJ journal article looking at cold cases, and the prospect of serial offenders in those cold cases and we bring up that timing that, you know, there may be an opportunity here using DNA property crime just to try in that more serial offenders in multiple crimes.

We're starting to see advanced DNA analysis methods come before; Donia concluded her presentation talking about one. We did a study with ITF, looking at familial and partial match searching, and we found that many states didn't have definitive authorizations but through conversations with crime lab directors, you know, this is something that's definitely willing and many crime labs were kind of exploring it. And of course these findings came in in 2017 so this may be dated, but we're talking about only a handful of states at the time under 10, either expressly authorize these types of searching or expressly prohibited it. And many policymakers at the state level were just worried about liability and just wanted more education about what it was. It was hard for a state level policymaker to engage in legislation when, you know, this is a pretty esoteric conversation we're having about different types of familial and partial match searching when it comes to DNA analysis. And then also — and Kevin spoke to this a little bit, the relationship between the crime lab employees matters a lot in terms of backlogs and what evidence is being submitted. Having definitive policies on what evidence to submit, when it's going to happen, and what's going to be tested really goes a long way in helping — curb backlogs from generating and proliferating. And then we had Peterson was mentioned, he did a study in 2010 for the portfolio, this was replicated by the University of New Haven in 2018, and then others, looking at different jurisdictions around the country, and they all kind of tell the same story. James Anderson said that was a very similar story. As well, we still at this point, have a hard time having forensics impact investigations by large, and this was found in ballistics too. We did a search of the NIBIN system and that network, and what can be generated from it. And the potential is great. The potential for case leads is strong when there — where there are sites at the jurisdiction, but there's just a disconnect in getting that information to detectives' hands in a timely matter, at the time of that study.

And then finally, and this was in response to an NIS report but Arizona State did a study on how juries — how they consider forensic evidence and if they did a mock trial, randomized control experiment, and one where they varied the judge instruction to the jury on what to consider. And they found that the experience in years of the forensic tech was very salient in the juror's mind on whether to — what weight to give to the testimony within 10, 15, 20 years' experience and that's what we stated, you know, when there's testimony given, that seems to matter a lot. And Jon is going to talk about a current work we're doing in, you know, researching potential different avenues having standardized guidelines for a forensic text testimony and how those can be delivered in a courtroom. So I'll turn it over now to Jon, and he will round up the presentation.

JONATHAN MCGRATH: Thanks, Eric. So one thing I want to mention too is that all the resources that Donia mentioned from the FTCoE as well as the information that Eric mentioned too, the publications and products from these research activities, they're all freely available, you can find them in the FTCoE website, as well as NIJ.gov and feel free to reach out to us if you can't find something that we've referred to. So I'm going to move on to other publicly available documents from NIJ. We have developed a number strategy research plans and each of these plans are five-year research plans. So the ones that we've been able to inject forensic topics into the last several years have been the safety, health, and wellness strategy; the policing strategy; and the court strategy. So each of these plans has a fairly similar format, it talks about the strategic priorities and objectives for NIJ. So I'll just touch upon these as well. You can go to the website and download all of the other research plans as well.

The safety, health, and wellness is just like it says, it's to promote research to help the physical and mental health of criminal justice professionals and promote science-based tools and strategies to monitor the health and improve the health. Policing is a forensic topic that was introduced in the plan, discusses the examination of the role of forensic science in the investigative processes. And then just this past spring, NIJ released the courts plan and that talks about the priorities of promoting and understanding the research to advance court practice, including the impact of forensic evidence on court operations, on case management, on case outcomes, as well as the use, and court policies and procedures and using other forensic evidence. So I encourage you to check out these plans, they stand as a guide post for NIJ's activities, and hopefully we'll build into our research activities from the next out years.

So I want to talk a little bit about some of the solicitations that we've had since kind of the first, second, and third waves that Eric mentioned, and one of the solicitations that we've introduced forensic topics as the research and evaluation on drugs and crime. As you can see here, there's a number of awards that we've made in 2017. We made a reward in Case Western to evaluate Cuyahoga County, that's Cleveland Areas, heroin-involved drug investigations protocols. And this is a protocol that guides medical examiners and law enforcement in the collection and preservation of forensic evidence from heroin and other opioid-related deaths to help secure faster indictments and successful prosecutions of drug dealers in the cases of overdosed deaths. In 2018 we made a related award at the University of Pittsburgh, looking at Novel Quantitation Workflows for Drugs Surveillance. And so this project is assessing the protocols and collections of actionable real-time quantitative forensic drug chemistry information to support investigations and prosecutions. To kind of note, there's a trend here: we're trying to look — instead of looking at the end results of using forensic evidence in the courtroom cases but to actually use it upfront in the investigations.

So similarly in 2019, NIJ made an award to UAB to look at both cyber intelligence capabilities and methods to discover, disrupt, and disseminate information on a listed fentanyl markets. And so this is the collaboration between UAB's computer forensic research and their drug chemistry laboratories. And also in FY19, NIJ made an award to RTI to examine the prevalence of fentanyl and its analogues in court-ordered mandatory drug testing population. And so they're doing this by developing and implementing toxicology methods to analyze hair samples, to have a greater detection window for those types of drugs that are typically either untested in routine drug screenings, then also look at hair as a sample of interest beyond just urine, or blood, or oral fluid. And then just this past year, NIJ made an award to the University of Kentucky to develop a novel drug surveillance strategy to identify stimulant drugs and their concentrations using what's called a wastewater-based epidemiology approach to comprehensively non-invasively monitor drug flows within communities in the real time.

So I'm going to shift over to a few other solicitations where we brought in forensic topics. In Fiscal Year '19, NIJ made an award to Florida International University under the Research and Evaluation on the Administration of Justice solicitation, and that project is to study efforts to improve juror comprehension of forensic instructions using randomized control trial to determine if instructional videos will assist with forensic testimony, both alone and when paired with conventional instructions for jury understanding and comprehension. So this kind of flows for some previous projects that others and Eric had mentioned as well. And FY19, under the Safety, Health, and Wellness award — there were two awards, the first to University of New Hampshire to examine the impact of work-related exposure to child exploitation evidence on the stress, mental, and physical health of digital forensic examiners. It's also going to look at agency level practices and training that can help mitigate trauma and promote resilience amongst the examiners. It's also going to develop a screening instrument to help agencies assess staff suitability for investigations requiring exposure to this type of evidence. So as Kevin mentioned at his presentation, NIJ is keenly aware of the issues that come up in stress, trauma, and vicarious trauma in particular with these types of cases.

And under the same solicitation, NIJ made an award to RTI to examine work-related stress amongst medicolegal death investigators, those that go to the scene and work with the victim's families to inform in their interventions and preventative training to reduce and alleviate work-related stressors and improving organizational outcomes such as turnover, job performance, and improve personal well-being. So as we know, this concept of staffing sustainability and maintenance and support for the staff is incredibly important. And then, in the recent Research and Evaluation on Policing solicitations, NIJ included research topics to address strategic priorities identified in the strategic plan to examine actual impact of forensic innovations on police practices and outcomes, such as through research-related state application and timeliness of data and outputs for forensic examinations, technologies, and innovations to support those police investigations, as well as system-based strategies. And so the solicitation aimed to promote and support research to optimize workforce development for both officers and civilian personnel handling forensics and that includes work in crime labs and medical examiner coroner offices. So while NIJ did not make specific awards directly related to this policing solicitation that were related to forensic sciences, the strategic plan still continues to serve as an example of NIJ's interest in these areas, and you'll see that in some of the other activities I'll discuss.

And this is the next activity that really aligns very closely with that policing strategy, which is the development of NIJ's Forensic Intelligence Model. And I saw a number of questions discussing the return of investment and the use and impact of forensic databases, such as AFIS for fingerprints, CODIS for DNA, and NIBIN for firearms. And so I think this project speaks directly to understanding that impact of using this information out for investigations. So forensic data is typically collected, analyzed, and reported on a case-by-case basis from crime labs for criminal investigations and court proceedings. By gathering — using this data earlier in a criminal inquiry cycle, forensic intelligence can be used across cases to help detect, prevent, and investigate crimes, and primarily, to identify links, patterns, and trends across cases and correlate other information that can be actionable and used directly by law enforcement investigators. So NIJ is currently gaining information about the successful implementation of these types of approaches at the federal, state, and local levels, and also models that have been used by other countries, and we hope to report additional information as this project moves forward.

So I'll touch upon a report that NIJ published back last year on the needs assessment of forensic laboratories and medical examiner/coroner offices. Just — without being too much in the background, this is a mandate from Congress to look at the workloads, backlogs, equipment, and personnel needs of the labs. But from the key findings that we found, we used Project FORESIGHT data, working with Paul Speaker to come up with some estimates and use the lab, the BJS survey data. But sufficiently consistent funding and strategic planning is incredibly needed to address the increasing amounts of forensic evidence and address fluctuations in evidence submissions that are driven by the supply and demand of the forensic science request. And so, not only do labs need approximately $640,000,000 annually to reach an optimal balance of incoming requests and case work reported, but it's also estimated that an additional 900 full-time staff are needed by state and local laboratories to address these workload issues. In particular, there's a shortage of board-certified forensic pathologists that needs to be addressed as well. And then, oh, what we actually needed the increase of systems-based communications to help communications between the forensic science laboratories and the service — the customers that they service.

And so this report sort of exists as a framework to understand some of the new research questions that NIJ is exploring, such as what research approaches can be taken to address system-level issues and coordination amongst the system stakeholders? How can we examine the impact and implementation on these standard and best practices, such as those that are being developed by the OSAC and others? What research is needed to better understand workforce issues, resource constraints, and impact on court operations? And how can we evaluate the impact of research of these organization and system levels? What research and development is needed to remote the sharing of data, information, and other important knowledge in a timely manner, and how can we evaluate the adoption of new technologies by forensic practitioners to help augment the work that the FTCoE is doing? And finally, how can NIJ promote further research, especially for projects that direct impact and include the participation of the forensic science practitioners?

So I think — I thank you for joining us today because I think the participation here, in the future, will help bridge this gap to help understand and address these questions. So one way that NIJ has provided the resource to make those connections is through a Connecting Researchers with Forensic Laboratories webpage. Now, this is specifically geared towards research partnerships on the STEM fields and physical science research, it can also be used to connect researchers with the forensic practitioners, to generate ideas in advance to a forthcoming solicitations, as well as once solicitations become public, to make those connections. And so whether it's this website or working with the American Society of Crime Laboratory Directors, or your — even your local state and laboratories that — in your surrounding areas, we definitely encourage you to reach out and make those connections. So, I will — that's my slide, and I will turn it over to Eric to moderate the Q & A.

ERIC MARTIN: Thank you, Jon. And for all of the participants, if you have a question for one of our panelists, please put it in the Q & A, and if it's directed to specifically to a panelist, please indicate that, that too does help. We already have about 25 or 30 questions. Our panelists have a hard stop at 3:15, so if we don't get your questions, Jon and I's contact information is right there. I believe the panelists' contact info might be on the registration page. If you don't — if we don't get to your question and you still want to describe it further, definitely feel free to write to Jon or I and we'll, you know, notate that. 

So one of the first questions we have, James, this is for you. And I understand the study was done a little bit ago and you may not have specifics on this but this came up in a couple — a different question. Could you tell us about how the lab were identified for the — your study? And if you had any information on that, we'd appreciate it, thanks.

JAMES ANDERSON: Sure. No, no, no. We — originally, we lined up — this is about 2012. Originally, we lined up, I think, seven or eight labs. But we wanted to get jurisdictions where we have cooperation and buy in from the prosecutor's office, from the lab, and the police — and the major police department. So that was relatively difficult to do, and we also wanted to get kind of a range of size, and if I recall, these were also NIBRS jurisdictions, or at least a couple of them were, for data. So, I mean, at the end of the day, it was essentially a convenient sample where we were aiming towards a diverse set of jurisdictions.

ERIC MARTIN: Oh, sorry. Thank you, James. While you're unmuted, on camera, I have a follow-up question. Do you know if there was any division between civilian and sworn crime scene officers in your study? And again, that would speak to the lab.

JAMES ANDERSON: Yeah. I'm going to have to punt on that, I don't know.

ERIC MARTIN: No, that's understandable. Thank you, James. Kevin, the next question is for you.

KEVIN STROM: Okay.

ERIC MARTIN: When assessing sexual assault kits for cold cases, how far back did you go to find jurisdiction on the study, as far as reinvestigations of cold cases and wrongful conviction cases?

KEVIN STROM: So with regard to the NIJ study, we were really looking at current cases, so the ability to either submit, or in the cases of the forensic lab, to process cases submitted within a year, within that calendar year, and then we also provided a lag period of 30 days within the following calendar year. So it's not — it's not necessary — it's kind of challenging to collect that information via survey, so it's not sort of a dynamic measure, you're sort of taking two pieces of information and comparing them to each other. But that's what we are interested in, it's the ability to process the sexual assault evidence within that year. And like I said, some lag time allowed, 30-day lag time within the following calendar year.

ERIC MARTIN: Thank you. And this next question, I think Kevin it would go to you, but I think it's applicable to all of the panelists. So we're going to start with you and then if James or Donia want to jump in. Based on your research, does it make sense to test all sexual assault kits, or triage? And if triage makes more sense, what approach should be used?

KEVIN STROM: Yeah. I mean, I think the recommendation we'd come out, as far as the SAKI project, is a test all approach. I think ultimately, that is the best path for the system, recognizing that, you know, there — and some of what I was talking about, there needs to be account — you know, resources — and accountability from policy makers, whether at the local or state or federal level, to make sure that we have the resources to do that. But we did find, in the absence of a “test all,” there are some — you know, I talked about the — prioritizing the SAK evidence from the crime lab's part and then assessing the secondary evidence if the — if the SAK kit is not successful. And then we also found that there was a positive impact on the law enforcement side — or I would say a negative impact. There were some agencies, we asked if agencies had to practice waiting to hear from the medical providers before they went and picked up the kit, and why this was — why this is important is that there's some jurisdictions that haven't really — maybe don't have a sort of a coordinated process in place, from moving the kits from the hospital or medical facility to the — to the law enforcement agency, and then to the crime lab. And I think if it's — somebody that mentioned — even some places that are looking at some direct submission, so those are — those are some other things that we found some promise for.

JONATHAN MCGRATH: Hey, Eric, and I'll jump in with a comment, too.

ERIC MARTIN: Yes, go ahead.

JONATHAN MCGRATH: I'm going to — I'm going to put a link in the chat box to the Stanford University paper that came out a couple years ago, but they did a cost-benefit analysis towards the entire backlogs, so testing all kits in the backlog. And they — their approach identified that for every — about $1,600 spent to test the kit, avert sexual assaults, the aftereffects, and all the other associated costs, on average of a $130,000. So there's some really good research out there, you can check out the paper, and, yeah. I'll refer to the other panelists, if they want to weigh in.

ERIC MARTIN: Yeah. James and Donia, do you have anything on this portion?

JAMES ANDERSON: I do not.

ERIC MARTIN: Thanks, James.

DONIA SLACK: Me neither.

ERIC MARTIN:  Well, Donia, since I have you, the next question is coming your way. Are you aware of any studies looking at the ROI of expanded CODIS and the types of offenses that improve hit rates versus unnecessary expansion?

DONIA SLACK: I am not aware of any studies that are looking at the expansion of the — of CODIS. And I don't know if that's a little bit of a loaded question on the unnecessary part there. But, you know, one of the areas that I know we're definitely interested in expanding some return on investment analysis is on alternative uses of DNA when you don't have the CODIS hit, so that's why in the — in the coming soon slide, we did have a couple of — or have one bullet point on the use of forensic or genetic genealogy searching. And so, I wouldn't call that unnecessary on one end, but I would call it as just complementary to the use of forensic DNA evidence to aid an investigation. So that's coming soon, we're looking at — of starting at research here, or that evaluation here very shortly, and then also tying it into what does that look like from a legal aspect of it across all 50 states with the ability to search against databases that are not administered by the Department of Justice or, you know, office of FBI and if CODIS [INDISTINCT] again if — so.

KEVIN STROM: Eric, I believe…

ERIC MARTIN: Yes.

KEVIN STROM: This is Kevin. I believe that Becky Gamble and some colleagues did have a paper on that topic, so that's something that I could track down and get to you.

ERIC MARTIN: Thanks. And that's — for those of you who may not know, that's Rebecca Campbell at Michigan State University.

KEVIN STROM: Correct.

ERIC MARTIN: The next question, I can take initially and then I'll turn over to the panelist, and then with Jon, we have a question specifically for you, so stay tuned. For this question, much of the research presented here at the panel focused on quantitative approaches. What types of qualitative research has been applied to forensic? I'll take that since it, you know, deals with the portfolio and large. As far as the different research portfolios at NIJ, I think Social Science Research and Forensic Science has benefitted from a lot of qualitative methods. If you remember back to the presentation we're talking about, those different ways, especially early on, when the portfolio — I'm sorry, the portfolio is in its formative years, we did a lot of qualitative research and facilitated a lot of different qualitative investigations. Holding focus groups with crime lab directors; looking at technicians and how their job went, the work flow going in; talking to state-level policy makers; talking to police chiefs and leaders about just what is forensics, what is it — what is it — what is it now, what is it need to be, how could it best improve investigations; how is it used in the courtroom; and of course I then collected conversations with prosecutors, defense attorneys, and judges as well. So there's actually quite a bit of qualitative work especially early on, but even, like I mentioned, the ICF was a major study and what I presented, most of that was qualitative. You know, there will be, you know — you know, national level surveys and some testing of efficiencies and return of investment, but a lot of it is just grounded in talking to people. Holding focus groups about who the major players in the industry are, what do we need to learn from them, that kind of thing. So I don't know if any of the panelists have anything to add to that.

JAMES ANDERSON: I mean, I'll just add that I think that, you know, certainly, we couldn’t have done our study without extensive qualitative research and arguably it's the most important component to really sort of understand the dynamics. And, you know, and at least in my research group generally, I certainly find that the qualitative stuff complements the quantitative stuff so importantly because it helps you really understand what the quantitative stuff is actually measuring which is, you know, more often than you would like not quite what you thought it was by hand.

KEVIN STROM: I would agree with that. We did — we did talk to six jurisdictions in the law enforcement, crime lab, and prosecution side and I think providing a little bit more context, for example, we talked to multiple Sergeants on the — on the law enforcement side, the Supervisors, Sexual Assault Unit who had an informal submit-all policy where it was just not — it was — where the evidence room would just process and batch the kits without approval from the Sergeant, and they actually thought that that was a great practice for them. I think it gave them a little bit of reassurance that they didn’t have to find kits that, you know, that maybe they thought had been submitted. And then also on the crime lab side, some of the context around just not adding more staff but how to — how to sustain them and the issue of burnout, which was a major theme for them. So I think those types of additional contexts are really important.

ERIC MARTIN: Excellent. So, Jon, I teed you up, this question is coming to you. How does the work NIJ does, in the Office of Investigative and Forensic Sciences, how is it different and how does it complement the work that NIST does in support of the forensic sciences?

JONATHAN MCGRATH: Yeah. That's a really great question. So NIST and NIJ have a — had a long partnership. There's a number of things that the NIST certainly can do that we can't do. NIJ does not have a physical laboratory. And so what's great about working with NIST is that they have the physical laboratory expertise. We have, you know, scientists on both sides, but we're able to manage things in a way to complement the development of both the research phase, the best practices; they complement the work that the OSAC is doing. One example is the OSAC, not only are they developing standards for practice in all the forensic science disciplines, but as each of the OSAC subcommittees identifies the standards to be developed, they also are identifying research needs associated with those standards. So we're actively monitoring those research needs and within our solicitations every year, we cite those research needs as well. So we want to be able to use our forensic science portfolios to build, you know, a more comprehensive support to what the practitioner community needs.

ERIC MARTIN: Thanks, Jon. The next question specifically for Kevin, but I think any of the panelists can also jump in as well. But, Kevin, the question is: do you see some of the current issues in policing right now, or, how I should I say, the calls for reform in policing? Will that impact staffing of crime lab?

ERIC MARTIN: That's a pretty tough question to answer.

KEVIN STROM: Yeah. I mean, it could — I mean, one concern I do have is that — and I think you're already starting to see this in some police agencies and survey agencies, is a reduction in civilian workforce. And especially when the crime labs are under the umbrella of a police organization, that could be the case. I don't know, I haven't heard that happening yet. But I also mentioned this practice of what, you know, my colleague Jim Marky and I have seen and recommended as a promising practice, which is to have a criminalist work within a Sexual Assault Unit to take that burn off detectives and have someone that's specifically focused on coordinating the forensic evidence, also when it comes back from the lab. So that would be my concern is also it could affect the crime lab but also those operating within the law enforcement agency that have a role in forensics.

ERIC MARTIN: Donia or James, do you have anything to add to that question?

JAMES ANDERSON: Yeah. I mean, you could make a case, right, that you should put more emphasis on the authentically objective forensic science if you have distrust of the potentially more subjective conventional investigative practices. I don't — I haven't heard anybody making that argument right now, but it's — you know, it's at least — it's a logically plausible argument.

ERIC MARTIN: Donia, did you have anything to add or…

DONIA SLACK: No. No. I do not.

ERIC MARTIN: Thank you. And I think we have time for one more question and I see one more kind of distinct question from what we've already had. This is interesting, this is about the impact of COVID on forensic processing, both the collection and processing. And the question relates to the report to Congress and staffing levels due to COVID but you could really expand it out, if the participants who asked the question wouldn’t mind, you know, what — have we seen anything — it goes beyond the kind of the scope of the panel ,but this is definitely a major issue for our country right now. So I'll just open it up to anybody on the panel. Are we seeing a COVID impact to the collection processing of forensic evidence and also the staffing of the labs?

JONATHAN MCGRATH: Yeah. This is Jon, I can — I could start and put or throw out some observations I guess. I think…

ERIC MARTIN: And I think we have Donia after you.

JONATHAN MCGRATH: Yes. I think we've seen a number of impact statements and response statements from professional organizations, not only do they provide, you know, resources to their membership but also to provide, you know, guidance where they can to safely operate and to get the resources they need to get the job, you know, you know, the day-to-day job done. In particular, I know National Association of Medical Examiners and International Association of Coroners and Medical Examiners have teamed up. They started providing some biweekly webinars in the spring and they provided resources on their websites. And so, certainly death investigations, when you walk into an unknown or uncertain scenario, there's a lot of measures that they're taking but also to, you know, to meet the CDC guidelines for the collection of specimens for testing purposes, in those types of investigations. So you look at the crime lab operations as well, I know I believe ASCLD and The American Academy of Forensic Sciences have put out information and guidance. So I think that NIJ were monitoring this information, and I know the question may have been specifically related to the effects on personnel and loss of resources and staff. This is something that, you know, we're very, very, you know, concerned and interested in understanding, you know, what the short or long-term ramifications could be. I think that's something that is reflected in our greater interest areas in understanding workforce development. One thing I — and I don't know if Donia have touched upon it, but we've seen and heard federal, state, local laboratory levels that the remote capabilities for LIM systems are incredibly important to continue doing the work and be able to have shift work come in person laboratory testing and then, you know, allow staff to then go home, work on reports and the, you know, interpretation of data remotely. So there's a number of promising lessons learned but also a number of laboratories that don't have that capability just yet to do remote access into the laboratories directly into the instruments that house the data. So whether it's remote or cloud resources, we're looking at expanding that LIMS report series to look at some lessons learned so far. I'll turn over to Donia.

DONIA SLACK: Yeah. Thanks, Jonathan, that's exactly what I was going to bring up. We are — we are taking a look specifically at LIM systems and how agencies were able to quickly pivot during this pandemic, especially ones that already had a lot of the electronic remote capabilities and they were able to, almost seamlessly, continue work as usual. And then, you know, there are other agencies that we've spoken to that did not have those capabilities quite in place as robustly as they had hoped. And, you know, the challenge that that presented just even having to maybe change the shift work and having overnight work so that you did have social distancing in the lab. And what that even means for stress and burnout of the forensic professionals who are now on this, you know, off-schedule that they weren't used to having. So we are looking at that currently, so that is something that the FTCoE is evaluating specific to LIMS and shift work and the computing requirement needed to put the lessons part on that.

ERIC MARTIN: Thanks, Donia.

JONATHAN MCGRATH: Another comment, just to throw in, I forgot to mention is that I think it was through our Office of Justice programs through BJA offered a COVID-related supplemental funding over the summer, and so that funding started going out the door incredibly fast once we understood the impact on law enforcement in law enforcement operation. So I believe crime laboratories had the ability to access that funding. I’d have to go back and check and see what the outcomes were on the forensic science community. But it is a resource that was available over the spring and summer.

ERIC MARTIN: Thanks. James or Kevin, do you have anything to add on this last question?

JAMES ANDERSON: Nothing for me.

KEVIN STROM: No. I do not.

ERIC MARTIN:  Well, I think I took a — the majority of questions, the different topics that I saw a lot of questions kind of coalesce together in the same general topic. I'd like to thank our panelists here. I appreciate their time to discuss this with us today and also all their hard work and the research that we discuss. I want to thank the participants as well. This was an excellent turnout, we're very excited, we're energized at NIJ to be able to share with you the work we support and, you know, really work together to promote fair and quality criminal justice outcomes as we improve practices and policies through science. So, thank you everyone. I'll give Jonathan the last word, my co-chair before we end for the afternoon.

JONATHAN MCGRATH: Yeah. I think — stay tuned for what NIJ has in store for the future. I think we've definitely laid a framework to continue these types of research and opportunities and also, you know, developing innovative tools for the practitioner. So I'm going to thank the panelists, thank the researchers who have already participated in this portfolio, and please reach out to us if you have any questions or need access to the resources.

ERIC MARTIN: Thanks. Bye, everybody.