Multilevel Evaluation of Project Safe Neighborhoods

DARYL FOX: Good afternoon everyone. Welcome to today’s webinar, “Multilevel Evaluation of Project Safe Neighborhoods,” hosted by the National Institute of Justice.  At this time, it’s my pleasure to introduce Basia Lopez, Social Science Analyst with the National Institute of Justice for some welcoming remarks and introductions.

BARBARA “BASIA” LOPEZ: Thank you very much, Daryl, for the introduction. I am a Social Scientist and Analyst for the NIJ in National Institute of Justice the Research Development and Evaluation Agency for the U.S. Department of Justice. Please note that the evaluation project presented here is supported by award number 2019-75-CX-0008 from the National Institute of Justice of the U.S. DOJ. The opinions, findings, and conclusions, or recommendations expressed—or recommendations expressed here are those of the authors and do not necessarily reflect the positions or policy of NIJ or the U.S. DOJ. 

So, let me very briefly tell you a little bit about PSN in this webinar. Project Safe Neighborhoods is one of the Department’s flagship violence crime prevention programs which brings together various department components and state, local, and tribal law enforcement community. Four key elements comprise this program: community engagement, prevention and intervention, focused and strategic enforcement, and accountability. This panel presents information about the enhanced PSN model implemented since 2018 and the National Evaluation being conducted by RTI International and the Justice Research and Statistics Association. Our presenters today are from RTI, James Trudeau, Lynn Langton, and Christine Lindquist, from JRSA, Roger Przybylski. After the presentation, our discussant Ronald DeWald, a Counsel to the U.S. Attorney for the Northern District of Illinois, will facilitate the discussion.  Detailed biographies can be found on the registration website for this webinar. After that, as Daryl already mentioned, we will facilitate a Q&A with the audience. With no further ado, I want to give the floor to James Trudeau with RTI International.

JAMES TRUDEAU: Thank you, Basia. And thank you NIJ for hosting this webinar and to everyone who’s watching. Thanks for attending. I’m Jim Trudeau. I’m leading the evaluation for RTI. Let me quickly walk you through today’s agenda. I will give an introduction to PSN and the National Evaluation. Then I’ll turn it to my colleague, Christine to talk about the case studies. Then our colleague Lynn Langton will talk about the national assessment. And Roger from JRSA will talk about crime statistics and some challenges that we’ve been facing related to that and are—the solutions that we hope to apply. Then I’ll close out the discussion part of the presentation with a look ahead as to where the evaluation will be going. As Basia mentioned, Ronald DeWald from the Northern District of Illinois has graciously agreed to serve as a discussant. He will offer his perspectives, and then we will save some time in reserve for a Q&A. And as was mentioned, if you have questions along the way, please put them in the Q&A box and we will compile them and address questions at the end. So through this presentation, we will look both at the forest and the trees and we hope to shed a little light on our evaluation. 

First some background on in case folks are not familiar. PSN grants are allocated to each of the federal districts. If you’re not familiar with the federal system, there are 94 federal districts across the U.S. and the territories, 31 states. Washington D.C. and the territories each have one district, 12 states have two districts, nine states have three, and three states have four. There are actually 93 U.S. Attorneys because Guam and the Northern Mariana Islands share one U.S. Attorney. The attorneys are appointed by the President and they serve as the Chief Federal Law Enforcement Officer in their district. 

The history behind PSN, it was initially launched back in 2001 as a formula grant program. There was a prior National Evaluation led by Ed McGarrell at the Michigan State University. I wanted to mention that Ed and a few others involved in that National Evaluation are serving on our advisory board for this evaluation. So we are benefiting from their insights. So as I mentioned, it was started as a formula grant program, it was then later switched to a competitive grant program, which means that not all districts are receiving the grants. And of importance later in this discussion, some of the districts received competitive grants in 2015, 2016, and 2017. Our evaluation is really focused on the enhanced model or PSN 2.0 that Basia mentioned. It was introduced late 2017, and really came into, you know, be, you know, actually implemented 2018. And it is—it was returned to a formula grant program with awards going to all 93 of the U.S. Attorney’s Offices. In this—the model introduced in 2018, there are five key elements, leadership by the U.S. Attorney’s Office, partnerships with law enforcement and community groups, targeted and prioritized enforcement using data and various technologies, the prevention of additional violence through various approaches that we’ll touch on later, and accountability for producing results, not just actions. And so this model is the focus of the evaluation, although I’ll touch on how we’re also adjusting to address the—some new enhancements introduced in this current year. And this slide touches on what Basia was talking about in terms of earlier this year. The current administration changed the focus a bit to community engagement, prevention and intervention, focused and strategic enforcement, and accountability. There’s also continued focus on using research and analysis to inform decision making.  Let me tell you a little bit about that, PSN varies greatly in local practice. The—in terms of the target enforcement areas that, you know, where the focus is, in some places it’s the entire district, which as I mentioned, could be an entire state. In other districts, it’s one or more entire cities. Or in some districts, they focus on specific neighborhoods, again in one or more cities. Most target firearm-related violent crime, but they’re also in places focused on other specific types of crimes, domestic violence or gang involved crime. And as I mentioned, the approaches pretty much reflect the key elements, but with different—you know, it looks different in different places. And we will delve into some of these differences in Christine and Lynn’s presentations. 

So the evaluation, our overarching goal set by NIJ is to develop and disseminate evidence of the effectiveness of PSN in reducing violent crime, and of the factors that influence that effectiveness. We’re doing both an implementation study, looking at, you know, what PSN looks like on the ground, and then an outcome evaluation to see how outcomes have changed. And we’re doing that at the national level and in the 10 selected case study sites. We’ll develop findings, materials from reports both for, you know, U.S. DOJ, but then also the local PSN practitioners. And we’ll disseminate findings to the practitioners to policymakers and variety of researchers. And then we’ll also submit our database with documentation that will support future analysis and replication. And the evaluation is anticipated to run through 2023.  

So as I’ve touched on a couple of times already, we have the National Assessment going on in all 94 districts, which will be a broad relatively high level look, and then the case studies in 10 districts that we selected collaboratively with DOJ. We’re going to start—I’m going to turn it over to Christine in a minute to talk about the case studies to give you all a concrete sense of what PSN looks like, then we’ll back up and look at the National Assessment. But the case studies will of course incorporate all the information used in the National Assessment as well. So there’s things in common, you know, we’re looking, as I mentioned, at the implementation, how it’s being implemented locally with a focus on those key elements. You know, leadership by the USAO partnerships, targeted and prioritized enforcement, preventing additional violence and accountability for results. And in terms of the outcomes, changes in violent crime and prosecutions, looking at the pre and the post of the, you know, the—with the enhanced being introduced in 2018, while we look at the several years 2015 to 2017 as sort of a pre-period there. And how do the two differ? Well, in terms of measuring implementation with the case studies, we’re having monthly calls with the PSN coordinators, we’re doing semiannual site visits with stakeholder interviews. We can’t do all that in all 94 districts, so we’re using materials, test courses that provided to DOJ, and we’re doing—we’ll—we will be doing some interviews, and then also web surveys. In terms of outcome data, again, in case studies, we can get way down in the details, we’ll use incident based and geocoded crime data from the local law enforcement agencies. In some places, we will use actual calls for service data. We’re also considering conducting resident surveys in the target enforcement areas. At the National Assessment, we’ll be using FBI Uniform Crime Reporting data, and then prosecution data that the taskforces provide to DOJ. And Roger’s going to talk quite a bit later about the data challenges as I mentioned. A unit of analysis in the case studies, we’re going to focus on their target enforcement areas, you know, where the areas that they’re focused on, that’s where we’ll focus on. But at the National Assessment, we will extend the lens to look at the entire district. And then also in some places look more closely at the target enforcement areas. And then just in terms of the compared to what question which is always important in evaluation, in the case studies we’ll identify similar comparison areas. In the National Assessment, as I said, we’ll have this pre/post enhanced PSN comparison. And then, if possible, we’ll look at areas within districts.  So that’s a real quick overview of the evaluation. We’ll be digging into a lot of the details, and then I’ll be back towards the end to give a look ahead. But for now, I will turn it over to my colleague, Christine Lindquist to talk about the case studies.

CHRISTINE LINDQUIST: Thank you, Jim. And thanks everyone for joining today’s webinar. I’m really excited to share some preliminary findings from the case study component based on the first round of site visits that our team recently conducted.  

I’d like to start with a brief recap of the main goal of the case study component, which is to assess both the implementation and the impact of the enhanced PSN model. With this component, we can explore this goal in an in-depth manner because we are focusing on a subset of PSN sites. In terms of our research questions, first, I’d like to provide a reminder that for the case study component, the unit of analysis is the target enforcement area where PSN activities are concentrated, not the entire USAO District. The main research questions for documenting implementation are within each case study TEA, how were the core PSN principles implemented? And what partnerships, strategies, and activities were leveraged? The main research questions for documenting impact are, how did PSN effect violent crime and each TEA over time in terms of pre and post enhanced PSN implementation, geographically in terms of comparing trends in PSN TEA’s to comparable areas not targeted by PSN, and then based on specific crime types targeted and not targeted by PSN? The case study sites were selected through a rigorous selection process with key criteria including the site’s ability to offer a strong test of the enhanced PSN model, the availability of incident-level crime data for the TEA and USAO interest in participating as a case study site.  

So based on web survey data, phone interviews, and reviewing administrative data, we recommend 10 USAO districts and within those, 14 TEAs for selection. As I believe Jim noted, the case study sites are not necessarily representative of all PSN initiatives, but they do collectively offer some geographic and programmatic diversity, which we will really be able to leverage and learning as much as we can from the case study component. Okay. 

In this map, the 10 USAO districts we selected are shaded in blue, and the specific TEAs are pinned. But I do want to point out that some TEAs are actually specific neighborhoods or areas within cities rather than the entire city. 

Based on census categories and starting on the West Coast, we have two districts in the western region including Oregon and Colorado, three in the south including Texas, Northern Louisiana, middle, and Tennessee Eastern, three in the Midwest, including Missouri, Eastern Illinois, Northern and Wisconsin Eastern, one in the northeast, including New York Western, and one site in the Mid-Atlantic, which is Maryland. One of the key sources of implementation data for the case study component is site visits. The first round was held virtually and we expect to do two additional rounds, which will hopefully be able to be done in person. We conducted one-hour video interviews that cover topics such as the district’s PSN history, the community context, partnerships, how the five principles have been implemented, challenges and innovations. Thanks to the support of the PSN coordinators in the 10 districts, we were able to interview a hundred and forty five stakeholders. And the types of stakeholders we interviewed are listed on the right. From the USAO, we interviewed U.S. Attorneys, acting U.S. Attorneys, PSN coordinators, and even some former coordinators and law enforcement coordinators. We interviewed a number of partners from local police departments who are key partners to PSN. We interviewed federal LEA partners including ATF, FBI, and DEA. And in some sites, we interviewed state law enforcement agency partners, partners from the District Attorney’s Office, corrections partners, community and faith-based organizations, and research partners. We analyzed the interview data and produced site specific reports and logic models which have been provided to each site. And our team reviewed and synthesized the findings to identify cross-site similarities as well as site specific themes. The focus of today’s presentation will be key themes regarding community context, implementation of the enhanced PSN principles, challenges in implementing those principles and stakeholder perceptions of innovative PSN activities. This is a high level logic model that shows our framework for what PSN is and the outcomes it might achieve. I don’t have time to go into detail on this logic model, but I did want to highlight a few things. So first at the top, you see community contextual factors that influence how PSN is implemented as well as the outcomes that it can achieve. On the far left, we have identified some of the key inputs and resources that affect PSN strategy, including USAO leadership, which is principle one of the enhanced PSN model, and local partnerships which are the second principle. These resources are then developed—I’m sorry, are then leveraged to develop key PSN strategies and activities which are shown in the middle column. Including strategies that reflect the targeted enforcement effort, which is principle three, prevention of additional violence, which is principle four, and accountability for results, which is principle five. The outputs of these activities can then be quantified as shown on the far right in the outputs box. And these outputs then affect the key outcomes shown here at the bottom such as increased victim cooperation with prosecution, increased community engagement, improved prosecution outcomes, and reductions in violent crime. One other thing I want to point out about the logic model is that as we quickly learned, PSN is not a structured program or set of activities that can easily be labeled as PSN. And this is why we put PSN in quotes in this slide. It’s more than the specific activities that are funded with the BJA formula grants. It’s the district’s overarching approach to violence reduction through the five principles. So in all sites, there are many complementary violence reduction initiatives that are taking place, but PSN tends to be viewed as the broadest and most consistent initiative. So our evaluation won’t attempt to parse out specifically which activities are unique to PSN versus other initiatives, but it will focus broadly on violence reduction activities that are consistent with the five principles of PSN. 

So moving on to our findings, we first wanted to share what we learned about the context in which PSN is being implemented. It won’t surprise any of you to learn that violent crime especially firearm related violence is increasing in all of the case study TEAs. With stakeholders noting increases in crimes such as armed robberies, carjackings, and drug-related gun crimes. Other crime trends are more youthful and gang involved offenders with a lot of retaliatory violence fueled by social media posts. Also some sites are seeing crime that is more dispersed geographically, and many communities are seeing a proliferation of guns.  

The second cross-site theme with community context is one of strained relationships with community members and law enforcement. Stakeholders noted the erosion of public trust, and police legitimately—legitimacy and talked about how this has reduced victim and witness cooperation, and effective crime clearance rates. There is some site specific variation in the extent of this problem but the general theme was one of deteriorating relationships in this regard. The final cross-site theme was the widespread influence of the pandemic. And not only was this partially behind the increase in crime, but it has also impacted PSN activities in terms of how they’re implemented with many activities that used to be done in person either suspended or being implemented virtually. And I just want to point out that in addition to these main cross-site themes, there are several site specific factors that influence PSN implementation differently in some communities. So we have policy or legislative changes taking place in some states such as bail reform, we have the varying magnitude of the opioid epidemic, and then issues such as homelessness and poverty. So we’ll be looking at this community specific variation in our case studies.  

The next five slides highlight the key themes from each of the principles of the enhanced PSN model. The first principle is USAO leadership for PSN efforts. And we found that in almost all sites, USAO leadership is perceived to be very strong and central to the district’s violence reduction strategy. PSN coordinators are very integrally involved in executing this principle and respondents that we interviewed noted a number of ways that USAO was showing leadership with PSN, such as convening all relevant stakeholders which is often through coordinating steering community meetings, taskforce, and working group meetings. USAO also mediates disputes among partners, identifies and leverages federal resources such as funding and training, and helps to formulate strategy by guiding but not dictating strategic planning. The main source of variation in this first principle pertains to the PSN coordinators tenure with PSN and their time available for the position, and then relatedly the US Attorney’s commitment to PSN. 

In some sites, these factors have resulted in very little institutional knowledge of PSN or limited bandwidth to complete PSN activities. But overall, USAO leadership was quite strong in all of the case study sites. 

The second principle includes partnerships at all levels of law enforcement and with the community. The first part of this, which is partnerships with law enforcement, was very strong in almost all sites. There are many examples of partnerships that were provided, especially partnerships between federal and local law enforcement agencies and these include partnerships such as interagency taskforces, embedding police officers or federal agents in one another’s units, leveraging shared resources, and establishing data sharing agreements to share crime data with one another. 

But the second part of this principle, which is partnerships with community-based organizations was weaker in most sites. A number of challenges to developing formal community partnerships were identified, and I’ll say more about that in just a minute.  But I do want to point out that several sites had already recognized that they needed to do more in this regard, especially given the earlier finding about strained community relationships with the criminal legal system. And several were already actively engaged in planning activities to increase community-based partnerships. A few sites did already have strong partnerships with community-based organizations in place, including some sites that are using a public health approach to address violence in their communities and have found good ways for community organizations to work with law enforcement. 

Targeted and prioritized enforcement is about figuring out where to focus investigation and prosecution resources. So this principle includes using data and technologies to identify the offenders who were driving violent crime rates in the most violent locations.  And we learned that most case studies sites have very robust data and technology driven processes to identify the most significant offenders, groups, and places. These activities, which are typically done by crime analyst at local police departments, use both data analysis and human intelligence. So some examples include using ShotSpotter technology, NIBIN or National Integrated Ballistics Information Network intelligence, calls for service data and human intelligence based on investigative work such as doing social media analysis. Many police departments share weekly reports based on these activities with PSN stakeholders, and often they develop a top 10 or top 25 list of individuals or groups. And some TEA specific neighborhoods are identified for targeted efforts as well. So in addition to identifying the most significant offenders and groups and locations, most sites also have very structured robust processes in place to share this information with a PSN team and discuss steps to take in terms of coming up with a plan for going after high value cases. So the most common example of this type of decision making is shoot reviews or significant offender reviews, which take place in almost all case study sites. And these results in a plan for what to do next in terms of investigation or arrest, but also in terms of discussing how to interrupt possible future shooting events that might stem from the current event.

So almost all sites have these reviews but there is variation in a couple of things, including whether community-based organizations are included, which is very rare, but does happen in a couple of sites, and then the extent of mitigation efforts that come out of this discussion. So this means using the information in the reviews to prevent cascading acts of violence. In some sites, mitigation efforts focus only on the individuals who are directly involved in the case. But in other sites, resources are provided to victims and witnesses who were involved more indirectly. And some sites even use—even implement place-based interventions that might prevent more violence in the community based on characteristics of the shooting that’s being reviewed. 

The targeted and prioritized enforcement principle also has to do with making decisions about the appropriate level of prosecution. And we found that this decision-making process was typically unstructured and involved informal communication with state and federal prosecutors. Most districts have expanded their prosecution resources through PSN by bringing on Special Assistant U.S. Attorneys to assist with federal cases, and implementing a purchase to get better evidence to prosecute cases more effectively.  And these are also part of the targeted enforcement effort principle. The prevention principle is described by DOJ as ensuring public awareness of the violence reduction strategy, communicating directly to offenders about the consequences of continuing violent behaviors, and supporting locally based prevention and reentry efforts. We found in our site visits for the case study sites that the most common activity by far under this principle was offender notification programs. So these involve identifying high-risk offenders, typically when they’re about to be released from jail or prison, and notifying them both that they are being monitored closely and will face swift responses for noncompliance, but also providing them with services and reentry support to help with employment and housing and other necessary services. But these activities don’t tend to take place very frequently, and they have been limited by the pandemic. Some of the other prevention activities include victim or witness-focused interventions to prevent further violence and increase cooperation with prosecution. And in some sites, these efforts can be quite intensive, such as a de-escalation program in one site, a full- time victim and witness coordinator position in another site, and a hospital-based victim intervention program. Other activities listed here under this principle include social media monitoring to identify and prevent violent acts before they take place. Place-based interventions such as Crime Prevention Through Environmental Design, efforts to raise community awareness of PSN, such as public service announcements or billboards, and then agency involvement in existing community activities that are already taking place. 

The final principle, which is accountability, includes data-driven activities to foster accountability for outcomes and not merely outputs. We found that in most sites, activities under this principle were limited to the sharing of data such as daily crime reports or metrics on the number of cases prosecuted, but there’s not typically discussion about pattern or trends and it was fairly uncommon for sites to use this type of data to refine their PSN strategy. But some respondents noted that activities of the research partner were considered to provide accountability. Also, in discussing this principle, we learned that many stakeholders felt that it was difficult to identify appropriate indicators of PSN effectiveness. There wasn’t really a clear consensus on what success would mean. And some of the things that stakeholders hope to impact can be quite difficult to measure such as looking at the long-term impact of strategic decisions on public safety, or looking at whether the highest risk individuals who were targeted by some of the activities actually change their behavior. And I also want to point out that some stakeholders felt that crime rates in particular were not necessarily appropriate indicators of whether PSN works because they are dependent on many factors other than PSN. When we asked stakeholders about the challenges they had experienced in implementing the key PSN principles, the most commonly mentioned challenge, which I touched on earlier, was engaging the community through formalized partnerships. Specific challenges were difficulty in identifying and vetting partners, finding ways to involve them in a meaningful way, limited resources for these partnerships, and lack of community interest in USAO involvement. Other challenges to implementing the five principles were limited resources and personnel, and also turnover in key roles and several sites. There was turnover in the police chief position, PSN coordinator position, and U.S. attorneys. And this was felt to create some instability and threatened collaborations, many of which took a lot of time and effort to establish over the years. We also learned about COVID-related restrictions on in-person activities, with some stakeholders finding that virtual meetings were less effective than in-person. And a few stakeholders noted that the number of overlapping violence reduction initiatives were a source of confusion. 

The final set of findings pertain to what stakeholders felt was innovative about their PSN activities, with the most common response across the sites being the strong collaboration among stakeholders. Other responses that were highlighted here pertain to site-specific activities that respondents thought were innovative such as having a full time victim and witness coordinator position, having a strong mentoring component, a hospital-based victim program, and specific activities such as ATF’s NIBIN program, the practice of co-locating certain personnel, County Crime Analysis Centers and having in-house investigators in the USAO. 

So I think the key take homes from our initial site visits are that all of the case study sites are working under very challenging conditions in terms of spikes in violent crime, strained community relationships, and continuing to operate under pandemic conditions.  And we will certainly be able to learn a lot from the diversity in community context that we’re seeing, as well as the diversity in PSN strategies. Another take home is that USAO leadership and violence reduction strategies was very strong across the case study sites, with the PSN coordinator being critical to success. As I mentioned, most districts have very strong targeted enforcement efforts to focus their investigation and prosecution resources on the most significant offenders and groups driving violent crime. And a final take home is that most districts could benefit from developing stronger community partnerships, which is something that many stakeholders already recognized as being important for addressing some of the strained relationships between community members and the criminal legal system. And this is something that will likely be addressed as districts refine their PSN strategies to reflect the new design pillars, which Jim will talk about at the very end. So thank you very much for your time.  And with that, I will turn this over to my colleague, Lynn Langton, who will share more about the National Assessment component.

LYNN LANGTON: Great. Thank you, Christine. So you just heard about the case study component and details about how PSN is implemented in specific areas. And now I’m going to be taking a step back and talking in more detail about the National Assessment. So Jim gave a great overview of the National Assessment already. But just as a quick reminder, with this component, we want to understand how the five PSN principles were implemented across the districts. What impact PSN 2.0 has had on crime rates nationwide, and then the extent to which variations in PSN implementation from one district to the next are related to violent crime outcomes. Now with the case study component that Christine just discussed, we’re focused on specific PSN enforcement areas, but as Jim mentioned, for the National Assessment, the unit of analysis is the entire Federal District, which is overseen by a US attorney. And this is largely because PSN formula grants are allocated to each district. And it’s up to the district to determine how many target enforcement areas they want to designate in order to have the greatest impact on crime rates. So you can see in this chart on the right, over half of the districts have one target enforcement area.  Well, at the far end, about 20 percent have four or more. So this is really part of the district’s PSN strategy. And it’s something that we’ll incorporate into the analysis. But the focus with the National Assessment is on the districts.  And then consistent with the case study component for the National Assessment 2015 through ’17 is the baseline period. 2018 through 2020 is the initial intervention period, what we’re referring to as PSN 2.0. And then we’ll have the revised PSN period or PSN 2.1, that will go from ’21 to ’23 that we’ll be incorporating into the assessment down the road. 

So again, we have two primary components of the National Assessment that we’ll be looking at. The implementation assessment, which is understanding how the five principles are being carried out in each district, and then the outcome assessment, which is looking at the impact of PSN on violent crime rates. And I’m going to start by talking about the implementation assessment and then we’ll move on to the outcome part. 

So, we have several key sources of data for the implementation assessment. The first is what I’ll be sharing findings from today. So from October of 2017 through June of 2019, DOJ collected semiannual reports from each of the district’s PSN coordinators about their PSN activities related to each of the five principles. And these reports covered all PSN activities, not just those funded directly by DOJ. And they were completed for each of the target enforcement areas in the district. So for our purposes, we’re aggregating data for each of those target enforcement areas up to the district level where the district had more than one TEA. Now, because DOJ stopped collecting these reports in 2019, for future analysis, we’ll be conducting semi-structured interviews with all PSN coordinators, and also conducting web-based surveys with key stakeholders in each district to get their perspective on how PSN has been implemented, whether they feel like it’s had an impact on crime. So these two efforts, the interviews and the surveys, have been delayed, but we’ll hopefully launch in early 2022. 

So there are three key stages or steps in the implementation assessment. The first is to look at basic descriptive statistics, some of which I’ll be sharing shortly, just to get a sense of what districts are doing around each of the five principles. But then there are many different activities or strategies related to each of the principles that a PSN taskforce might be engaged in. And so in addition to the basic descriptive statistics, we’re also using a type of factor analysis to determine how best to group measures that are related to each of the principles. So, for example DOJ—the DOJ reports, collected data on about 20 different enforcement efforts that a task force might be engaging in. And so then using factor analysis, we might condense this list to focus on a much smaller number of underlying dimensions that are being captured in the different combinations of responses that we get from the districts. 

And we want to do this collapsing, because we ultimately want to use a statistical method called latent class analysis to identify groups of districts that have similar configurations of implementation. So maybe there are some districts that have really strong leadership from the U.S. Attorney’s Office, and they have strong partnerships, but they focus more on enforcement than on prevention. Or maybe there are some that are characterized by really strong partnerships but have less leadership from the U.S. Attorney’s Office. And so we’ll use latent class analysis to create groups of the districts based on how they implement the five principles overall. And then, so first, we’re using the IRT factor analysis to identify which groups of variables go into those LCA models.  So it’s really a multistage process. 

And now I’m going to get into some of the descriptive statistics and findings from the factor analysis. So the first of the five principles is leadership from the U.S. attorney to convene all partners, Christine talked about this. And we’re actually using several different measures here to get at the extent of leadership, including things like whether the district received training and technical assistance, whether they’re working with a research partner, the extent to which they’ve dedicated federal resources to PSN. And then what you see in this figure is pretty basic, just how frequently they convene meetings of the PSN taskforce. And what you see here is that in just over half of the districts, the taskforces met at least monthly. But of course, there’s also a variation in this and we want to capture the extent to which that variation may be related to PSN’s effectiveness and reducing violence. That’s what we’re eventually trying to understand. 

Okay. So the next principle is leadership—I’m sorry, partnership at all levels of law enforcement and with the community. And here, the factor analysis is suggesting a two-factor solution. So I’ve created a break here between the two sets of factors. Basically, with the top four items that you see in this list potential partners, there’s very little variation. Basically, all districts have partnerships with local prosecutors, and with federal and state and local law enforcement. But then beyond those top four, there’s a lot more variation. So about 75 percent of all districts partner with community groups or service providers, about 60 percent with a research partner, and then if you go further down about 50 percent partner with schools. So we have this long list of partners, but with this second factor, because all of these different types of partnerships load on a single factor, it means that the districts are really distinguished by how many of these partners they’re working with, rather than which partners they’re working with. 

And then this next slide here shows the number of partners, average number of partners reach PSN district. And what this shows is about 12 percent have one to five partners. So pretty much they’re partnering only with law enforcement and local prosecutors. But then if you jump down to the other end of the chart, about 20 percent reported 14 or more types of partners. 

Moving on to the third principle, which is targeted and prioritized enforcement efforts.  Here again, we have several sets of variables that we’re going to be looking at. But for this presentation, I’m only showing different enforcement activities or strategies that law enforcement might use within the PSN district law enforcement and prosecutors might use within the district. And again, here, there’s considerable variation in the use of these different strategies. So at the high end, nearly all districts go through case screening process, a case review process to determine if they can be federally prosecuted. Then down towards the bottom of the chart, you know, less than half of the districts are employing tools like gun courts, drug courts, gunshot detection technology with which Christine talked about, and then things like anti-gang ordinances down at the very bottom are much more rare. But here, again, the factor analysis suggests a one factor solution, meaning that the districts are distinguished by how many of these activities they’re engaged in, rather than which of the activities. And again, this slide shows you a count and the vast majority are utilizing 10 or more different enforcement activities or strategies. 

The fourth principle is prevention of additional violence. And here again, we also have a single factor solution for the types of PSN Prevention Activities that districts are engaging in. On the more common end, you see that the majority of districts are engaging in some kind of neighborhood development or community building programs or they’re doing things like implementing media strategies to make the community aware of PSN, and then less common activities down towards the bottom of the chart are things like outreach to victims of violence and hospital trauma centers and university-based violence prevention programs. 

And here we have again, the count of the number of prevention activities in each district. About 30 percent are engaged in four or fewer, so—and then just under 20 percent are engaged in 10 or more different types of prevention activities through PSN. So, they’re generally engaging in fewer prevention activities than enforcement activities. 

And then finally the fifth—the fifth principle is accountability for results based on outcomes rather than outputs. And one of the measures that we’re using to get at the extent to which this principle is being implemented is what types of data the district is using to track PSN outcomes. And here the factor analysis suggested two-factor solution. So again, you see the split there between the two factors and basically at the top end, what we see is that the majority of districts are tracking outcome data, if they’re tracking outcome data. They’re using traditional criminal justice sources like law enforcement arrests and intelligence data and data on, you know, prosecutions on filings and sentences. And then the second factor is the less common data sources such as public health data, data from residents, surveys, or focus groups and then other data from community organizations. 

So, just to summarize the findings so far from the implementation assessment, you know, the descriptive statistics present a great picture of how PSN is being implemented nationwide and then the factor analysis generally suggest one or two factor solutions for each of the individual principles. So now, our next step in the implementation assessment will be to plug all of these factors into a latent class analysis model to identify classes of districts based on how they’re implementing the five principles of PSN 2.0. 

And then with that, we’re moving on to the outcome assessment. So, when we first started the evaluation, the idea was that DOJ would be collecting monthly crime counts from each of the task forces and that these counts would be used to track the impact of PSN on violent crime. Unfortunately, the task force has struggled to provide these data and DOJ stopped trying to collect them in June of 2019. So, we need crime data from another source and as Jim alluded to the best option is data from the FBI’s Uniform Crime Reporting Program. The summary reporting system covers the vast majority of law enforcements nationwide. So, we’re using that to analyze changes in violent crime from 2015 to 2020. However, because we can’t make this easy, the FBI’s sunset of the SRS at the end of 2020 meaning that those data are no longer being systematically collected from 2021 and beyond. So, we’re working on a plan to develop the outcome analysis from 2021 and beyond. And Roger will be talking about that more in the next presentation. And then eventually we’ll also be looking at outcomes related to the federal prosecution of PSN offenses and the length of sentences for convicted offenders under PSN. But today, I’m going to be focusing on trends in violence based on the SRS data. 

So, for the national assessment we’re using multi-level longitudinal growth models to examine trends in crime in the intervention period starting in 2018 when 2.0 was implemented, again, compared to the baseline period or the pre-intervention period. And I—as Jim mentioned because some of the districts received PSN funding during the baseline period when the grant awards were competitively awarded. We’re also separating the districts by whether they had competitive PSN funding in 2015 through 2017. Grants are typically for three years. So, for districts that received awards in 2016 or ‘17 in particular, those funds were still in place during the intervention period which is why it’s important to separate the two groups. And this enables us to look at whether PSN 2.0 had a differential impact on districts that already had a PSN program in place compared to those that did not. 

For the preliminary analyses that I’m going to be sharing today, we’re using monthly SRS data from 2016 through 2019. We still need to fold in 2015 data and then we also need to fold in 2020, but because 2020 was such an anomaly, we’re anticipating that it will look a little bit different. We’re focusing on local police departments and sheriffs and have excluded state police and special jurisdiction agencies because their jurisdictions often extend beyond district boundaries making it a little bit more complicated to incorporate them. For this analysis, agencies are nested within districts. So, again, the full model is at the district level excluding the territories for this preliminary analysis.  And then, another thing to note is that for now we are using counts rather than rates which are standardized by population. And one of our next steps will be to rerun the models with rates. But unless the district population is changed dramatically during the four-year period, we don’t expect that to make a huge difference. And then finally for this presentation, I’m going to show findings for all violence which includes murder, rape, and sexual assault, robbery, and aggravated assault, homicide or murder by itself, and then also robbery with a firearm. 

Okay. So, without further ado, this first graph shows trends in all violence. The orange line is the group that did not get a competitive PSN award in 2015 through ’17. And the blue line, which is the much smaller group is the group that got one or more competitive awards during the baseline period. And what you see in this graph is that violent crime was flat during the baseline period for both groups, but then both groups saw declines in violent crime during the intervention period. For the group that got the formula awards only, so did not get competitive PSN funding, the decrease was significant. For the group that only got—or that got competitive awards in 2015 through ’17, the decrease was nearly significant. So, these declines in the intervention period are what we expect to see if PSN is working as intended. 

Next, we’re looking at the models for trends and homicide. Again here, the trend is flat in the baseline period for both groups, but then both groups also show a decrease during the intervention period. In the case of the PSN 2.0-only group, the orange group, the relative decline in the intervention period compared to the baseline period was nearly two times greater, which suggests that PSN is working for the majority of districts that we’re going from no PSN to PSN 2.0. 

So, again, this is as we would expect to see. Okay. And next, this last graph shows the model trends for robbery with a firearm. Here, we see significant decreases in both the baseline and intervention periods for both groups, suggesting that robbery with a firearm had a downward trajectory. But the relative decrease during the intervention period was greater than during the baseline period. So, although the interaction term was not actually significant, it was still in the direction that we would expect to see. So, this is—this is consistent with what we might expect. 

Okay. In terms of next steps, these were preliminary models and though they’re promising, there’s a lot more that we want to explore in terms of looking at other crime types, using rates rather than counts, considering moving averages to smooth the trend line, and also adding in the additional years of SRS data. And then another big step is to begin to fold covariance into the model, including, you know, the classes that emerge from the latent class analysis and then other community factors and characteristics that we want to control for in this analysis. And then we also plan to use data on federal case filings and sentence lengths as covariance in the model as well as modeling those as outcome variables themselves. And then finally we’ll continue to assess how best to measure PSN outcomes in 2021 and beyond when we no longer have the SRS. And with that, I’m going to turn it over to Roger to talk about changes in the FBI’S UCR Program and what it means for the intervention.

ROGER PRZYBYLSKI: Thank you, Lynn. And can you hear me okay now, Daryl?

DARYL FOX: Yes, yes, indeed.

ROGER PRZYBYLSKI: Okay. So, yes, I’m going to spend a few minutes talking about some of the data challenges that we’re facing in the evaluation. And primarily, I’m going to focus on the implications or recent changes that have taken place with crime data reporting within the FBI’s UCR Program as this is what Lynn was alluding to and it’s really significant challenge that we’re now trying to deal with. And I’m going to start with a few words on the importance of crime data for the evaluation both, you know, Jim and Christine and then Lynn, of course, in presenting some of the outcome data as well. All alluded to the notion that crime data that relates to violent crime and then more specifically to firearms offenses is really key in terms of outcome goals for PSN and we’re needing to be able to obtain measures or operational data on both of those, for both the national assessment as well as for each of the case study sites. And there are a number of different dimensions that are important for us to be considering and we’re thinking about the data that we’re able to use or access for this. So, there is a geographic dimension to this. We need data at multiple levels of aggregation if you will.  We certainly need the violent crime and the firearms offense data if you will at the district level. We need it for targeted enforcement areas and for the comparison areas that we’ll be doing with as well. Temporally, when we look at the time factor, we need data pre-intervention and then post. So, we’re thinking to get operationally and to focus on the enhanced PSN principles. In practical terms, we’re needing to obtain data beginning with the calendar year 2015 and then running at least through the end of 2022, maybe even into 2023, something that we’ll be able to take us as far out but give us the ability to conduct the analysis prior to the end of the project period. And as Lynn alluded to, we’re also going to be needing some data pertaining to other crimes, not necessarily targeted by PSN, but that’s going to help us to understand what the impact has been generally in the districts and then the TEAs relative to the comparison areas. 

So, when we think about decision-making, the best data sources to use and the collection methods that we’re going to be able to use here, again, there are several kind of factors that we need to be taken into account. Certainly, when we think about the project budget and timing, that’s a central feature of what we need to be considering here. The data collection has to be economical and efficient and when we’re thinking about quality and completeness, we need to make sure that the data that we’re using has been consistently reported over time and that we’re able to get those information at a consistently reported format for these different levels of aggregation. We need that to be able to support the interrupted time series analysis and we just need to be able also to be able to do these comparative analyses that Lynn was alluding to when we’re working in comparisons with—comparison areas and being able to look at things across different implementations of PSN in different jurisdictions and so forth. So, there are a number of factors that are taken into account when we’re thinking about, again, data sources and data collection. And what we do know is that it would be really most economical and most efficient if we were able to be able to obtain data from a single central source if you will. And when we’re thinking about, again, what Christine talked about in terms of the case study sites, we’ve got 10 case study sites. And our goal here and what we already are making progress on is being able to obtain incident level crime data directly from local law enforcement agencies that are participating in PSN in each of those 10 case study sites. We’re—we haven’t already been able to do this and we’re working with other law enforcement agencies that are involved to be able to collect that data through a secure file transfer protocols or through an analytical platform that the RTI’s developed and is used with law enforcement agencies in the past. But using a similar approach, if you will, dealing directly with local law enforcement agencies simply isn’t feasible from the national assessment standpoint. We’re talking about, in this case, we’re looking at things nationally, more than 18,000 local law enforcement agencies nationwide and this is where we’re coming back to the notion of economics and being efficient in the data collection. It would behoove us if we were able to work with a single central source of data and that one would obviously the FBI’s Uniform Crime Reporting Program. Not only would it be an economical way to obtain data for all of these different districts, TEAs and comparison areas, but at least it’s also been theoretically reported consistently over time and across jurisdictions. If you look back at the history of the UCRs, there have been only a couple of changes that have taken place historically and those particular changes do not impact the data that we would be using or the outcome assessments that are taking place within PSN, but we are facing some very substantial other challenges that are there. 

So, let me give you a little bit of background just on the UCR Program in general and I’ll be pretty brief here. The program has been running since late 1920s, 1930, joint initiative initially between the International Association of Chiefs of Police and the FBI that ultimately resulted in a nationwide cooperative effort of now more than 18,000 law enforcement agencies voluntarily reporting data on crimes that are brought to their attention. They report information, crime incident data if you will on serious violent and property crime. This is what goes into crime rates and the crime index that we all hear about in the media and so forth. But it’s important to remember that not all crimes are indeed reported to law enforcement. When we’re working with UCR data and Lynn was alluding to summary reporting and I’m going to talk a little bit about the change that’s taken place to National Incident-Based Reporting. One of the key things is to remember here is that not all crimes are reported to law enforcement. We probably have an excellent count of homicides, a pretty poor count at things like sexual assault. But violent crimes in general are going to tend to be reported more consistently than what we see on some property crimes. And with few exceptions, the definitions of those crimes have been consistent over time and across jurisdictions for a very, very long period of time. 

What’s really beneficial about using UCR crime data from the FBI, it is also available in various forms that could be aggregated or disaggregated at multiple levels.  So, not only are their data available nationally, but you could disaggregate it to PSN districts to TEAs in some cases to the comparison areas we want to work with. There’s a great deal of flexibility at least that originally exist within the UCR data, so again trying to be able to use that as a single central source would really make things much more economical and efficient within the context of the PSN evaluation. 

Traditionally, over all these years that the UCR Program has been in place, Part 1, violent property crimes that we are most sort of interested in here and this notion of violent crime and firearms violent—firearms crimes have been reported in the summary statistics. And this is what Lynn was alluding to as the SRS Reporting, the summary reporting statistics that are there in the UCR Program. This is what is essentially changed as presenting a significant challenge for our work going forward. But when we look at the situation that I alluded to regardless of what we’re looking at in terms of summary reporting of the change that I’ll speak about in a second here in terms of National Incident-Based Reporting. There is a time lag between reporting and when that data becomes released or available for analysis and that’s something that we have to take into account here as well. But beginning back in the late 1980s, a plan was put into place or put down on paper, a blueprint, for transitioning crime reporting nationwide from the summary statistics, these aggregate sort of statistics to incident-based data being—that would be reported by every law enforcement agency participating in the UCR Program. That blueprint was laid out in the late 1980s. And then in the 1990s, we began to see some movement in terms of local law enforcement agencies and states actually accepting and reporting incident-based data rather than summary data to the FBI. One of the key differences between NIBRS or the National Incident-Based Reporting System and the Summary Reporting System is this notion that the data that’s reported as an incident-based rather than summary and it contains for more—far more detail on crimes known to law enforcement. So, you’ve got a great deal of information regarding the characteristics of the crime incident, the victims involved, the offenders involved if they’re known and so forth. And the movement to this type of data used in this type of reporting by law enforcement is really welcomed and is going to make a substantial difference in our ability to understand what’s taking place with crime from a number of different dimensions and angles. 

But reporting under the new system is something that has not taken place, and I’m going to say, expeditiously. And in about 2016, I believe is actually when this originally took place, the FBI issued an order if you will, that they will stop accepting summary data being reported from any local law enforcement agency at year end 2020. And going forward as of January of 2021 they would only accept incident-based data being reported by local agencies or through the state programs that serve as sort of a clearing house or repository for accepting that data from local agencies and passing them to the FBI. The challenge that this presents to many people but especially, I’m going to say to us in our evaluation of PSM, is that at the time that the sunset of SRS reporting at the end of 2020 calendar year, only about one-half of the nation’s eighteen thousand plus law enforcement agencies were capable of reporting NIBRS data to the FBI. Now, that roughly accounts for about 55 percent of the nation’s law enforcement agencies and about 50 percent of all of the nation’s population. When we look at the exact number that can’t report NIBRS or can’t report it to the FBI today, one of the issues I wanted to just mention very briefly here is that several major cities across the U.S. are currently not accounted for in NIBRS data. Cities like New York City, Los Angeles, Chicago, Philadelphia some of the largest cities in the country are still only capable of reporting summary data and, hence, this information again now starting with January of 2021, summary data is no longer being accepted or will no longer be available through the FBI, only NIBRS data will be. And when we look at the numbers here on reporting and where local law enforcement in the states are in terms of being able to report, accept, and pass on NIBRS data, we’re basically facing a big hole here with about half of the country not being able to if you will, provide the kind of data that we need moving forward from 2021 into the subsequent years that follow. This is a serious problem because it interrupts our ability to be able to get that data that we need for the national assessment and when we look at what new reporting is likely look at as the months and the years continue to move forward. There is going to be an increase, a substantial increase in the number of local agencies that are reporting NIBRS to the FBI but that’s going to occur incrementally as it has in the past. We’re not looking yet substantial increases if you will in the number of local law enforcement agencies that will report NIBRS data to the FBI and we’re not looking at a situation in another six months or perhaps a year where we’re able to take a look at only the FBI as a single source and have substantial coverage both in terms of geography and time series, to have consistent data to be able to use to look at outcomes for PSN. So what this essentially means is that the FBI UCR program can’t be used as a single source of crime data unless we’re basically going to drop out a number of districts or TEAs in terms of our analysis and assessment at the national level. So this is the challenge that we’re facing and what are we trying to do about it. The picture looks pretty, pretty difficult to deal with but we are taking steps to try and mitigate that and overcome the challenge and be able to get the kind of data that we need to be able to do the complete coverage that we’re looking at both geographically and temporally. 

So there are two key lines of inquiry if you will, that are taking place to mitigate the impact of this UCR changes. So number one we are doing an in-depth analysis and we have done this to great extent already of what the current NIBRS coverage looks like to be able to determine precisely which PSN targeted enforcement areas are currently covered by NIBRS reporting agencies. We have a couple of numbers that are up here on the slide both in terms of absolute numbers and percentages. And what we are looking at is about 32 percent of our TEAs are currently covered by NIBRS reporting agencies. It’s about a third but not a, you know, substantial number, substantial percentage. So we’re still facing if you will a significant problem even though we’re able to be able to access NIBRS data going forward and what I want to mention here and then maybe up on here on the upcoming slide, summary data and NIBRS data I’m going to say generally is somewhat interchangeable for analytical purposes because there are algorithms and procedures that can be used to take NIBRS data and translate it or transform it back into summary data. So if you got a consistent time series of data to be able to use for analysis. But if we don’t have that NIBRS data going forward and we don’t have that summary data, we’re out of luck. So the other angle that we’re taking here—or the other line of inquiry that we’re pursuing here is an inventory of state UCR programs to determine which of those are going to continue to accept SRS data or summary data from agencies and their state. We know why the FBI made the move to sunset summary data reporting and they do want to expedite or try to get as many agencies reporting NIBRS data as quickly as possible. We can all agree that that’s a good thing but when we look at the practical nature of it, you know, it’s going to take some time to get a substantial percentage of the country’s agencies reporting NIBRS. So what we’re doing is trying to try take an end run on this and be able to work with state UCR programs which are the sort of accepting or repository organization at the state level that takes local agency data and then passes it on to the FBI. And we do know already from some of the inventorying that we’ve done and analysis that we’ve done that there are a number of states—state UCR programs that are going to continue to accept summary data going forward and while that will expand the number of agencies that we have to deal with. We feel that, that is a practical approach to being able to address the challenge that’s being presented by the sun setting of the SRS program by the FBI. So we hope that we’re going to be able to if you will kind of close that gap by using SRS data up through the end of 2020 then using any NIBRS data that we can going forward as well as any SRS or summary data that’s being accepted at the state level by state programs. So we would still be working with the FBI as a central source but then with a limited number of state agencies to expand coverage to be able to deal with this challenge and problem. It keeps our data sources limited but if you will expands the geographic and temporal coverage in a way that we need. So we’re able to deal with the problem of the sun setting of the SCR—SRS and then the lack of complete coverage that’s available through current NIBRS reporting. 

And then there is one other challenge I want to briefly mention here that’s important and that is the fact that we want to be able to take a look at of course firearms violence and the ability to access valid data on non-fatal shootings presents another challenge that we’re trying to deal with. Both through summary reporting and NIBRS data, when working with UCR data crime again reported by local law enforcement crimes that they’re made aware of.  We are able to determine within either one of those aspects of the UCR program whether a firearm was involved and used that—whether a firearm was involved in that crime. But the data whether it was summary data or the NIBRS data does not actually tell you if a shooting occurred. In other words whether that firearm—the trigger was pulled and anybody was shoot at or hit. There are several sources of non-shooting—non-fatal shooting data that exist but again we’re dealing with a challenge here in terms of the economics and the efficiency of being able to collect data according to the geographic and temporal requirements that are needed for the evaluation. 

Non-fatal shooting data does indeed exist at the local level, at hospitals and emergency rooms but we could never deal with those large number of sources across the country in able to—in order to get this data. So we’re exploring other kind of data sets if you will to use as a proxy measure of this and one of those that we think might be beneficial of—might help us is something called the gun archive which is just one data set or one initiative that’s out there to try and obtain information on non-fatal shootings from a number of sources both law enforcement-related, media-related, and others. And we’re in the process right now if you will of trying to investigate the validity of some of that data and how well it might meet our needs. 

But this notion of being able to get non-fatal shootings or this notion of being able to look at a measure of firearm violence for an outcome of PSN presents another challenge, where the data that we’re able to access economically through a—kind of a centralized source just doesn’t give us what we need and we’re trying to find other aspects of being able to get at this data in a way that would really serve our purposes, be economical and efficient in nature. 

And let me stop there, that highlights the two major challenges I think that we’re facing in terms of outcome data for crime and what we’re, at least at this point in time, trying to do to mitigate those problems and overcome the barrier and I’ll stop there and I think turn things back over to Jim.

JAMES TRUDEAU: All right. Thank you, Roger. Yeah. We’re going to just take a very, very quick look at looking ahead.  We’re running a few minutes late. So, basically we’re going to sort out this date issues, we’re going to work on how exactly to address 2020 and then as we’ve talked about we’re going to incorporate the 2021 enhancements.  We’re going to—this is more or less repeating everything. So skip right over, we’re going to get the data, we’re going to analyze the data. We’re going to turn those data and analysis into reports to—both the DOJ and specific to the case study sites. And we will disseminate two practitioners and then also through research and as I mentioned earlier at the end we will archive the data to support future analysis. At this point I will turn it over to our discussant Ron DeWald, Ron?

RONALD DEWALD: Thanks, Jim. As Jim says I’ve been asked to be a discussant for today’s webinar and give you some perspectives about the evaluation on behalf of prosecutors. It’s important to know that I’m just one of 93 PSN Coordinators in the nation. Like many of my colleagues I’m an Assistant US Attorney, I prosecute cases, I supervise other AUSAs and their prosecutions. I oversee federal, state, and local collaborative initiatives that involve prosecution, intervention and crime prevention strategies. PSN coordinators are prosecutors, we’re not researchers. But I tell you, we really appreciate what the researchers are doing for us. 

One of the great things about Project Safe Neighborhoods is that it’s not a one-size-fits-all approach. The U.S. Attorneys have tremendous discretion within the PSN framework to implement their own anti-violence strategies to address violent crime in their specific district. So, while we expect the evaluators to see some basic similarities among the districts we know that we’re all different. We all talk and compare notes, sometimes in person at the Annual PSN Coordinators Conference, or more recently over the phone, or on emails, or on Zoom. And regardless of the particular districts PSN program, all of us are asking the same question: are we having an impact on violence? And I think this evaluation will help us answer that question. From my view as a prosecutor, it appears to me that the evaluators are looking at the right things and asking the right questions. I specifically applaud the recognition that PSN strategies are not the only game in town. There are often many other crime reduction efforts in any given district. Sometimes those efforts compliment PSN efforts, other times not so much, right? So focusing at how PSN fits into the array of other crime and violence reduction efforts already underway is a very important question in order to tease out the impact of PSN itself. 

I also applaud the recognition that the national trends—violent crime trends will impact each district differently. Many times there are district-specific incidents, or policies, or practices that may lead to a particular outcome that the other districts may not experience. We appreciate how difficult the task it is to analyze such data, those violent crime rates, and the impact of the district-specific PSN strategy on such a level. And we recognize it might indeed be impossible in some situations. And some of our dark days when violent crime rate appears out of control, the question many of us ask is whether it would be even worse without the efforts of the PSN taskforce.That may be impossible to measure. But any feedback will be valuable to us and that is why I’m particularly happy that the Northern District of Illinois was chosen as one of the 10 site studies.  That will help us make—take it even closer look at our data and our strategies to help make us make informed decisions in the future. I just have one more minute or so just to make one point and that is there is one thing to keep in mind. Today we had the discussion of measuring PSN outcomes through crime data and federal prosecution data. We can’t forget about the critical role of the state prosecutors. Most PSN taskforces partner with state and local prosecutors and have some sort of established method to review state cases for federal prosecution. In order to make an informed decision on which cases to prosecute the AUSAs must first assess the outcomes of those local prosecutions. Do state statutes have similar penalties as federal statutes? Is the most readably provable charge being aggressively pursued by local prosecutors or are the charges being reduced, dismissed, or never charged in the first place? What sentences are being imposed by the state judges? We ask these questions all the time. The answers help us to determine which cases are to bring the federal court and which to leave in state court. Therefore, when you’re measuring PSN outcomes it might help to think about how you can measure state and local prosecution data along with the federal prosecution data for a more complete picture on PSN’s impact on violent crime rates. 

The critical question for all of us, all the PSN coordinators is, “What works to reduce violence?” All the US Attorneys, all the prosecutors ask that very question. And as the PSN coordinator it’s a question that keeps me up at night. We all have a firm belief that what we are doing matters but it’s critically important now more than ever as violent crime rate continues to trend upwards. To continually re-asses our strategies. We anticipate that this evaluation will help us do that. Thank you all of you for all your hard work on this project. And I think that gives us a few minutes for questions and answers, Jim?

JAMES TRUDEAU: Yes, Basia, do you have questions assigned to us or would you like to read them and then assign them on the fly?

BARBARA “BASIA” LOPEZ: Yes. I can—I can read them to you, Jim. So it’s more of a comment here from Robin, who clarifies that the program consisted of instruction to the USAA independent of the grants which were intended to support the program. But I do have a specific question to Lynn. When you were talking about factor analysis the question came from Leway who’s asking, “What are these activities strategy is based on and what are the implications of factor analysis in this context?”

LYNN LANGTON: So the descriptive statistics that I was presenting the laundry list of activities that is straight from the DOJ data collection report that was administered to all the PSN coordinators and completed for each of the targeted enforcement areas. So that was a pre-defined list of activities and then the PSN coordinators were selecting which of those—their target enforcement areas were engaged in. So you know that’s one challenge with using survey data and not having the context behind it. You know, some of the differences we might see between the number of prevention activities and enforcement activities that a district engages in could be a product of the list of activities that they were given to select from. But that said, so with the factor analysis we did not see any—it was a single factor solution that came out of the factor analysis which really suggested there’s not—that these activities don’t hold together in any kind of meaningful way in terms of how districts were responding and so it’s less about what types of activities they’re engaging in that distinguishes them and more about the number of activities. So the magnitude of activities they’re engaged in versus the different types.  Hope that answered the question.

BARBARA “BASIA” LOPEZ: Thank you, Lynn. And the next question is to Roger. Roger, Frank is asking whether you are working with local law enforcement to get the data especially in the cities that do not participate in NIBRS?

ROGER PRZYBYLSKI: So when I started going to is that that would definitely be I think one of the contingency approaches that we would pursue. We’re somewhat optimistic that cities like New York and Chicago perhaps also like Los Angeles but that—those particular states, New York, Illinois will continue to have summary data available from those agencies going forward and that’s where kind of our fingers are crossed as we do this inventory and kind of look at that alignment with our districts and TEAs. If we do continue to get substantial kind of blockage or missing data that we’re trying to deal with here, then I think the notion of being able to deal with another limited number of local agencies could be a possibility to pursue here. We’re just trying to keep things efficient and economical but knowing how important large cities like New York, Chicago, Philadelphia, Los Angeles would be that would certainly be a path to be able to try and fill in some of the blanks given how much of the crime that those particular municipalities account for and how important would it be to have that data for the outcome assessment. So the answer basically is yes, we take a look at that if we needed to be able to do that or had to be able to go that route.

BARBARA “BASIA” LOPEZ: Thank you very much, Roger. We still have a few questions here on the Q and A but because we’re running out of time we encourage the participants to email Tammy Sando with any outstanding questions and then we’ll contact the panelists directly who will provide the answers to you. With this I will—I just want to see if Jim has any final thoughts but on behalf of NIJ I want to thank you very much for attending this webinar. Jim?

JAMES TRUDEAU: Yes. Basia, thank you. Two thoughts. First I would like to correct one of my slides. I noticed there was a typo on there it said—that part of it—part of the intent is to stem the flow of firearms, that should’ve read stem the flow of firearms that are used in crime, that’s really the focus. But I mostly wanted to thank all of you for your attendance I—hopefully at least piqued your interest in terms of the preliminary findings thus far and as we’ve touched on quite a bit. We are going to continue this evaluation and continue to sort out the issues that we’ve identified so far. There’ll probably be new issues also but we’ll sort them out in time. So I hope that you do stay tuned. And again please contact us if there are specific questions or anything we could address. Thank you all for your attendance.