Evidence-Based Policing: The Importance of Research and Evidence

DAVID MUHLHAUSEN: Good morning and welcome. Thank you for joining us today. For those of you who are coming from out of town, welcome to Washington, DC.

I want to extend a special welcome to our NIJ LEADS scholars. Can our LEADS scholars please stand?

I’d like to give everybody a round of applause.

For those of you who are not familiar with the LEADS Scholars Program, it stands for Law Enforcement Advancing Data and Science. Our scholars are sworn law enforcement officers from across the country working to implement evidence-based policing in their agencies. They are the true pioneers of the evidence-based policing movement and we are honored to have them here at NIJ for their LEADS summer session.

Everyone in this room understands the importance of data, evidence, and rigorous research. I'm preaching to the choir when I say we can't implement programs that just seem to work or make us feel good. We need to conduct research and evaluate programs to determine their effectiveness. We need to find out what works and what doesn't work.

As the U.S. Department of Justice's lead research and evaluation agency, NIJ has a commitment to evidence-based policing practices and using rigorous evaluation methods to determine what works and what doesn't. Our LEADS scholars embody this every day. Our panelists also embody this and I'm excited to have them here to speak about the good work they do.

NIJ began this Research for the Real World series back in 2009 to feature research that is changing our thinking about criminal justice policies and practices.

Today's event is our 30th Research for the Real World panel. Our panelists today come from across the country from a variety of backgrounds and experience. I am honored to have them all here today.

Each panelist will have 15 to 20 minutes to present some of the highlights of their work. I'll then start a discussion with a few questions of my own, but I'll also quickly open up for questions from members of the audience.

I want our panelists to discuss their backgrounds and work more thoroughly, but I like to briefly introduce the impressive group of panelists we have today. Sgt. Ryan Keck is the Coordinator of the Oregon Center for Policing Excellence. Chuck Tyree is the Program Manager with the New York Division of Criminal Justice Services. Dr. Angela Hawken is the Founder — Founding Director of BetaGov at NYU. And last but certainly not the least, Chief Louis Dekmar is Chief of Police in LaGrange, Georgia and President of the IACP. With that, I will turn over to you, Ryan.

RYAN KECK: Good morning, everyone. So, my name is Ryan Keck. I'm the Training Manager at the Oregon — Oregon Center for Policing Excellence at Oregon's Department of Public Safety Standards and Training. A long title. So, to clarify a little bit, the Oregon Department of Public Safety Standards and Training, DPSST, is like a post agency in the state of Oregon with the unique difference that it’s a centralized academy for all city, county, state, university, and tribal police officers in the state as well as corrections, parole, probation, and telecommunicators.

Within the department then is the Center for Policing Excellence, which I'm a part of and have the privilege of leading. The Center for Policing Excellence, I'll go into a little more detail as to how we came to be, but the — I guess the nuts and bolts of this is that we were created to promote evidence-based policing and we've been fortunate to have some unique experiences over the last four years doing so, and I hope to share some of those with you but more importantly share a few takeaways for you and you try to implement in your own jurisdiction this movement of evidence-based policing.

So, Oregon, just by way of familiarization for everyone, over what — far on the left side, of course, ninety-eight thousand square miles, ninth largest state in the country, but with only four million people, which sounds like a lot. But in terms of population density, you're talking thirty-ninth state in the country. To add, up in Portland, the tiny little speck on the top left of the state, has over 60 percent of that population. Everybody else is really quite rural. So, when they say there's 41 people per square mile, not a real accurate representation of the types of agencies and the size of our state.

Within the state, there's 5,500 police officers working for 181 police agencies. Now, 60 percent of those agencies have 10 or fewer officers. So, it's a unique group, if you will, of police officers to work with and try to promote then a concept such as evidence-based policing.

The benefit we have is, as I said, one centralized training academy, so every one of those officers when they start their career come to the same consistent training program in Salem, Oregon where I work.

Additionally then, we provide a number of in-service trainings that allow for consistency across the state in a number of different areas. And then we manage, out of the Center for Policing Excellence, two leadership academies both for supervisors and new middle managers, again, with the consistent focus and being that people at those levels need to be certified in our state to maintain that promotion, they have to receive certification training, which we put on.

So, evidence-based practices began in Oregon in the early 2000s focused on corrections. So, after some mandatory minimum sentencing standards were put into place in the state, it recognized there was a cost associated with that and a challenge to the aspects of, you know, recidivism. What happens when all of those people who've been mandated to spend a certain amount of time in prison then return to their communities, what happens next? And so, over the course of those years, a number of programs were developed both in the prisons as well as the county level for community corrections, parole, probation, focusing in on cognitive behavioral interventions primarily to try to make an effort to, you know, reduce, I’ll choose that word, reduce recidivism in our state. And we were fortunate to have some pretty high marks in that area for some time in our state.

In 2013, they wanted to take it a step further because while those efforts were being made and recidivism was being reduced, prisons still are expensive. There are a lot of people in prison beds and they didn't want to have to build more prisons in the state. So, House Bill 3194, the Criminal Justice Reform Act, was enacted in an effort to reduce those costs and divert through justice reinvestment the prison costs to the local agencies to do more evidence-based practices.

Fortunately though, there were also some savvy law enforcement champions who engaged in that dialogue as well as some supportive legislators who then suggested there's something police can do as well on the topic, and that if we could include policing in this conversation about reducing recidivism and improving offender accountability and improving public safety, we might get better results or even better results in our state.

And so, from that — those conversations, if you will, come — the first lessons I might share that maybe seem age-old and commonplace when you talk about leadership and buy-in and change management, but from our experience it's very true. If you align with the priorities of the decision-makers, your opportunities seem to go up. Right. The focus was on corrections but from the policing side by introducing that we could be a part of that conversation, too, and evidence-based practice had an application in the policing field, right, we gained attention, gained buy-in, and we're able to move forward.

The other thing that I would — I would add is that bringing in outside experts in those kinds of conversations is really important. You're never an expert in your own backyard. Many of you have probably felt that or experienced that. But when you bring somebody in, they can speak peer-to-peer with decision-makers and talk about what they know, what they've experienced, results that they've had, and why it would be important in your jurisdiction, it can go a long way. I can talk all day long for the last five years with even leaders in our own state about the value of evidence-based policing, but they go to a conference like a symposium yesterday, they go to a, you know, ASEBP conference or something like that and all of a sudden they have these great ideas and they've heard about these wonderful things called evidence-based policing practices. So, it's always a benefit to bring in peer-to-peer outside experience. And that's what helped lead to the legislation that passed in 2013.

You see their purposes of the Center that were generated out of that statute. The other screen here I'll work through awkwardly there, is the instruction guide they gave us, which is what they wanted us to do or how they wanted us to do it. But as you can see, it doesn't give a whole lot of direction per se as to how, it's just what we needed to accomplish. We tackled these in a reverse order. Being that we're an educational institution, we focused on D and we developed training programs first, focusing on — first — front-line supervisors and how we can include in their conversations about being leaders and developing productivity in their agencies, where evidence-based policing fits.

We then in 20 — that was in 2014. In 2015, we moved up to A, where we created opportunities for the efforts and the initiatives being made in police agencies to be shared. So, we posed or we created the Oregon POP Conference, focused specifically on what was going on in our state in the realm of problem-oriented policing and evidence-based policing. And we launched the Oregon Knowledge Bank, which I'll talk about a little bit further and many of you may have heard of, but a platform online that allows agencies to put those ideas out front in public and talk about the problem they were trying to address, the lessons that they learned from that experience.

2016, we moved on to B, where we were able to hire our own full-time research analyst who provides technical assistance and support directly to the agencies. So, now you get that, you know, the A, where practitioners can put their information out, researchers and analysts can learn about it and provide their input, and our researcher who can speak to the agencies and share with them the value and the benefits and what's been learned through research and how to improve.

C is that missing piece for us. That's where our goal is still set in what we're trying to build are those true partnerships between the police agencies and the researchers. Right now, they're getting, what some, you know, has described and you may know the literature about cooperative partnerships. We would like to see that co-production, those collaborative partnerships. But there's a piece from our lesson to share and say that before you can get any one of those or any other level of partnership of researchers and practitioners, you got to start really on the — on the front end with information and inspiration, at least from a practitioner's side, and I might argue also on the researcher's side. That's where we come in, I think our lesson is learned. Education is a key piece. If you look at these lists here, you're talking about behavior change, you're asking practitioners and researchers to do things they may not traditionally do. So it's changing behavior. And education is exactly that if you believe in it the way that we do. It’s designed to influence change in people's behavior. And so, that's the focus we've taken through our experience thus far.

So we have a self-generated vision statement, but more importantly you see the triad there, if you will, that we designed as our foundational approach. It kind of captures the level of growth that's necessary. Be at the top of the pyramid there, is of course what we'd like to see long-term and it — it needs that vision of people at all levels using research, data, analysis, evaluation to contribute to public safety, public value, and improve the criminal justice system. But it's a long haul to get there in many cases as you all know, and we recognize that like any other kind of educational effort you have to meet people where they're at, to create a learning environment that accepts where they're at, their opinions, their perspectives, and slowly move towards something else or at least allow them to construct meaning around new ideas or other information. And so we look at it from the Know level, that foundational level of what do they need to know, what is research, what is evidence-based policing, what's the value of it, how do they obtain more information, what does it mean for them, and then into that realm of doing something with it and trying to make an effort to do something different in their departments or their jurisdictions.

So to talk a little bit more about that Know realm, this kind of answers probably many people's questions, so what do you do? So we provide free training and resources, is really the bottom line that all of which are designed to promote evidence-based policing. To address that Know level of individuals, the people that are learning about the concepts, we provide education to multiple levels. We've inserted lessons into our basic academy for all police officers in the state. We do in-service training where we take it out to the regions and share the information and again in these leadership programs I described, it's a critical piece of the education for new sergeants, new lieutenants, and captains.

Particularly for those leadership academies, we have a project-based component. That's where the coaching element comes in. Our staff are more facilitators, coaches than they are instructors in the leadership academies. Meeting people where they're at means that you have to work with them on their ideas versus try to tell them what the right idea is. So we just have dialogue, we — and we asked them what did they think about this idea of looking into research as opposed to tradition or dogma in affecting change or improving operations in their local police department. We asked them every challenge and said, well, what does that mean for you as a leader if there is additional information out there, there are more resources, there's a larger world than policing in your small town, what does it mean for you as a leader to take advantage of that or disregard that? And so the coaching environment is very heavy in these — in these training programs. And it culminates with a project in which we ask them to go out and address an issue very similar to like a POP or a SARA model type of approach to an issue that they're experiencing in their communities, and then they have time to work on that.

This kind of speaks to another lesson that I would say from my educational standpoint is we practice to do something principal. It's not enough to just tell people about great ideas or other information, you have to have them doing something, getting their feet wet, trying something, failing, even, if necessary. You know, the Do principle comes from a book that I won't describe the title right now, but it's pretty tongue-in-cheek about how to get through life a little easier and it speaks to the fact that action isn't just the result of motivation, it can also cause it. And if you put people into action, you can generate motivation through that experience and that's very much what we try to do. And we've had the good fortune of seeing many of these student projects turn into full-fledged POP or evidence-based projects in our state as we've moved over the last four years.

The last piece then are resources and we add to these training environments by supplementing it with more information and support. We do so through the knowledge bank as you see the logo there on the web address, and I encourage you if you haven't looked to take a look at it. Just note that the knowledge bank is not evidence-based policing, it's generation of ideas and collaboration. It poses either projects going on in our state or efforts that have been made to address an issue, and then it provides summarized research for agencies, for officers to be able to grab the nuggets of pieces of research that support the work that they're interested in.

In the Do category then, that's where we're get — after we got the hook in them and the agencies or the individuals start moving forward, we continue to support them. Technical support comes from our research analyst who's able to continue to talk with them about identifying their problems, more research that can support the direction that they're headed and even help with the development of effective measures so that they can start to get a sense of whether or not it's working for them. This is a service available to any and all agencies throughout the state of Oregon. So, for many agencies, they don't have analysts even to look at their cost per service data, now we provide them that opportunity.

Micro-Grants is a program that we have in place, thanks to a BJA grant, which allows us to add to the support by reducing the financial barriers that some agencies run into when trying to implement even small projects like CPTED improvements or community engagement opportunities. So, we work with them to ensure that they've got a POP or evidence-based type of model to their efforts. And with that in tow, and the responsibility to report back to us their measures and share it on the knowledge bank, we'll give them some seed money to see those projects move forward.

Lastly, then, are the networking opportunities. We continue to provide the Oregon POP Conference so that they can come together and share what they've learned, what they've experienced, and network around these opportunities. We've started bringing in key experts and speakers from around the country, not just out of Oregon the last few years to help broaden the horizons for the folks that attend, as well as then we try to scholarship individuals to go to other related type of trainings or experiences throughout the country which we sent people to the National POP Conference, to the American Society of Criminology, et cetera, to help kind of broaden their networking ability to move forward.

I'm sure I've got to be close to out of time if not already. I don't have a clock, I apologize, but let me just summarize that some of the lessons learned, that I hope will be meaningful for all of you. I would — I joke sometimes and say this is, kind of, how to start a movement list. And just based on our experiences, none of which I think are an automatic fix, but I think they go a long way to making efforts, whether you're an officer just trying to get more officers in your department to think the way that you do, or whether you're a person in a leadership position trying to move your agency further in this direction. You got to start with keeping it simple with practitioners. That’s not to say anything about their intelligence. That has to do with their workload and their priorities, so if you can simplify the information for them and cut right to what they need to know, they're far more receptive to it.

The officers that are involved in focused operations tend to be the ones more receptive and most interested in moving forward. Those are the ones that you can kind of tease out that interest and that excitement, that creativity, and push them towards that next step of being proactive by utilizing research data, et cetera. Not to say that your day-to-day shift officer cannot, but those that are engaged whether it's gang enforcement or community crime prevention, those individuals seem to be most apt to want to learn more about other ideas and other ways to be effective, at least in our state.

For all officers we recognize that when we meet them where they're at again and help them confirm the things they already think are effective or help them understand what, you know, makes it tick or what can make it better, receptivity seems to improve. We often will field research inquiries or help requests for project assistance on things that we offset go, “yeah, that's probably not going to — we're not going to have a lot to do with that.” But we'll explore it with them anyway and see where it goes because that's what they're interested in, and we want to — we want to guide them in the direction that they're going, and then move from there.

Leaders and trainers, they need to promote critical thinking versus absolutes. As I mentioned, this "Do something" principle. But whether you're in a leadership role, you're a sergeant, you're a field training officer, or you're engaged in your local academy, engage people in critical thinking. That's how you get people to receive the scientific method and think about doing more and learning from mistakes and, you know, computing, I guess, what else can be done versus just telling what is and how it needs to be, now go forth and do it.

The training, as I already mentioned, needs to have those practical exercises. The officers need the opportunity to observe more than just what they're doing. As I mentioned, getting out and going beyond their own jurisdiction. Our — I don't think I mentioned that at our leadership level, it's a multidisciplinary environment. So it's not just a supervisor’s or sergeant's academy for police officers. They engage in a — through the program with corrections leaders, parole and probation leaders, as well as telecommunication leaders, so that they're learning together the justice system, what it offers, the challenges throughout it, and how those components can all work together and change.

Lastly then, as I mentioned in the beginning, these projects need to align with these goals, these needs of decision-makers. When you do so, anyway, they seem to receive much greater priority.

Real quickly, as if promoting evidence-based policing wasn't enough, CPE then over the years has then been given additional tasks by the legislature. Very briefly, we recognize that legitimacy and procedural justice is a part of the dialogue, so it's not just about crime prevention and community liveability, but it's also about building or maintaining public trust. So much of our training and the efforts we do in the world of curriculum development goes into these topics, and what works, what's known.

We were given the opportunity, is the way I would describe it, to take on a statewide approach towards crisis intervention training as well and develop standards for that in our state and provide consultation, technical support, and training in that realm to help our agencies. We’re pleased to say that at this stage after three years with that program, we now have over 50 percent of the counties in our state holding a crisis intervention team training, and have a steering committee in place to address the needs of those suffering from mental illness.

Lastly then, the newest is the — we call it Statistical Transparency of Policing. This is your STOP data collection related to racial disparities. That's becoming a mandatory requirement in our state beginning next month. And we're there in place, and sided in fact with the legislature as having to devise or provide technical assistance to agencies once these reports are presented in terms of the data on how to make improvements, or analyze and understand their data from the evidence based to the best practice's standpoint. So, a lot of work to come for the Center for Policing Excellence in the state of Oregon for many years to go.

With that, there's my contact information, as well as really the nuts and bolts of this work that I've described to you is our researcher Annie Rexford, and so I put her information up there, as well. Thank you.

DAVID MUHLHAUSEN: Thank you very much.

CHARLES TYREE: Let's just take a second and just thank the folks at NIJ for the invitation to come here today, and yesterday, for that matter, to participate in these two events. I think that they're very important. I'm very passionate about the role of evidence-based policing, in policing around the state of New York, specifically. And I'm very interested, in particular, of learning about what's happening across the country because there is a lot of very good work that's happening around the country. And for me to become more aware of that and to share that information and those lessons learned with the folks that I work with in New York from Buffalo, on the west end of New York, all the way out to Long Island, and everywhere in-between. I feel really blessed to be able to learn from all of you, and the work that you participated in. And as was mentioned yesterday, sometimes that work is, you know, not necessarily found to be truly effective and I think that's an important piece of what we look at when we're involving research into what we're doing. You know, it's not all that what's working, and it's also about what can we improve upon to make it even better.

So that's something we've been involved in in the state of New York through a program that I'm responsible for called the Gun Involved Violence Elimination program. And the program is a grant-funded program in the state of New York, provided through state funds primarily, and there's a couple of exceptions to that that I'll get to in just a little while, but the majority of the funding actually comes directly from the state of New York. We are now actually just a few weeks away, not even, at the beginning of the fifth year of this approach to reduce gun violence. And I think that's a key term that we've transitioned into in the city of New York, calling the GIVE an approach to reducing gun violence, not just a program or an initiative because I think that those titles suggest that they're short term and not lasting and sustainable. And what we're looking to create in New York is an approach to reducing gun violence, primarily using evidence-based policing approaches that have been proven to be found effective and, at the same time, encouraging innovations within those particular evidence-based strategies around the state of New York.

So GIVE has been in existence, like I mentioned, going into our fifth year now. Before we started GIVE, we had a program called IMPACT, called Integrated Municipal Police Anti-Crime Teams, and it was not an evidence-based program. The crimes they focused were very different than what we're focused on now, which is gun violence and aggravated assaults in a few of our smaller jurisdictions. And the strategies varied significantly from jurisdiction to jurisdiction. All that being said, one of the major benefits that we saw through this program was a collaborative effort to all the different partners that are engaged in these efforts to reduce crime. So, this program continued on that premise, that collaboration is a key element to fighting gun violence.

So we fund directly — we directly fund 20 police departments around the state of New York in 17 counties, along with their partner district attorneys' offices, the local sheriff's office, and the local community supervision office. That's direct funding. And you could see the number there, since the beginning of this program in 2014, we've provided $66.5 million to the local jurisdictions to help them combat gun violence in their local neighborhoods. That's just the direct funding. There's other indirect funding that I'm going to discuss as I go through this presentation. But in addition to the direct funding to those jurisdictions, they're able to, through their application process each year to receive this funding, articulate other agencies that they want to collaborate with locally to help them combat gun violence. Some of the examples here, nonprofit organizations, social service organizations, academic institutions, which is a very significant part of it, and others. We believe that gun violence needs to be a truly collaborative effort, engaging all available resources to fight this pervasive problem.

And this is the state of New York map. You could see highlighted in yellow all the different counties, the 17 counties that actually receive funding through this approach to reducing gun violence, all the way from Jamestown, Niagara, and Buffalo on the west coast, if you will, of New York, all the way down to Long Island, including Nassau and Suffolk Counties and the other counties in-between. Those 17 counties have been involved in this program since the beginning and even actually through the previous program, IMPAC, that I mentioned just a few minutes ago.

So, the GIVE Foundation. What is — what is the basic foundation for what we're doing in New York to reduce gun violence, and the first and probably key element to it is our commitment to evidence-based work. Our commitment to looking at what's working, what's been proven to work, and tailoring that to our specific problems in the state of New York, in our local jurisdictions regarding gun violence and aggravated assaults.

A key component to that is education and technical assistance. And Ryan already mentioned their commitment in Oregon to that, and it's — listening to Ryan's presentation really brought me back to thinking about how similar our approaches are, even though this is the first time I've ever met him. But you'll see, we've actually stolen some things from Oregon to use in New York and I plan on looking at some of the other ways that you're look — that you're doing to institutionalize this work in Oregon and using that in our state as well.

Part of the education and technical assistance obviously comes from academic partnerships. I mentioned that. So, since the beginning of GIVE, we've provided an additional $3 million directed towards academic partnerships to help our jurisdictions through education, through training, technical assistance, and research and evaluation. So, another $3 million has gone towards academic partnerships directly and we look to continue that as this program moves forward.

Lastly is the GIVE Network. I really believe that I mentioned in my opening remarks today that there's a lot of knowledge and a lot of great work taking place in policing, in prosecutors' offices, in the sheriffs’ offices, that we need to share with one another. It's no longer appropriate, I believe, to have these things operating in a silo and not be willing to share the good work that's taking place with others. So we've taken a number of steps over the years to promote that GIVE Network, we called it.

So, the GIVE core principles that we have had since the beginning of this initiative is the People, Places, Alignment, and Engagement. So, we require our jurisdictions to focus on the small amount of people that are responsible for committing gun violence in their cities. And research shows that it is a very small percentage of people that are actually willing to pull the trigger and shoot another person. And we are requiring them to direct all their efforts to those people using data and intelligence to identify the small percentage of the criminal population even that pulls the trigger and causes people to get shot. And research also shows, as you all know, that this occurs in very small micro locations even within our cities. So, focusing those efforts not on a city-wide blanket effort but directing the limited amount of resources that are available to police departments in the small areas of their cities where this violence takes place. And aligning all the different organizations, present state, local, federal law enforcement, nonprofits, social services organizations, academic institutions, community, anybody who has a role in knocking down gun violence, aligning all that into a comprehensive plan to address this issue. And at the same time, engaging all these key stakeholders in the area, specifically the community and what you're doing, being transparent, giving them a voice in the strategy of implementation efforts that are occurring in the — in the local jurisdictions, hearing from them about what problems they are seeing, especially as it relates to gun violence, but engaging them and not just blanketly targeting a whole group of people in a small area without at least letting the community understand what the situation is and why you're directing those resources in those areas.

So those are the four core principles that make up the GIVE initiatives.

So evidence-based policing. We require our jurisdictions to implement more than one of the bottom four approaches that have been proven effective to reducing gun violence. Many of you are familiar with all of these. I'm sure hot-spots policing, focused deterrence, CPTED, and street outreach workers.

We also require our jurisdictions to utilize the problem-oriented policing framework to actually look at what is the specific issue that's driving gun violence in their city. And as you can imagine, New York's geography, different demographics, things of that nature make it so those issues are really different from jurisdiction to jurisdiction. Actually, they could really be different even within the same city and different parts of the city. So we encourage them, we require them actually to use the POP framework, the SARA methodology to identify the underlying issue driving gun violence, and then tailor their specific strategy based on that issue.

At the same time, we require the jurisdictions to implement or embed the concepts and the pillars of procedural justice into everything that they do. And this was a change that we recognize after the first year of GIVE, where we had procedural justice as a separate thing that the jurisdictions can do. And as you can imagine, not many of the jurisdictions chose that in year one, so we looked at that and said, "This is a bad message that we're sending, that you can use procedural justice as some separate thing to do." We saw that as an opportunity to have them embed those principles into all of the work that they were engaged in. And I think the 21st century taskforce on policing reports — report that just came out a few years ago really solidified that for us, where the first pillar was basically procedural justice and how you could then look at that into everything else that you're doing in policing, and we feel that's an important piece of the work that we're involved in to reduce gun violence.

So GIVE technical assistance, I mentioned this, and we're talking about, you know, education and training, and I want to dig a little bit deeper into some of these things here, but right from the very beginning, we knew that if we were going to require our jurisdictions to implement these evidence-based policing strategies, we had to do some education on what they were. And we started out even before the first year of GIVE even started, we were out there, conducting this training and engaging subject-matter experts from around the country into our efforts. So you could see, we've trained over 2,600 people in these different — and this is not an exhaustive list of all the different training that we've done since the beginning of GIVE, but we really actually have built on and had a progressively intense program of training.

So we initially started out doing training on a broad regional level where we’d have multiple jurisdictions come in to kind of understand the basics of each one of these evidence-based approaches.

Then we build on that by doing one-on-one type of training, partnering jurisdictions with academic partners to work with them one-on-one on these different evidence-based approaches.

And we've also done even more direct technical assistance with jurisdictions like Albany, Newburgh, Niagara Falls, and Kingston that worked directly with the John Jay folks of the National Network for Safe Communities and the Group Violence Intervention, for example. The Finn Institute, which is located in Albany, has provided direct technical assistance for focused deterrence with two of our jurisdictions, Utica and Schenectady.

We've contracted with Craig Uchida and his folks at Justice & Security Strategies to do two series of hotspot policing workshops, one in the first year of GIVE and one just last year.

We partnered with the National Crime Prevention Council on doing a basic CPTED course and an advanced CPTED course. And then we partner with Facilities Management International to do a train-the-trainer format of CPTED, and that's a model that we're going to increasingly, a train-the-trainer format for this, to build up the local expertise, to have the locals driving the training in their own departments and agencies. I think that's an important piece of it rather than just having these outside people coming in to tell you how to do it, now you're building up your own local experts to help train your own people. It's also a force multiplier, where we can’t — we can’t train, you know, the hundred thousand or so law enforcement officers in the state of New York, but the locals can train their own people, so we've gone to that model with a number of these different trainings and vendors that we were utilizing.

Procedural justice has also been a big focus for us. We've done three train-the-trainers with Lorie Fridel and her group at Fair and Impartial Policing. And implicit bias, three of our jurisdictions are working directly with Blue Courage, on training from them. We've also adopted the procedural justice one and two training that Chicago had in a train-the-trainer format. We now have New York state certified instructors that go around to teach the local jurisdictions in the tenets of procedural justice.

But that's not all, there's many other types of initiatives and training that we've done over the years with other vendors and other agencies to increase the knowledge level of evidence-based policing in our jurisdictions.

So, some of the other training that we are planning to do, and one of the things I'm really excited about, is partnering with Cynthia Lum and Chris Koper from George Mason on an evidence-based policing course for practitioners and trainers. This gives you a 10-week online course via distance learning at a college level, where we're going to have — give police department members participating in this training to, again, increase that local knowledge.

And these are some of the other activities that we have that are going on to continue to advance evidence-based policing.

These are examples of some of our academic partners that have worked directly with our agencies, and I mentioned that funding previously.

Lastly, I want to get into quick, the GIVE evaluation. We thought that it was not just important for us to be looking at the work and encouraging evaluations of the work that's taking place in the local jurisdictions, but also looking at it from the GIVE initiative, as we look at it on a statewide level. So, we partnered with the folks at RIT right from the beginning of GIVE up until the end of 2017. To look at that, they did a lot of work with our jurisdictions in capturing data, which we now continue to use to move the program forward.

I mentioned the GIVE Network already. Very similar to the work that's taking place in Oregon. We've done a number of things, and the DCJS Knowledge Bank, the last bullet there, is actually a direct — we didn't exactly copy it, but we look at the Oregon knowledge bank as a way to share this programmatic information, the good work that's taking place around the state of New York in an easier to use format via the internet. So, that's the knowledge bank.

Just a couple of outcomes to show you the effect that we are seeing on gun violence in New York since the beginning of this program. So, the 2014, and this is January to May, because I wanted to give you the most up-to-date statistics, regarding our shooting violence problem, so you could see, we've had a 23%, almost 24% reduction in incidents, 31% reduction in people injured, and an almost 29% reduction in shooting victims that were killed as a result of gun violence, in that — in that particular timeframe.

Looking at it from 2017 last year to this year, year to date, we're continuing to see really significant reductions in gun violence. Almost 28% less incidents, 35% less victims, and 35% less people killed than we did in 2017.

I'm very hesitant to give GIVE the credit for all of this. I'm very aware of the fact that there is a lot of work taking place within our partnerships outside of the GIVE framework even. And the credit really goes to those jurisdictions, who continue to address the pervasive issue of gun violence in their communities with all available resources, including GIVE.

This slide, lastly, is the rest of the state versus New York City, and if you would have looked at this in 2014, you would've seen that New York was — the rest of the state number was actually up 6%, okay, while New York City saw significant reductions. Now, we're minus 11%. So, significant efforts have been done to combat this issue and, knock on wood, we're seeing results. So, last, we'll hold off on questions until the end.

My contact information is there at the bottom of the screen. And I look forward to questions later on.

ANGELA HAWKEN: I have a good reason for delaying everything and standing up, I've just returned from India, and I'm 17 hours ahead. So, standing is going to help get through — get through this presentation.

I'm delighted to be here. And the reason is that it was in this room in 2014 that I stood at exactly this podium, at the end of the presentation, where I was kind of frustrated with some traditional research I was involved in, and I said to this group — well, not this group, but the group sitting in the room — I have this idea, I think we can do better. I think we should try to do this differently, and I think what we need. And so it was the first — I hadn’t even thought through it very carefully. It was just — I was just, kind of, vomiting on the spot, I think what we need is a center for practitioner-led trials, let's make this practitioner-led because the language I hear about research to practice is really — or the bridging the divide is really about just bringing people to your side of the divide, so why don't we put research into practice, and if it was in this room, and I called it the Center for Practitioner-Led Trial, which by the way received zero dollars in funding, and the first application went to BJA, and they responded and they said they loved it, but it was unfundable by the federal government. It just didn't really fit within funding framework, so I went back to my office very depressed thinking my baby was about to die, and I thought, well, that wasn't a very good marketing exercise because actually when you call anything the center of anything, it just sucks the air out of the room, right? It sounds so academic. And that wasn't the idea, we wanted to create something different. So, that day, I said, if we want to do a better government, right? If we're going to bother having government programs, just make sure they work, we need better government, that means we need beta testing of government agencies, so we needed BetaGov, and I made a call, and I received the first commitment from funding that day, and the idea, really, is that marketing — and I'm terrible at it, but I got a little lucky with calling BetaGov BetaGov.

It's since rebranded to BetaHub because we do a lot of work, and this really primarily stemmed to my interest in government agencies, but we had so many silos that become quite unproductive. So, last year, we launched BetaEd, which is on education, and whether you're imprisoned or in probation, or in juvenile justice, the education side really matters, and this year, we're launching BetaHealth, primarily focused on opioids.

I want to quickly give a shout out also to LEADS, and blame Maureen for, really, all the work I've been doing in the last year, and I'm so — I'm so much older than I should be right now, thanks Maureen. And really, that when she invited me to San Diego to give a talk to some LEADS scholars and we had — really up until last year done no work in policing, and it just led to some amazing introductions. You know, I met — I met Jason Potts and then Jeremiah Johnson, and then the — all the others, who in this room onboarded and we suddenly created this like growing community of practice of people who just wanted to improve outcomes in policing, so I don't know whether to just thank you or not, Maureen, but I'm really grateful to have had the opportunity. This engagement with the LEADS scholars has really created, I think, some of the best products that we have, or at least it's really setting the framework for what this could be, in terms of rapid cycle research.

So, for those of you who don't know us, and if you're from LEADS — or from at least the pracademics that we have, who are from LEADS, my apologies because you can basically give this presentation, but I was asked to give some background.

Most of what we do, and I can see I already have formatting issues, and so I apologize for my crappy slides, most of what we do has never been tested because we've made it very difficult to do testing. And I want to start this by saying, I am really not declaring war on traditional academia, and I think there's a very important role for the more traditional academic model, where we do this — the careful, thoughtful research that does cost a lot of money. But in the front end of that, we should also have space for more experimental research. Just think of it as the front line of figuring out if something is going to work. The private sector does this beautifully, Amazon, Google, Netflix, they do hundreds of thousands of tests a year, we do so few in the public sector because we've made it so hard to do it. We need to break through this unnecessary bureaucracy for pilot level research, and that's what I'm going to talk to you about today, because we can't have all of our ideas untested, when we're trying to keep ourselves safer or smarter or healthier, we can break through that bureaucracy.

And so this is my unflattering characterization of traditional research, and it's true because by in large I mean just look at this, within criminal justice, there's — it took an average four years from the time the idea is had and put onto paper until publication, between four and five years, but health services research is really much worse, anything between five and seven years, and again, this is great for Cadillac more deep theorizing sorts of research, but if you try to do innovation, that is killing innovation. If you don't seize the moment in time when all the players want to try something and strike right at that moment, you're going to lose the opportunity to innovate and lose the opportunity to learn something really important. So, we have to find a way to speed up the more traditional process because that window of opportunity is pretty small. And if you aren't there in time, you're not going to make it.

This is the other problem is that traditional research really is kind of expensive, and when it costs a lot of money to do a study, the result is that you can only do few of them, right? So that's — so most things will be untested, will go on, assuming things work without barely tell us they work, or history tells us. We’ve just always been doing it this way. And expensive research, again, at the exploratory level, creates this other problem, and that is when you only can afford to do a few studies, who gets to decide what their studies will be? In other words, who gets to own what becomes an evidence-based program or practice. We have a monopoly. A monopoly on who produces the research, and who produces the research are the people who can afford to go through that bureaucracy process, right, of creating the expensive research program. We wanted to make it easy for a front line law enforcement officer or a prosecutor in a small town in the middle of nowhere that none of us have heard of, to have an idea, and have that idea tested, breaking the monopoly.

The other thing that we really thought of that was really important was this idea of sometimes something isn't going to be great the first time, and ask Googled this, we have Google on Beta testing right now. It's not always great first time around, but when they are in version seven, it's amazing. In the public sector, first time around, we failed, we're dead, we're all embarrassed, we never speak about it again. Well, sometimes there's an idea and you have to work with it a bit, until it gets to version seven, which is going to change the world.

I really did have to say I’ve never met him, and I tend to quote him all the time, I should stop. Jeff Bezos talked about how he attributes the success of Amazon to how he made it so cheap to do testing. He said the only way we're really going to learn a lot is if we make it easy to do lots, and lots, and lots, and lots, and lots of tests because most things will fail, but some of them will work. And to get to those ones, we have to do many of them. And that was really the model. How can we reduce the cost of doing pilot level tests so we can start to figure out what works and what doesn't?

But that means really kind of creating this — a program of homegrown research. Something else we found — now that we're doing lots, and lots, and lots of tests in lots of different places — transferability isn't that great. Something that works over here doesn't necessarily work over here, and might actually stink over here, but it's still really great over here. Right? Which means we really have to be able to do research in our own backyards, and figure out if something is working for us. Some things will generalize, but not everything will.

We looked and found that the people closest to the problems, the front — if you're in the front lines in a prison, if you’re law enforcement officers out there in the community, you're closest to the problems, and therefore, often closest to the solutions. If we don't make space to harness fair knowledge, we — it's a squandered opportunity. I was — I was in India, speaking about policing, and we're working on a counterterrorism project there. And he said — and this is the head of a major agency, and he said, "You know, we hire people because they're smart. We try to make sure that they're smart. And as soon as we get him through the door, we treat them like a pair of hands. And we have to stop doing that, right? We have to harness the heads of our public sector."

But this requires a new way of doing research because the standard model doesn't work well for exploratory research. And that is, we have to take this staff innovations. We also have to listen to the public we're serving. A lot — our best work comes from client-initiated ideas. And then also look at the data. Sometimes the data can really push you in a promising direction of where you should go look for an idea.

So we launched Pracademia to do this, I've realized there's one small team, we were never going to get to Amazon style testing, we have to harness the power for public sector. We need to create Pracademia, which is basically a practitioner who gets just a little bit of help to do the research that they need at the pilot level, to figure out if something is working or not. So we do that every month. And we now have a growing network of what we call social sector innovators, working across the country right now to try something new. At last count a few months ago, we were at 400, my guess is right now we’re probably at 500 people across the country somewhere, trying and — trying and testing an innovation with us.

We hosted an event in April, some of you will see your faces in this. This is a group of pracamedics who either completed work with us or are working with us. Some of them have — well, one of them, Jeremiah Johnson here, who has the PhD. He's unusual in that crowd. Most of our pracademics have — some of them have college degrees, several of them have high school degrees. And I can tell you, one of the high school degree recipients here, who has run a randomized controlled trial, has run a randomized controlled trial to withstand its own against any substantial randomized controlled trial done under the right conditions that can really produce impressive work.

So this is how it works, you submit your idea to us, to our website. We vet it with you and your leadership. Leadership has to buy in, otherwise this is going to die. It's a one-hour webinar, that's all it takes. So we tell you what you can expect of us, based on the resources we will give you as we work together. But also what we expect of you — and on a point that — when I mentioned the work of Jeremiah Johnson and Jason Potts, said that sounds easy, what we expect of you. What we expect of you is actually a lot. I call it stretch government. Everybody who works with us has a full-time job in a government agency, they're going above and beyond to help their organization become a learning organization. But our expectations have been very clear. This is what it takes to really get this research done. And it isn't always easy.

They're then given a BetaGov team, it's — they get a PhD level person working with them, a statistician, some case manager for day-to-day logistics, and a writer to produce a one-page snapshot, which we, by the way, through research, can now tell you for sure is the magic number of pages for a research document. We — I can tell you because we've studied it. No one wants to read between — they might thank you for the 20-page document. They will never click into it if they know it's 20 pages.

With a team in place, off they go. Some of this work launches very quickly, some of it has to get delayed because it's obviously much more layers of review. Depending on the nature of the pilot, you're coming to us about a text message-reading pilot, we can have that launched within a week. And often we do. If you're coming to us with something more complicated, it might take a few months. Most of our projects were launched within a month or two, once we clear review.

A typical project on average of the model time to completion is three to four months from first conversation to snapshot in hand. We do have projects that complete within hours. And you want to hear about those? I'm happy to tell you from beginning to end, randomized controlled trials, done.

We are fast. And we focus on what you care about.

The other thing we do is try to raise funding from either government agencies or philanthropy so that we are always free for the government entity that we are working with, which is very important to our model. And notice what it says here, "This is your pilot." It's your pilot. This is your research, not ours. We are merely the facilitators, right? This is all about you, not about us. When I first told my team that I was taking on names of all documents that were produced, they nearly had a heart attack because they were in an academic institution. And I reminded them that I have five girls between the age of 18 and 25, living in my home at the time. I'm an — aunt to several people who come from abroad and live with me, and I pay tuition. They did not know who Elvis Presley was. If I have five kids in my house that don't know who Elvis Presley is, do I really care if my name is in that publication? What I care about is effecting change. Which is why we always give credit where credit is due to the creator of the innovation in that agency, who went to stretch government, and did more than they needed to do.

Couple of examples and I'll be quick. Feel free to just give me the thumbs whatever, if I'm — if I'm — we started work — we were very interested in new technologies. We're working on virtual reality, there's lots of applications in prisons — as well as law enforcement training. And some of you in this room will actually be hearing from us because of our partnership with Google on — it's a surprise, and please stay tuned for those of you who will hear from us. We're very interested in just small things on how technology can make small things happen.

One of the first trials we did was on text messaging reminders for probation appointments. We've now done it for court appointments, pre-warrant issue, all sorts of things being done texted via phone. This is the first one we completed, by the way. And it was a text messaging trial done by a woman who has no formal background in research, perfectly, nicely done small experiment, just to see whether the text messaging will make a difference. Her trial has now been replicated in at least a dozen jurisdictions. That trial led to work with adults and juveniles in all sorts of ways. We have found that it works well with some populations and doesn't with others. And it really gave us a quick lesson about replication, when we can expect it and when we can't.

I think the important lesson about this was also how quick you can replicate. Once you create a community of practice, like, we had one thing that held, and then the next thing, something we had 10 versions of that study happening.

I want to make a quick mention of Pennsylvania before moving onto LEADS scholars. Because this is the example of what we'd like to do moving forward. We met — we met Pennsylvania DOC. This is the Department of Corrections about two years ago, and they invited us in to give a presentation to the two top hundred. And we really created a relationship that I think will become the new model for how you create learning organizations within government. The relationship emphasizes innovation and testing, it shows the importance of trying something before rolling it out. We had stopped that agency from doing several terribly bad things. And one thing, unfortunately, we did not get to stop them, and it resulted in an officer death. We had an idea that was put to them, and they just had a recent tragedy. But we — they were about to roll out several practices, and there's small tests demonstrated failing forward that there was not a good idea. So this is an agency that's so nimble now in terms of how it's learning. And we have an entirely expedited process for how BetaGov integrates into their agency.

So here’s — and I'll go through this very quickly — two years old, this relationship, first meeting a hundred and forty ideas. They then opened it up to a bigger group, another hundred and sixty ideas, basically, we're now over three hundred, I think, nearly pushing four hundred ideas that were submitted to us by staff at the — at the department. It quickly triages through — it quickly triages through their general counsel, and we lose about half of them to general counsel. Not feasible. We didn't lose it, but another 30% through our own ethics review and the rest are right there, ready to be tested. So we have really, like, hundreds now on the books with Pennsylvania Department of Corrections. And I think that's the model moving forward. Make it easy for the information to flow. Make it easy to fail forward, shut things down, replicate quickly, and here's some examples of just what we're doing there. It's all sorts of things. It's micro environmental design, smell, color, light, incentivizing positive behavior. They really focus on officer well-being. We're testing Vitamin D, we're testing yoga for officers because that's what they wanted. These are their ideas, not ours. Now we focus on facility safety, optimizing programming, all sorts of things happening very quickly. And it's just considered no big deal because that's what they do. They are a learning organization now, and we just try everything.

This is one of their virtual reality pilots that is really informing work we're doing now across the country, including a new submission we made to BJA on integrating virtual reality into treatment and custody for opioid-use disorder.

There's an example of one from this facility, collaborative design, inmates, mental health stuff, corrections officers came together to test a practice that has worked beautifully. No big deal. We can just learn together.

And LEADS scholars are our favorites. For many good reasons, they tend to be just more sophisticated researchers going in, I think, because of the great training that you do here, so please send us more. You know this from Jason Potts, by all accounts, just a remarkably, nicely done RCT of Automated License Plate Readers. We learned so much about the technology just from that pilot drive, which is not just in terms of what works best but just learn — but just basic data that we just could never have had before. And I lost Jeremiah, but Jeremiah's cruised — multi cruise light experiment is really world class.

We've gone on to work in other countries now. We're now working on policing in six countries. They really like what you're doing here. And it's growing very quickly in other cases because the model is kind of inexpensive and it works, no matter who you are. This is the line of sight underway now in Mexico and Canada, simultaneous study.

And I have a quick — and very quickly, but really make a plug for this idea of failing forward. Nothing useful can be done without the ability to fail cheerfully. And we have to let agencies get to the point where they can fail cheerfully, to figure out what doesn't work as quickly as possible, move that out of the way, so you can focus on the things that do.

So, three great examples of programs we loved, they failed miserably, moving on.

We launched BetaEd last year, and I want — I want to come back at some time in the future and tell you why everything we do there is relevant for criminal justice, if you’re trying to get information or if it's — its training or sharing information, every lesson that we learn, we learn lessons — we have a — we have a user network of 22 million people here so we can learn very quickly. So those are trials that launched and complete in a couple of days, where we can learn very quickly how to help people learn, it's one of my favorite new projects.

Big idea or small idea, please don't lose sight if it's small ideas. We often test the things that would never be considered research-worthy, it would never drop — get to the level of an NIJ application, but the small things really matter. The small things can accumulate. So pay attention. Thank you so much. I think I've probably overshot. I'll clear — get out — clear out of the way. Thank you again for the opportunity to be here.

LOUIS DEKMAR: I was shopping a couple of months ago, and I ran into one of our superior court judges, and he said, "I got to tell you, Chief." He says, "The polygraphist you have, it's just not fair." And I said, "What are you talking about, Judge?" Well, he has a British accent, and every time he speaks, he seems like he's the smartest guy in the courtroom. Angela, I appreciate speaking after you.

I'm Lou Dekmar, I'm the police chief in LaGrange, Georgia and president of IACP. I think as a practitioner I was invited, and that's why I don't have a PowerPoint, is to react to what I've heard. And I know that having been in the business 41 years, and the police chief, 27, that, how research is accepted by law enforcement heads is, kind of, a mixed bag. Some view it much like the fellow that studied swimming, wrote about swimming, jumped in the pool, and promptly drowned.

The fact is that there's sometimes I think a lack of clear adaptability from the research to operationalizing it. And I'll give you some examples. I'll give you some examples that I think helped create the critical view that some practitioners have.

But let me say this. I'm a data guy. I think those of us that have an appreciation for data though, and are practitioners are more involved in fact-based decisions, as opposed to evidence-based decisions. And the reason that we're involved in fact-based decision-making is because the research oftentimes is not caught up to those emerging issues that we're dealing with today.

Let's take use of force in CIT. IACP's appreciation for research, I don't think, can be understated. We've set up the center for police research in policy for the very reason of looking at those things that generally have been accepted as, you know, those best practices, but in fact when looked at critically, have not really been studied.

I had the opportunity to work on an initiative, several years ago, that resulted in what's known as the One Mind Campaign. And we brought together researchers, practitioners, attorneys, mental health providers, psychiatrists, psychologists, and we had, in essence, a round table. And for a day, we talked about what is the best current police response for mental illness. Now, those of us that are familiar with the challenges of mental illness or familiar with what got us here, which is certainly not the making of the police, but rather a failure of — public policy. My colleagues here from IACP get tired of the stats, but there's — it's such an indictment of the way we treat the mentally ill, and I feel compelled to always share them. When we were a country of 150 million in 1960, we had 600,000 treatment beds — in-patient treatment beds — now we can argue about the nature of the treatment. But the fact is it was recognized that these folks needed an in-patient environment. We're now a country of 330 million, and we have less than 60,000, which is why we have the homeless population challenges, and it's why 25% of our fail use-of-force incidents are those affected by mental illness. The three largest treatment centers for mental illness in this country, Rikers Island, Cook County Jail, and Los Angeles County Jail. The Los Angeles County Sheriff serves on our executive board. And I asked him in February, I said, "How many folks you got in your jail?" "Seventeen thousand." "How many are in your mental health ward?" "Five thousand."

I say all that to say this. So we get everybody together into the room and we come up with the One Mind Campaign, which has four components, identify and collaborate with your mental health resources, based on that partnership, develop a written policy to provide guidance to these officers when they respond. Ensure that everyone gets at least eight hours of mental health training. Frequently, what is mentioned is Mental Health First Aid, with an emphasis on identifying and employing de-escalation techniques, and then 20% of the agency receives CIT training, crisis intervention training.

But after we initiate the One Mind Campaign, which I think most folks would admit is a sound program, we recognize that there is no research on CIT. Now — and that's why I say we tend to be fact-based, those of us that looked at data because I can tell you since 2006, I've got just under a hundred officers. Since 2006, all my officers had been CIT trained. That training has not only influenced the outcomes in the department, but I believe in our state. Our use of force cut almost in half, 50% because the same techniques translate over to effectively de-escalating with folks emotionally disturbed, drunk, difficult, on drugs, or drunk — or alcohol. So, those are the facts that I know, but is there research to support?

The critical — I think the critical notion of how police executives look at research is how oftentimes — and that's the nature of research, the findings change. I'll give you a couple of examples. I had a hip replacement in December. Four hours later, I was up walking. A colleague of mine had a hip replacement 10 years ago. He was home six weeks and was — had to sit or lay. They discouraged him from moving. And in fact they said, "Well, that's not the case." The quicker you — I was back to work in six days. He was back to work in six weeks. Same as you recall with those that received heart — an operation or a surgery involving the heart used to be bed rest. Now, before they leave the hospital, they're up and moving. And so I think research, by its nature, attempts to provide answers. But it also has to be looked at with the lens of what is the practical application of this research and how is that research going to affect my community.

We all recall — well, maybe we all don't because you guys here probably weren't in policing in the late '80s. They rolled out DARE. And I remember being perplexed when they rolled out DARE, because I, like, "Oh, man, this is — this is going to be a great program. This is going to deal with adolescent drug use, middle school drug use." And I'm thinking, "Wait a minute. What is this DARE Program?" They get an hour a week for 17 weeks and that's going to mitigate the circumstances and surroundings that they're in 24 hours a day? Are you kidding? So, you know, I kind of poo-poo'ed it. I didn't get the DARE shirt. But that was as a detective. But as a CEO, what do I recognize in terms of the importance of DARE? What are the facts in terms — it's a vehicle to get the police inside the schools and to develop rapport. So, just because a program or an initiative doesn't bear the fruit that we think, it doesn't mean that there isn't residual value. Same with CIT, maybe it will turn out that CIT — now I doubt it.

I'll give you another stat. Washington Post, as you all recall, in 2016 published what they represented to be the number of fatal police shootings in 2015. Shamefully, we have no research on that. So, our best evidence, something that ought to be obvious, we have no research on. So, the best evidence that Washington Post has, I think, the numbers have changed a couple of times, but 992 comes to mind. Nine hundred ninety-two fatal police shootings. Now, prior to that, based on the voluntary submission of information, it was believed to be somewhere between three and four hundred. In fact, it was significantly more than that. And those of us that were taking those numbers and representing them to our communities — because those were the best numbers, then had to, you know, kind of back away from those numbers and then qualify why we made those representations. But it was a failure of research to recognize an important component of policing and provide the kind of information that law enforcement needs.

The whole issue of trust. Community trust is a big deal now. What's the research as it relates to it? Well, I could tell you in my jurisdiction, I was confronted with a situation where I learned that our police department had been involved in a lynching in the 1940s. And that was still a big deal in the African-American community. I'm looking for all kind of research on how to deal with this, couldn't find any. The closest thing I found was a YouTube video of an apology by President Clinton because of the syphilis studies at Tuskegee between 1929 and 1969.

So, there needs to be a clear and vibrant partnership between those in the field, those that are CEOs, and those that are doing the research. And Angela's point that, you know, a research has taken four, five, six years, police chiefs are dead and long gone by the time the research comes out. The average for a police chief, three to five years. You know, I'm an anomaly. I've been a police chief 27 years. But most in Georgia is five years.

Another point that I would make is rely on research of other institutions. Human behavior and organizations generally is the same. Human behavior, period, ain't changed much since the New — or the Old Testament. Read. We haven't changed much at all. The only thing that's changed is the way we do stuff to each other. And there's a couple that I'll share with you.

A Harvard doctor, a graduate of Harvard, I think he taught at Harvard, he was intrigued by the number of staph infections, and the variance between staph infections in some hospitals compared to others. My wife is a microbiologist in a hospital. So, she has really sensitized me to these. They've changed it now she says they call it, they call it HAI, Hospital-Acquired Infections, as opposed to staph infections so they just say HAI. By the way, 100,000 people die a year in hospitals. We kill less than 1,000. If we killed people like hospitals — anyway, my point is he looked — he looked at — he looked at the variances, and what he found is those hospitals that had a strict protocol, checklist, and accountability, had great outcomes. And the culture was everybody had a responsibility for enforcing it. A doctor comes in the room. He starts to put hands on patients. The janitor is in there cleaning the place — "Hey, doc. Get your hands." Everybody's expected to enforce it. What I learned from that was a couple of things, number one, checklist. We have 24. I have a checklist on checklist. Because if we — and I challenge you. Go back, pull three or four burglary reports, are they consistent or does it depend on the personality of the author? If it depends on the personality of the author, that's the responsibility of the agency.

The other one I'll share with you is this was managing malpractice suits against doctors and they found out five things that influence whether or not a doctor was sued even though the outcomes were the same. And they found that doctors were sued less if they did several things. Oftentimes, you show up for your surgery, that's the first time you see your surgeon. He comes in and says, "Hey, you know, what is it, your right leg, your left leg? Let me mark it. Let me initial it, see you after the operation." Out the door he goes. What he finds, you develop a rapport and you connect, people are less likely to sue. What does that mean to us? A citizen comes in to make a complaint, how do we train our supervisors to develop that rapport? How do we train our supervisors to develop that trust so they don't leave there and hold a press conference? And there are four other steps. But — and to be respectful for time, I just wanted to share the — those two studies with you.

I look forward to any questions. I'll just close with this. I commend you all for being in this — in this program. You are in leadership roles now and will be the leaders in the future at a time that is frisky, at a time that's challenging and a time where we have great support from the community, more than you know or appreciate.

And I'll give you another quick stat because I am a data guy.

One year after Ferguson, as you recall, we got beat up a good year after Ferguson. There were half a dozen other bad outcomes that were videotaped. Gallup Poll, I saw it on USA Today. I picked up the paper. Gallup Poll, the headline is "Confidence in the police, 20-year low." Ugh. We were at 62%. I think we went to 51. So, I pull up the Gallup Poll and I look. "Well, wait a minute. They scored 20 institutions. We dropped from third to fourth. That same poll, President Obama was at 44%. The Supreme Court was at 32%. The media was at 19%. And Congress was at 9%. In our worst time in 20 years, those folks would kill for those numbers. And I had to ask myself as a student of our profession, "Why have we — why have we remained relatively high confidence level despite the mess that we've seen?" And it occurs to me that United States, 86% of our police agencies, which confound our international partners, because they don't understand how we operate with 18,000 agencies, but with 86% being municipal agencies, when folks think of the police, and ask about the police, they think what's my relationship? What's my experience? They don't project what they see in the media. And happily, a year ago, I saw — we've had the highest that we've had, 74%. So thank you for what you do, God bless you.

DAVID MUHLHAUSEN: Thank you very much. I'm going to start out with just a few questions to the panel and then I'll turn it over to the audience.

My first is what advice do you have for law enforcement advocating evidence-based policing? This is for the entire panel.

LOUIS DEKMAR: These are middle managers, right?

DAVID MUHLHAUSEN: Yes.

LOUIS DEKMAR: And you're asking what advice do we have for them?

DAVID MUHLHAUSEN: Yes.

LOUIS DEKMAR: I can tell you as a practitioner. There's a model about how you affect change, and the first thing you do is you don't talk about the change to other people in the department, because before you get the support you need, which is going to be the police chief, you need to provide him or her with those filters. There's a phenomenon, I forget what it is but every police officer in here would be familiar with it, you show up on a call, the complaining party gets to you, they give you this horror story, and by God you're going to do right by them, and put the habeas grabus on that suspect. And you find the suspect and you're getting ready to snatch him or her, they're going, "Wait, wait, wait, stop." And eventually they penetrate that filter, and you go, "Wait a minute, they ain't the problem, they are." Whoever provides the filter first provides how that executive is going to look at it. So, if you're looking at instituting change, influencing change, I would recommend playing it close to the vest until you get the CEO on board. You don't want folks badmouthing it, or getting folks — because the last thing somebody wants that's a chief position, is the boat rocked on something that they're not even sure is a good — they got enough issues. They don't need stuff going on inside the agency internally, so that would be my suggestion.

DAVID MUHLHAUSEN: That’s good advice. Any other thoughts?

CHARLES TYREE: Just, I want to try to echo the Chief's remarks, but also for the agencies that are already engaging in evidence-based policing practices and not necessarily seeing results right away, I'm reminded a couple of years ago at a symposium that we held in New York, I'm going to call him Roger Polisky, but I'm probably butchering his last name. But he's a researcher we had that was talking about implementing evidence-based strategies, and he talked about, like, a timeline of two to four years under ideal circumstances to implement one evidence-based policing model in the jurisdiction. And we looked at it from our perspective, we have 17 different jurisdictions, multiple evidence-based strategies that we're looking for our agencies to engage in, how long would that take? And I pretty much decided I was going to retire by the time it was fully implemented. But, what my message is that persistence and not giving up, not looking at it as some just, you know, short-term approach to come out whatever you're doing but looking at it on a long-term scale on how you're going to integrate that strategy into the way that you do business as a police department.

DAVID MUHLHAUSEN: All right. My last question before I turn it over to the audience is — and I want NIJ staff to take careful notes. What can NIJ do better to meet the needs of law enforcement and to communicate the value of evidence?

ANGELA HAWKEN: I think — I think this sort of event is a really great model for the — and you mentioned this in the earlier meeting. The LEADS Program, I hope, is the model for the future. I really don't believe that keeping the practitioners anywhere other than centered to any research project is going to have any long-term legs in terms of just the credibility of the research that's produced. So often we produce research that no one cares about and then we wonder why there's no adoption of the research results. When practice is integrated into the formulation of research, you produce research that matters. I think you're on a great path right here.

LOUIS DEKMAR: I would just  say contract with IACP (laughs).

RYAN KECK: I would just add, the more that NIJ can come out to the agencies, right? And so we've got the reverse going on, it seems, through the application process for LEADS scholars to get in, and get related — more connected. But the other way around is, I think, does wonders as well. Each time that NIJ is at, you know, we've been fortunate to bring them out and meet with people in Oregon, or meet BetaGov in Oregon, and it opens up doors as opposed to them just seeking it out via their own drive, their own research.

DAVID MUHLHAUSEN: Thank you very much. Now I’d like to turn it over to the members of the audience. We have three microphones strategically placed. If you can just say your name and affiliation and ask a question, don't be shy.

GARY CORDNER: Gary Cordner with NIJ.

LOUIS DEKMAR: The chair doesn’t recognize him

GARY CORDNER: I was going to throw one at Lou. I warned him that I would. Obviously, you're in the position with IACP. I — as you and I both know, you were previously the Commissioner with CALEA, the entity that accredits law enforcement agencies. Ryan, you're with the POST. But part of what I'm thinking about is there are some institutions across our police industry, like POST, like CALEA, like IACP that generate standards or in a position to encourage individual agencies to adopt standards and so forth, and we've got a couple of states represented here, and I think part of the reason that, you know, that you were asked to participate is because your states have adopted some initiatives, or in some cases, some requirements that local agencies have to, you know, have to abide by. What do you — what do you see as the role that these kind of either national or state-level entities can play to encourage the adoption of more evidence-based practices by individual agencies? And, you know, is that the — is that the direction that we’re going to see more and more of?

LOUIS DEKMAR: Regretfully, I don't think so. I think at the POST level, I've not seen the kind of leadership from POST generally. I think Georgia and Florida probably have some of the most robust. I served on POST Council on Probable Cause, which we'd listen or review 1,500 cases a year and revoke 600 licenses, so officers were no longer officers. Florida, very similar, but you have some — a lot of states that don't have that kind of oversight, they just have the training component. But I do think that with the technology and the information that POST generate that that is an untapped source for research on police misconduct. As it relates to CALEA, as you know, there's 18,000 law enforcement agencies in the country and less than 1,000 are accredited. The good news is because most accredited agencies are medium to large, one out of four officers in this country work for an accredited agency. And so that is a way to influence behaviors through the accreditation's standard process. So, yes, I do see a significant role for them, for NOBLE, IACP, PERF because if you look at all those organizations, they touch the CEOs in most departments. And I think peer exchange of information at workshops I think is probably the — one of the most effective ways of affecting change.

RYAN KECK: I wish to add, in our state the certification standards don't speak directly to what needs to be taught, but they lay a picture for what education an officer needs in order to be certified. I say that because I think there is a place for POST or Officer Standards to grab hold of the opportunity and define clearer what you want out of police officers. When I talked to leaders throughout our state and abroad, of course we all want problem-solvers, we want critical thinkers, we want decision-makers, right? And we want communicators. But the standards don't often reflect those needs and those expectations of officers. And so I think there is a place to describe that type of thinking. And I think that type of thinking opens the doorway for more concepts like evidence-based policing, et cetera.

CHARLES TYREE: If I could just add New York is not a POST state. We actually — we do have a few agencies that are accredited through CALEA, but we have our own state-wide accreditation process. And just recently, I mentioned the RIT evaluation of the GIVE Program that concluded at the end of January or at the end of 2017. We're now in the internal process of developing our own methods to capture the data — the strategy implementation data from the sites to continue that evaluation and to look at that. And one of the things that we've been engaged in is reaching out to some subject-matter experts and developing a key elements list, if you will, almost like a checklist but not quite. We're calling it a self-assessment tool where here's the 10 key components for an evidence-based strategy and then the components that fall within each one of those key elements that should be present if you're marking that as actually present. And what we've come to realize is some of those have some real opportunity to transfer over to our accreditation standards. So we're in the very, very, very beginning, early phases of looking at how some of those, especially regarding like procedural justice on an [INDISTINCT] EF laboratory put out a report not too long ago on the principles of procedural justice and specific recommendations that they had. You know, and so looking at it, how can we translate those into accreditation standards in the state of New York is something we're looking at also.

TARRICK MCGUIRE: Hello. My name is Tarrick McGuire. I'm the Deputy Police Chief with Arlington, Texas Police Department. And so I thank each of you for your comments and I'm also a LEADS scholar. Several things that you all pointed out that I would like to just kind of ask a few questions about. I think as a — as a police department, there are a couple of things that we look at on an evolving basis in relation to crime. And a lot of times, we see several projects and we see problem-oriented policing projects associated with it, and those two areas are primarily property crime and violent crime. So when you look at technology and other parameters on how those things affect, you know, what we do operational on a daily basis, I think evidence-based policing clearly applies, you know, across those methods. But to Chief Dekmar's point, when you look at different training and when you look at different things that are institutionalized in a law enforcement agency, you know those things are great because you see positive outcomes or results. But you really don't understand probably more so the effectiveness of measuring that and I think sometimes it's because we don't have those metrics in place before we implement programs that would tell us the benefits of those outcomes. But outside of what we deal with on an everyday basis, I think that there are some national contemporary issues that law enforcement deals with right, and I think when you look at, you know, most recently in Texas, when you look at the school shootings, when you look at, you know, the mental health crisis, when you look at all of these national issues that are at the forefront of policing, the criminal justice system, how does evidence-based policing methods apply to those areas? So my question directly is, what do you see the emerging issues, in policing? And how do — how does evidence-based policing apply to those national emerging issues?

ANGELA HAWKEN: I am — I'll make a quick comment on some contemporary issues that we're focused on and maybe say something that will be quite unpopular. I'm increasingly not a fan of the term evidence-based anything. I'm a big fan of the term evidence-generating whatever. And the idea is that we obviously have to focus on evidence and evidence is key, but sometimes when we think about a universe where we really require to go shopping from a list of evidence-based programs, it's quite constraining. So, the idea of taking those evidence-based programs that we know that works as well as you know, from the basis of the most recent science, but adding to that and leaning in and doing as much as we can, on mental health, we've really been doing quite a bit of interesting work with law enforcement now and really getting law enforcement to kind of reconceive its role in the community. And really making sure that all the right players have been in the — have been in the room and together, given the power to create that evidence-generating practice, right, so they're trying new strategies together informed by other public health agencies, the community corrections is there, corrections is there, police is there, and they're creating their own evidence base now, their own homegrown evidence base and I think that sometimes if the evidence base isn't there, sometimes there is science and it's great and you know you can trust that science, but sometimes there's a vacuum. And if there's a vacuum, you really have to be in the business of evidence-generating practices, right, where you have to try to something on your own, make sure you're keeping a close eye on your own data, and then move it forward if it's working or modify if it isn't. We're doing something similar with opioids now, but the idea really is that sometimes it is great literature to lean on and sometimes it isn't. But whether it's there or not, you should still have that evidence framework in your head. Sometimes, it just takes the extra work of having to make sure you get your own data around the issue.

LOUIS DEKMAR: And I think you hit the nail on the head, which is there are a lot of interrelated complicated issues that affect crime that just does not lay at the feet of the police, which is why IACP and the other major organizations have called for a Commission on Law Enforcement Administration and Justice because it's been 50 years since we had the presidential commission on crime. A lot has changed. And it was 50 years before that before we had one, but the last presidential commission resulted in increase in education, training, all the POSTS were set up after that time, 911 center. So convincing the legislature and the administration that this isn't just another research project, but there's going to be action-oriented conclusions and recommendations that will affect the community and improve the safety of the community.

BILL RUSH: Hi, my name is Bill Rush and I work for Lewis-Burke Associates. So, we're government relations firm out here in Washington and all we do is represent nonprofit universities and research institutes. So we are the research side of things and I really appreciate all the practitioner evidence-based knowledge and know-how and programs being developed by NIJ, but I think that universities are very well-positioned to kind of address the threats of tomorrow. And maybe larger multifaceted systemic threats that could maybe be a five-year complete deep dive across campus, different disciplines, bringing everyone together, including practitioners, to address these challenges. And I know that the Defense Department has done quite a bit to, you know, make sure that there are technologies and make sure that there are processes for the threats of today but I'm wondering if there's anything just based on your experience, it's kind of a loaded question, that kind of keeps you up at night? Do you kind of see maybe five or 10 years down the line that if there's an evidence base now or if there's some really people digging into it, I think that — or in addition to persistent problems like the mental health issue as you were saying, that you think academia can really get started on. So how can we really help you right now for the threats of tomorrow?

LOUIS DEKMAR: Well, I'll tell you what concerns me is — and this is a result of some futurists in the '90s and I recall — I teach a graduate class and I recall sharing this with them. And in fact, I looked up yesterday and I found — because somebody else had asked me about it and I found the newspaper clippings. If you recall in the '90s, 20% of the positions in law enforcement could not be filled because of the competitive private sector and people just were not interested in being police officers. And so you saw a corresponding decrease in selection criteria where previously minor crimes, drug use, would result in an exclusion were now accepted. And what the futurists said in the '90s is that based on the hiring practices today, 15 to 20 years from now, you're going to see scandals that rival the 1970s. And the reason is the people you hire today are gonna be in the leadership positions. And so my concern — because I know in Atlanta Metro area, there's 10 to 15% of those positions that they can't fill and I imagine you all have similar challenges with recruitment is that if the public is not aware of how important it is to select good police officers, not just for what happens today, but who's going to be running these departments in the future, that's going to be paying it forward in a way that we're going to want to avoid.

ANGELA HAWKEN: I'll make — maybe make one comment and then flip it a little bit so this will be very unsatisfying to you, but the idea of really where our universities poised for great opportunity, and, you know, the truth is universities have had — had been relatively naughty when you talk about multidisciplinary and interdisciplinary because it still tends to be within the disciplines and then sometimes we convene, but we really have to get that smushed right and when we can do that bonding well, it's really magical. I was recently putting together a proposal around prosecution, and the two senior people I brought into my team were two astrophysicists. And it seems unlikely that you would have astrophysicists working on prosecution, but they're exactly the right people for the job because they know messy data better than anybody and how to extract signal. So I think the teams, even within criminal justice, moving forward, we have to bring in the machine learning experts, the big data scientists, as well as the — as well as the subject-matter experts on the field because we're moving so aggressively forward, the future of data, the five years of now, what looks like the research we're doing — even the research we're doing now is going to look so dated. We have a new project we’re about to work on that's blowing my mind. It's going to quite frankly put all of us out of business. And so we really have to be ready for that. Right? So the future in terms of learning is where the big data — where the tools are, the new data tools are so that you should really start to integrate the science teams into these issues if they're interested in criminal justice. We're going to make sure this field is way ahead when all of the big learning happens and those integrated methods are integrated into our very messy criminal justice system.

HOWARD SPIVAK: Howard Spivak from NIJ. This is primarily for Angela. Angela, I think you know I'm a fan of the work you do. Big fan. But I — it would be helpful for you to speak to how you see the relationship between the work you do and the larger, more cumbersome sometimes work that NIJ is able to fund because there is a relationship, I think, between the tortoise and the hare here.

ANGELA HAWKEN: No, absolutely. I'm still a big fan of the hare — oh, the tortoise, sorry. I'm a big fan of the hare, but also a fan of the tortoise. They're just different. And from the beginning, when I was thinking of this, I thought I wondered — based on my colleagues, too, I want — I just want their work to be pointed in the more productive direction. I think — I think about some massive studies I’ve been involved in that were really expensive, but if there was this kind of formative research, just this exploratory research on the front end, those researchers doing their very comprehensive studies would've been so much more productive. To think of this as really just like helping point where are the glimmers of light where we should be spending the hundreds of thousands, millions of — millions of dollars on? But right now, it's still just in the formative phase so it should cost $25,000 to $30,000 to do this exploratory research. But if there's nothing there, let's move on. It's all right to move on. I think they're completely complementary. And in fact, they're starting to really be sought out now by academic partners, institutions, realizing not only can you get publications out pretty quickly in this model, but also there's something there which we can create the pilot data for when they are sending an application to you.

That said, I'd like to see NIJ write an RFP that I can respond to. I think this is important and I've never seen one that I can respond to that makes sense because I'm not going to be able to tell you today what the theoretical framework is because I don't know what I'm going to study today because I'm going to jump in and leap on something important when it arises. It just doesn't work with the traditional funding world, but I think we're going to be very helpful down that, and we're still very new, right? And when we have enough going out there where we can start to really produce kind of, you know, typologies of what's really, really working well and where should be — where should we be concerned about replication, but the best gift we're going to give academia very soon and I can't wait to write it for you, is to tell you what the natural failure rate should be and how many of your studies you should be killing because if you aren't killing one-third of the studies you fund within the first few months, you're being irresponsible. We kill at least one-third of what we start because we know within a few months in, this is not doing anybody any good and it should shut down. So, I think we'll have those lessons  very quickly, but others might learn from too, but the natural data is very important.

DAVID MUHLHAUSEN: All right. Well, with that, we are going to conclude this event and I thank our panelists and everybody for attending. Give a big round of applause to everybody.