From the Academy to Retirement: A Journey Through the Policing Lifecycle

Ellen Scrivner: Thank you, Kris [Rose] … You make it sound like I can’t hold a job. 
                
[Laughter.]

There were a few you forgot in there, and we’ll just let that go ...

[Laughter.]

Thank you for that nice introduction, and even though Laurie [Robinson] is not here, I’d like to comment on the fact and thank her. This is a really big day in her life, and for her to take the time to come here and speak to us, even briefly, really not only says a lot for what kind of person Laurie is but for the investment she has in OJP and NIJ, but the investment that she has in science-based research. So we’re delighted that she could come, and that was one of the reasons we delayed starting, and I’m sure you would have preferred to hear her than rather have Kris and I dance around up here and try to keep this going until she got here.

But it is really my pleasure to serve as the moderator for this very distinguished panel of researchers many of whom needs no introduction — I don’t think any of them actually need an introduction — and to introduce you to the National Police Research Platform that is just one of the many exciting projects that NIJ is supporting. But this project differs from other policing studies. Kris made reference to some of them in her comments in the fact that it’s a longitudinal study across multiple sites, and it collects systematic data about the life course, not life events, but the life course of the individual police officer, their supervisors and other organizations —they are talking about three different levels there. This type of information, or it is this type, that will help us really establish benchmarks for excellence in American policing. And it gives us a far better understanding of what happens in that journey — that journey from the academy to retirement — the journey through the police lifecycle. This project is of particular significance for me since I spent the early part of my career involved in the psychological assessment of police applicants. We collected a lot of data on what candidates look like at the time of the application, and we generally followed them through their probationary year primarily to validate the screening process. And then we also did a lot of work in police stress, organizational stress, organizational practices and police tactics — all of those kinds of things have been looked at, but we had little data regarding what actually happened to police individuals and to their organizations throughout the life course of the career. That’s going to change with this project. And now we can put that all together so that we can develop evidence-based coherency to our thinking about what really happens in the real world of policing.

We think this research will clearly advance our scientific knowledge base of what we know about American policing, and that knowledge is going to be of direct benefit to police officers and to their organizations as well as to the communities that they serve. Most importantly, we will now have some solid evidence about what works and what doesn’t work in the shaping of police officers, their supervisors and their organizations. And that’s going to be very important — referencing back to the comments that Laurie made about a new day in terms of leadership of police organizations. This is the kind of information that’s going to be very helpful to them. This is going to be a three-year demonstration research project that will be carried out in selected jurisdictions — you’re going to hear about that as the panel presents — and we are fortunate to have this outstanding team of nationally known researchers who will be working on the project at different sites. They are also advised by a group of police practitioners who help keep things centered in a highly realistic and practical way while not interfering with the research. And what you’ll see is that they are also being advised by a number of other leading researchers in the field. But today they’re going to be talking to you about the work that they are doing, and when they are finished, there will be time for questions, answers and comments.

The panel and presentation order are Dr. Dennis Rosenbaum, the principal investigator and project director who is a professor at the University of Illinois at Chicago, Department of Criminology, Law and Justice and is also the director of their Center for Research in Law and Justice. He’s going to provide an overview of the project and some of the work that started in Chicago where Dennis is not only very well known but very well accepted as a researcher. He will be followed by Dr. Gary Cordner, equally well accepted, co-principal investigator and professor at Kutztown University in Pennsylvania. He’s going to be talking about the research at the recruit level.

Both Dr. Lorie Fridell and Dr. Steve Mastrofski will then talk about research with the supervisors. Dr. Fridell is co-principal investigator and co-director of the study, and she is an associate professor at the University of South Florida. Dr. Mastrofski is co-principal investigator on this study, is a professor at George Mason University and former chair of the Department of Administration of Justice at George Mason. He’s followed by Jack McDevitt who’s associate dean at Northeastern University’s College of Criminal Justice in Boston, and he’ll be covering the police organizational part. And Dr. Amie Schuck who is an associate professor at University of Illinois at Chicago, and associate director of research at UIC. She’s going to talk about some innovations in police training, which I think you’ll find pretty exciting. When they finish you will clearly see why we call this “Research in the Real World.” So before Dr. Rosenbaum initiates the proceedings, please join me in welcoming the panel.

[Applause.]

Dennis Rosenbaum: Thank you, Ellen. I’m Dennis Rosenbaum … A couple of things … I just want to thank everybody from NIJ — Kris Rose is very instrumental in helping us make this happen— we really appreciate that and Jolene Hernon for organizing the communications, and making it happen at that level. And of course Laurie Robinson — the support she represents with regard to integrating research and practice is phenomenal, and that’s exactly what this project is about. And of course we have Winnie Reed and Brett Chapman, who have been instrumental in making the project happen in the field, and of course Ellen who has been a guiding force throughout. We thank you for being a moderator. She’s a seed of wisdom here in this field.

Lots of people … I can’t begin to thank you. I’m delighted to be involved in this team of researchers — there’s many other people — the list keeps expanding as we move forward … Lots of people we can talk about; I want to thank all of them — some great police chiefs and others.

NIJ created a meeting in November of 2006 to talk about the future of policing and what kind of information we need over the next 10 years — a number of you were there. This project emerged from that meeting; it was a two-day meeting. Lots of big questions, like what will policing look like in the future? And how can organizations adjust? And what kind of research needs do we have? You might not be surprised, but the police chiefs there were in unanimous agreement, as well as the researchers, that it’s surprising how little is known about their own organizations, the people they employ, what’s going on nationally. Other than uniform crime reports, we don’t have any uniform data, really. What kind of leadership do we need in the future to deal with the challenges that police face today? What kind of training programs? What kind of supervision? There’s some fundamental questions that are being addressed in this project, and we thought it was a great way to bring it all together into one project.

Accountability issues … Things are changing; as you know our society is changing rapidly. We have information technology that is driving our social relations but also driving, to some extent, policing in America — the surveillance systems, the accountability systems that we have in place. But police are in a very fascinating position — as you know a challenging one — where we expect so much of them — to protect our liberties, to protect victims of crime, to take the repeat offenders off the street, to reduce violence. There is just so much pressure on them to do so much. But as we change — we know we have a multicultural society now; we have lots of immigration — the nature of the expectations has only increased. We no longer expect police to just be effective in fighting crime; we expect them to do so in an equitable and fair way. So that’s a big theme here as we move forward.

There are lots of knowledge gaps, and we are going to propose filling them. The previous speakers have already alluded to the fact that we need some generalized knowledge about police personnel — the employees, supervisors, executives — not just the case studies that all of us have done, which are very useful — but so that departments can see how other departments of similar size and type are behaving and responding as organizations. We need in-depth knowledge about these organizations. We’ve got some great data being collected through LEMAS and other mechanisms, but we need to dig deeper, and you’ll hear about that today. We need standardized measures and benchmarks. I find that police executives actually want to know how other agencies are performing and not just on UCR but on other things. And we need data streams, which we’re starting to evaluate innovation — you’ll hear about that as well.

These are actually the five presentations that you’re going to hear real quickly — about eight minutes each — the life course of new police officers, the life course of new supervisors, the organizational life course, training issues, innovations in training. And I’m going to talk briefly about community-based measures of performance, but I’m not going to go into that right now.

What’s new about this? We have new voices in shaping the agenda; this is a coalition of lots of people. We’re involved also in the Harvard executive sessions, so we’re exchanging information there. We’re involved with the UK’s National Police Improvement Agency. Ed Zedlewski, I don’t know if you’re here, Ed … Ed and Winnie have helped with the dialogue with them, and we’re hoping to interact more. We have New York University folks — Tom Tyler, George Kelling — there are lots of people who are connected to this project. Joel Garner is beginning to do good work with the folks at BJS and law enforcement , and we hope to have a mutually beneficial relationship there. We’ve met with the COPS office — we are trying to see what we can do jointly in the training area … and lots of other people.

Quickly, there are different tiers of data collection — some are very intensive and some are not so intensive that involve all the sites. There are 25 sites around the country — big, small and medium. The big ones are like Boston, Chicago, Los Angeles, Houston. Many smaller ones … and different types — there’s some sheriff’s offices, there’s also a tribal jurisdiction out in Arizona, Fort McDowell. Lots of benefits to practitioners we believe are from this project — learning about the research findings in general about patterns in policing — but also we hope to give them a mechanism for communicating and sharing information. A robust Web site that will allow them to download data about their own agencies and others, and we can begin to discuss organizational excellence — what are the standards? Kalia is doing some work in this area.  I know Gary Cordner’s involved, and Rob Davis at RAND — we’re working with them too. But we’re trying to establish some benchmarks for organizational excellence and what that means.

I see this as an opportunity as, I think Laurie was alluding to this, to develop learning organizations where researchers and practitioners work together to share information, to share data and to have feedback loops that help improve their organizations along the way.

Researchers, I can tell you there’s all kinds of benefits — lots of different datasets that will be available for them to work on at different levels. We will document trends in police practices through the life course. We test different theories about life course of police officers.

I do want to stress the life course thing a little bit just briefly to say that we have … no one has looked at policing in this way. In the health field we have, in the military we have — the life course trajectories, the career trajectories of different officers. It’s not just factors in life course analysis of getting married, having kids, getting a job — those things are critical for young police officers — but also within the career — where they end up working, where they’re assigned, whether they have critical life-threatening events, whether they’ve been disciplined in their mind unfairly in the organization, critical events, who their supervisors are — that may affect how they turn out as police officers. So we want to watch that we care a great deal about the life course of these individuals so that we can help make their jobs safer, healthier, happier, and in the end, that will affect how they interact with the community, which we care about a great deal in the 21st century.

The capacity for multilevel analysis … If you have an officer who’s embedded within his own family, peers, embedded within a particular police working group and police district, within a particular city, within a particular region of the country — you have all of these multilevel effects that can go on to shape that person’s behavior. So this project through a national platform has the capacity, eventually, to deal with that.

Rigorous tests … We’re committed to the most rigorous measurement, new methods, new measures — and I think that’s probably my last slide — and we are doing randomized controlled trials. Professor Schuck will talk to you about that as well, so I don’t want to take other people’s time.  Let’s get them started, and we’ll leave time for questions. Thank you.

Gary Cordner: Thanks. I’ll talk briefly about the data-collection efforts that we’ve got under way and are going to be growing, that start with police officers when they first join their police agencies and first get to the police academy.

As indicated, the focus really of this part of the project is on new recruits and questions like what happens to them over their life course — as Dennis has said — as they develop both in the academy and then also when they leave the academy, in trying to identify what kinds of factors of all sorts — the kinds of events and life experiences that Dennis mentioned, the kind of training that they have, the kind of field training and supervisors that they have, and all other kinds of factors — try to figure out how those affect how those officers develop. And then try to see if we can tie that then eventually to outcomes — what they actually do as police officers, how hard they work or don’t, what kinds of decisions they make, what kinds of production they have in terms of all kinds of activities. For example, we’re interested in trying to figure out which factors related to their background, their personal characteristics, the training that they’ve had, the experiences that they start to have when they get out on the job — what kind of factors like those and others affect their productivity, their reliability, the kinds of interactions they have with the public, the levels of stress, and health and well-being that they experience both initially and as they go through their career; and there are things like job satisfaction, morale, their decisions about whether to stay with the agency or not, decisions about whether to pursue special assignments, whether to pursue promotion, and all of those other kinds of organizational issues.

A number of just different theoretical frameworks that we are using to try to guide what data to collect and then how to use it and how to analyze it, include the life course theories that Dennis has talked about, developmental theories. Obviously Dennis mentioned the multilevel analysis that we’ll eventually be able to do because we’ll be studying individual officers, but they’re part of groups and groups are part of organizations and so forth. I might also mention that they’re part of generations, and there’s so much interest in the police management world about generation X and generation Y. [Break in recording.] ... is in generation Z, you know, these darn kids these days — what’s on their minds, what are they really trying to get out of this, how long are they really going to stay?—all those kinds of issues that bedevil old geezers like me, and even ones younger who are in charge of police departments — and then fitting that also into various organizational theories, to figure out how that all adds up in different organizations to different outcomes organizationally. These are among some of the different outcomes, of course, that we’ll try to look at — their productivity, the quality of their work, and so on and so forth. We do definitely want to try to focus on decision-making as well. Part of Amie’s presentation will talk about some special training efforts in Chicago to affect how officers decide and how they interact with members of the public and trying to track how that’s affected by their development over time.

So where are we going to get all this data? In all the sites where we’re studying recruits, we are primarily relying on self reports — basically questionnaires and surveys — questionnaires that they fill out before the training academy actually begins and then that they continue to fill out on a periodic basis through the academy, and then after they graduate and get out into the work world. The initial surveys in this phase of the project, like others, are typically paper and pencil when they first get there, but as quickly as possible we try to transition them to online surveys and get them in the habit of completing these online surveys on a regular basis. Obviously we’re thinking that eventually that will be the most efficient way for us to continue to collect data from a large number of people spread all over the country. And so that’s where we’re trying to get them online as quickly as possible. In the intensive sites, which would, at this point, include Chicago, Houston — primarily for recruits — besides collecting information from the recruits going through the academy and then after they graduate by the surveys that I’ve mentioned, we’ll also make observations; we’ll collect information from their instructors in the academy about the recruits; we’ll collect information from their peers also from their FTOs (Field Training Officers) and supervisors. So we’ll try to get that so-called 360-degree data about their behavior. Also, we will be collecting information from the public — and Dennis I think will talk a little bit about this at the very end of the presentation — to find out how the public assesses their behavior once they’re out there working in the community. And we’ll also be getting as much information as we can from departmental records — all kinds of departmental information about their performance and about their behavior on the job. Those are in the intensive sites.

As mentioned, a battery of surveys — either prior to their arrival at the academy or the very first moment that they get there, and then weekly online journals similar to online surveys, and then periodic surveys after they get out on a regular basis. And then also in addition to all of those surveys, mainly online as a means of collecting information directly from recruits, there will also be some videotaping of recruits in the Chicago site, and this might eventually be something that we can do also in other sites.

The online journals serve multiple purposes — document critical events, try to gain insight into how their thinking about their experience, the job that they’re going into, how they think about the community, the role of the police and so forth — document their views on a wide range of issues. Basically we’re trying to make these as pain-free as possible for all of these young police officers to fill out. We try to make them interesting; try to give them a sense that we want to know who they are and how they think and what they think about the experiences that they’re having.

The current status of this is — in Kentucky, which is one of our sites for recruits, but it’s not an intensive site — I’ll explain that briefly — we’ve begun last month with their recruit class number 411 collecting data from the entire class. Actually the second class in Kentucky started this week, and they pretty much have a class that starts every month year round on into the future. We’ll be tracking as many of those classes as we possibly can in this project. It’s relatively early, but so far we’ve got over a 90 percent participation rate from the recruits that have come into the process so far, including several weeks of the weekly online journals, as they’re called.

If I could just mention briefly about Kentucky … What we have in Kentucky is a statewide academy that serves the entire state other than the three largest agencies in the state. So they don’t provide recruit training for the Kentucky State police, Louisville or Lexington, but they serve the other 400 police agencies in the state. So officers come to this academy from all different agencies, mostly fairly small, which include sheriff’s departments, and Kentucky has 120 of those. It also includes some state agencies other than the state police, for example, Kentucky [Department of] Fish and Wildlife [Resources] and some agencies of that sort. So we’ll get quite a range of officers from different kinds of agencies passing through these academies. Chicago is beginning their data collection this month, and next week I think, next Monday, and other sites will be coming on in 2010.

And just to summarize, these are some of the benefits that we expect to get from the recruit data collection and following the life course of recruits: identify predictors of different kinds of behaviors — good and bad factors that affect their retention and so forth; try to identify key periods in an officer’s life course when they may be more likely to make certain kinds of decisions or have certain kinds of experiences; and try to begin to get a sense of what constitutes good policing, good behavior as far as we can tell. Lorie?

Lorie Fridell: Steve Mastrofski and I want to talk to you about the component of the platform whereby we’re looking at new supervisors and charging their course through their careers.

Why study new supervisors and their life course? Well first of all this topic came out of early discussions with our primary agencies. The police leadership in those agencies said we need to know about our supervisors, our first-line supervisors, and we need to do a better job of developing them. And in fact just like the military, first-line supervisors are the linchpin between the policies, the goals, the aspirations of the management and what actually happens on the ground — on the street. Unlike the military, the police profession has not been as strong at developing this role and maximizing its potential. There are issues in selection, in training, in professional development and so forth. So what we are doing is we want to understand who these people are, what they do and to what effect and how they grow and develop throughout their life course. We hypothesize that there are some early supervisory experiences that may impact on the quality of their performance, on the skills they develop.

We also want to keep track of this changeover from the mindset that goes with being that line officer and part of labor versus that person who needs to represent management and move those line officers towards the goals and aspirations of the top leadership.

This is also an opportunity to look at leadership trajectories. We are going to have first-line supervisors; some of which may leave policing after they’re promoted; some of which may be first-line supervisors throughout their career; others are going to be moving up the chain of command. It’s really important for us to be looking at who takes what trajectory and to what end so that we can be developing the future leadership in policing and facilitate quality leadership at the top. So we are tracking new supervisors right now in Chicago, Los Angeles, the Kentucky Academy and Houston. We’ll be doing this in the supplemental agencies because one of the things about the supplemental agencies in Kentucky is we can get small- and medium-sized agencies.

There are multiple methods and sources of data. First of all, we’re getting some data from those newly hired supervisors. They are completing surveys, both paper and pencil and online; they’re involved in focus groups, involved in interviews. We also will have surveys of subordinates and superiors, and then we’ll get information from the organization surveys, management surveys and department records that are also supporting other components of the platform.

This is kind of a simplistic but visual overview of the type of information we want to collect. On the right, some of the elements of the supervisory role; on the left, some of the hypothesized influences of those various elements.

Let me focus in on one source of data, and that is the surveys that are completed by these newly promoted supervisors. They are given a paper and pencil test on their first day of new supervisor training; they then move to online surveys — they’re completing one at the end of training. And then there are very quick snapshot surveys every month thereafter, and then the more significant surveys at six months, 12 months and of course we hope beyond. As I mentioned, they start with paper and pencil, and then they move to the online.

I want to turn it over to Steve who is going to present some preliminary results from the agencies and the people who have been involved. A cautionary note that these are preliminary data; as the agencies participate more, I expect some of these outcomes will change. Steve?

Steve Mastrofski: We’re just going to show you a small sample, just a little taste of our survey results. And in this we’re going to distinguish the responses of supervisors from large departments compared to those with small departments so that when there are differences you can see what those differences are.

One of the ways that organizations become effective is to articulate the goals of top management or the leadership and then to try to get the workers, in this case the police officers, to internalize them and to embrace them as their own.

This slide identifies four different policy objectives that we surveyed supervisors about. The top one, we’re asking them about policies about violent crime, then preventing crime, then analyzing and solving neighborhood problems, and getting the community involved. And as you can see, large majorities of supervisors embrace the priority given by their leaders to reduce violent crime and prevent crime, but many fewer are as supportive of their department’s priorities for community policing and problem-oriented policing. The ambivalence about engaging the public can perhaps be better understood by noting that the vast majority of supervisors do not feel that residents understand the problems that they face. Also you’ll see significant numbers of supervisors see aggression as more useful than courtesy in certain areas of the jurisdiction — at or nearly half feel that way. And as many of you know, research shows that whether the police show courtesy is a major influence on whether citizens comply or cooperate with the officers in the field, and this is especially so in neighborhoods where officers might feel the greatest need to be aggressive.

So if it’s community policing that top management in these departments hopes supervisors will inspire, these results at this stage suggest they have a bit of work to do here in building confidence in the community in that regard.

Getting supervisors to internalize top leadership’s values and priorities is one way to get people to do what you want. But another way, and one that we’re all familiar with who have worked in bureaucracies, is to just get the people to be good soldiers or good bureaucrats — that is, to follow orders. To what extent are these supervisors likely to be good soldiers? How willing are they to follow what the organization expects of them in the way of the rules and regulations? To what extent do they agree with them? And as you can see, while the majority of respondents appear to be good soldiers, a substantial portion is not oriented that way. For example, almost 40 percent of the supervisors in large departments do not agree that officers should follow rules that are viewed as wrong. And when it comes to governing their own behavior, between a quarter and a third of the new supervisors felt that doing the right thing sometimes requires violating the rules.

Very interestingly here, whether these results should be viewed as good news or bad news depends in part of what kind of organization you’re trying to put together — whether you want an organization that emphasizes good soldiering or something that is willing to bend rules, or shall we say make exceptions. And so we’re not making a judgment at this point, but what we are doing is trying to see what we’ve got.

Many departments provide little or no training on how to supervise, as Lorie was pointing out — but all the departments that we have studied so far — the new supervisors surveyed did so — and they trained from a handful of weeks to a couple of months.

And just to put this into a larger perspective … As Lorie mentioned, the U.S. military, from which I came before I became an academic, invests a lot of resources and a lot of time in training people to become supervisors —to take on more and more leadership roles at each level. And so what these organizations are giving one shot at, the military typically does at each level as you progress up, and of course at the commissioned officer level it goes even a lot more — the service academies, ROTC, Officer Candidate School and postgraduate school. So that puts it in a little bit of perspective.

We asked supervisors to assess the training that they just received, and we found striking contrasts between small and large departments. Those in small departments are far more impressed than those in large departments with the training that they got. One caveat here is, I don’t show you the slide, but the large department supervisors had received, at the time that they were trained, a lot more formal education and formal training then had the supervisors in the smaller departments — and that could be one this is that they are more sophisticated, more educated, or at least they feel that way. But for the people who are devising training in those organizations, it might be that they want to pay attention to that and accelerate or advance the training a little bit more.

We also tried to get a handle on it by taking a look at reactions to certain items before and after training, and I’ll just show you a couple in the interest of time here. This chart looks at the supervisor’s perceived role as someone who should promote management’s goals to their subordinates. Responses before training are then compared to responses at the end of training — both small and large departments; you note that there are increases in a positive orientation of supervisors toward this pro-management role. In this sense, training here seems to be turning the supervisors more toward a positive orientation to management.

Here’s another one that takes a look at how supervisors felt about top management support for stimulating initiative among their subordinates. And as you can see here, there really wasn’t any change from pre- to post-, and it’s not statistically significant. So this suggests, that at least on this item, if what you’re trying to do is to get your supervisors to stimulate more initiative, then maybe some work here needs to be done if training is going to help promote that.

The last slide we’ll consider tells us about the structure of the workgroups that first-line supervisors oversee. Stable workgroups are an essential element in any effective organization, again going back to the military and studies of World War II, and since then it shows that that unit is the key to effectiveness of the organization. And that those units that have stability, where you have the same people in that group operating with each other, who get to know each other and with their supervisor are going to be far more effective. So we were curious about just how much stability there was, and this chart gives you an idea where what you see on the top — both small and large departments, the majority of supervisors work the same shift schedule as only half or fewer of their subordinates. And this problem is especially severe in large departments. So you’ve got a lot of supervisors who don’t regularly see the same people, or if you look at it the other way — they don’t regularly see the supervisor or work with them. And we think that that is an area — that this survey suggests already — these departments might want to be looking into with full realization — because this is supposed to be real, get real, that these are not easy things to change. We know that a lot of stuff is negotiated that affects this.

So in conclusion, I’ve given you just a little taste of what we’re looking at here. And one of the things that we will be doing is the same kind of thing with these supervisors’ subordinates and with their superiors so we will be able to give you a broader perspective, and we’ll be tracking that over time. Thank you.

Jack McDevitt: Good morning. If this seems a little complex, it sort of is… and to try to go over all of this stuff quickly to get to the different components of it is a little challenging — so the idea would be that there will be opportunities afterward to talk about any of these things in more detail.

You’ve heard about the recruit survey; you’ve heard about the supervisors’ survey; this is the third life course survey or the survey of the organizations. It came out of that initial session that Tom [Tyler] had organized at NIJ saying that we don’t have a lot of good information about police organizations, and we have very little about how officers who work in those organizations feel about the organization itself. And so that’s the goal of this is to sort of give us some information about what do the employees in police organizations think about that? And also to provide a little bit of context for the supervisors and the recruit surveys that you just heard about. What we tried to do… now think about this… think of the people up here — we all want to know everything, right? So we’re trying to figure out what are the concepts, what are the dimensions that we can measure, and we can start to understand? So what we did is take a look at the literature on police organizations. We pulled together some focus groups of chiefs and talked to them about what is it in your organization that you want to know. We looked at different kinds of research that were ongoing in the field in this area and talked to programs and institutes around the country about what they were doing. And we tried to get a feel for what are the dimensions that we can measure. And we came up with — as you’ve seen in this thing — not one or two, but 10.

So here are the first five. The dimensions that we’re going to measure across all of these different organizations are first of all leadership. What do the employees think about the leadership in their organization? The supervision, obviously, as you’ve just heard, then accountability. This whole issue that came up in Cambridge, the Gates incident — the use of discretion — how do you use discretion? How do employees think they are rewarded or not for using their discretion within the organization? Integrity and discipline — hugely important. What do they think about the integrity systems in their operation — are they fair? Are they something they believe in? And then views of the operations — what are the priorities of this organization? Do they jive with your priorities? And as you can see in the supervisors, we’ve got that theme going across — are you trying to advance the goals of your organization? But also from all the employees, do you know it, and do you agree with it?

External linkages — that’s the top five — these are the second five. External … We’re looking at what are the linkages to the community or to other agencies? Do you know about them? How do you think about them? Infrastructure … Police will never have enough infrastructure. We’re going to measure that and sort of say  do you need better cruisers and all the other levels of infrastructure. Hugely important are the human factors — ideas about stress. How stressful is this job? Do you have opportunities to deal with the stress? Do you know about opportunities? Going through those dimensions — fatigue, life routines and all of that, and trying to do that. Relations among the members … We’re going to talk about the police culture. What’s the culture in this organization? How does it support you getting your goals as a member of this organization — achieving your goals? And also then finally, the structure. Now you may say, “OK, how are we going to do all of this? Are we going to stop being police officers for a year and answer the survey?” No. The way we’re going to do this is first of all online surveys as we’ve said before. This is going to be all an online survey project as it rolls out across. And one of the things we’re going to do is that all of the employees — and a lot of you in this room are very familiar with this — are not going to get the full survey. We’re going to divide the survey into components and then sprinkle it across the employees of the survey. So when we aggregate together, we’ll have the survey responses — no single person is going to be asked to talk about all of the 10 dimensions and all the questions about that. The goal for this, as you can see there, is that all of our surveys are going to be 10 to 15 minutes; we are not going to be so onerous or so burdensome that they’re not going to want to ever do them again because the idea is to keep coming back and providing some information back to people. So that’s one of the important things.

Now one of the interesting things we’ve found so far as we start to talk to agencies is that police chiefs are really interested in the answers to these questions about how their employees feel — they hear about it on an individual basis, but something systematic is something really new, and they’re looking for it to the point where a number of our agencies have said we will give them time at work to complete your survey. We’ll send them to our computer; we’ll give them time off their shift, and we’ll let them complete their survey that way. One of the things you may say is, “Well, will they do it? Will they go and fill these surveys out?” We don’t know. We think we have some idea that they will.

We did test two sites, and as you can see, we did it over a two-week period, and the response rates came up from the beginning up to the end — 93 percent in the first agency to 75 percent in the second agency. So you say to yourself, “Well maybe, at least in these two agencies, we got pretty high participation in these organizational surveys.” Then you say, “Well, is it only the ones that complain or the only ones that are happy?” And if you look —and these are data put together by Wes Skogan who’s another member of this team — this is the agency and then the survey. So we have the demographics of the agency — the number, percent, sworn male, sworn female, civilian male and civilian female — that filled out the survey. And you can see in agency RF the responses are pretty comparable, in OP they are comparable but particularly when they get to civilians, the ends are so small that we’re not able to do that — as we come and understand this more, we’ll have better incentives and better ways to talk to the agencies to do that.

One of the things that I mentioned is that as we talked to police chiefs, this set of data, the organizational data, is something that they see as a real value, that this is something that can help them manage their organization. They want to see it in their organization and maybe how their organization compares to organizations that are similarly situated, size wise. They want to know about what are the concerns of their officers? And what are the training topics that could be identified? And where are the policies not supporting their goals? And those are the things that came through in the focus groups we’ve done with police chiefs so far about how they would use this information.

The schedule for this is we plan to roll this out beginning of 2010, and all of our agencies … One of the things that’s really important is that we get these results and give them back to the agency — the agency wants them; they want to see them.  They want to know, “We put this time in; what does it tell us; what did we learn?” We got to get that back to the agency, and then we’ve got to get it back out to the field. And what are we starting to learn about employees and police organizations and how they’re working? So that’s what we’re really looking forward towards as we turn over 2010.

Amie Schuck: All right, so I’m going to talk to you today about recruit training, and the first thing I want you to do is to take a look at this picture. This represents an incredibly important issue to us, and I’m sure most of you are aware of the situation between Sergeant Crowley and Henry Louis Gates and the subsequent Beer Summit here in Washington with President Obama. But that’s just one story, and the one I actually want to talk to you about is one that happened in March of 2009. A video showed up on YouTube that shows a young Dallas police officer lecturing an African-American male in the parking lot of a hospital. In this particular video, the young African-American male is trying to explain to the officer that his mother-in-law is dying in the hospital and that he needs to get in there to see her, and the officer ignores the young African-American male’s explanation and proceeds to lecture him and give him a ticket. In the subsequent period his mother-in-law dies. And subsequently the chief apologizes for the incident and the officer resigns.

In another video … it shows an Oklahoma state trooper getting into an altercation with a paramedic from that particular county. The altercation occurred when the paramedic driver refused to yield to the police officer when he was riding to another call. Subsequently the trooper came back and pulled over the driver of the ambulance, and they got into another altercation and eventually the paramedic was arrested. Now this was all filmed by a patient’s son who was actually in the ambulance at the time that this encounter happened.

Now the question is what do these three incidences all have in common? Well when you look at these incidences you have to ask yourself, how did these incidences get so out of control? These were relatively minor incidences between officers and citizens, but somehow they went from very minor incidences to these incredibly major altercations. Now one might say these are just isolated incidences that happened. Actually this is from an actual local probability sample in a Midwestern city that asked people about their perception of police rudeness during traffic stops. And as you can see, 57 percent of the African-Americans felt that this was a big or somewhat of a problem in this particular area. So these aren’t necessarily isolated incidences. Now I’m sure this doesn’t come as a big shock to many of you in this audience.

The question is how can we prevent these kinds of incidences from happening? And that’s really our objective under the training model. We’re going to try to train officers; we’re going to try to train their skills, their habits and their attitudes, in order to deescalate these potentially hostile situations. Now for training, what we did was we went through all of literature and said, what are the best practices? What do we know about how citizens evaluate their experiences with the police? [Break in recording.] … of five elements that we found in the literature that consistently predict people’s assessment of their interactions with the police. Did the officer let the citizen have participation or a voice in what was going on? Did the officer show dignity and respect to the citizen? Did the officer show good intentions or motives? Was the officer knowledgeable and competent? And did the officer remain neutral in that particular situation?

Now what we did is we took these elements, and we said we are going to develop specific training strategies for each one of these elements for officers. And what we came up with was what we called the Quality Interactions Program, or QIP training. Now this training program has five different dimensions. The first dimension is what we call quality communications, and this basically trained officers in interpersonal communication skills. For example, active listening. How do you show the citizen that you are listening to them? We also are telling them how not to engage in bad communication skills — don’t interrupt the citizen, don’t blame the citizen. We also focus heavily on scripts. So we have a module where we actually sit down with the officer, and we develop scripts, that is, things that you can say like, “I’m sorry, ma’am; I know you’re really upset. Now I can’t do this particular thing, but what I can do for you are these particular things.” And with these officers, we go through and we give them some example scripts. And we sit with them, and we have them actually develop their own scripts. And the scripts are then laminated and put on a card that they carry with them. So if they’re in an encounter, an engagement, these scripts should be habitual — they’re just things that they should automatically say out of habit.

Our second component is emotional control — this idea of being able to stay neutral. Officers are trained to control the situation; we want to also train them to control themselves, to understand the role that emotion plays when you’re in a situation and things start getting out of control and some acute stress strategies to be able to get yourself back into control so that you can make rational decisions.

The third component is competence and confidence. For this particular screening strategy, each recruit is actually videotaped in an interaction with a citizen, and an officer goes through the tape and actually gives them systematic feedback in terms of how they handled this particular situation. We think that practice and feedback are one of the keys for recruits to learn these skills. We think about it … They go to the shooting range and spend a lot of time learning how to shoot a gun — well they actually practice doing that. It should be the same with their communication skills — they should actually practice doing them.

We also have a component on decision-making, that is, what are these key decisions that you are making at these critical moments?

And our last component is on resilience, which is really the chronic stressors — that is, how do you handle having to go to the same house, the same domestic call, time after time after time, and how is that affecting how you’re interacting with both those individuals and other individuals like them?

These are our methods. We have a wide range of methods based in adult learning. As you can see, all of it is really rooted in this idea, again, in practice and feedback — that this isn’t just talking heads — people standing up here like me telling you what’s important, but then actually getting out there and practicing and getting feedback from their peers and training instructors in terms of how well they’re doing. We obviously have some lecturers; we have integrative exercises; we have stories; we have case studies; and we have modeling. Now one of the unique things about this project is that it actually will be systematically evaluated. We have a randomized controlled trial design; we have pre- and post-test assessments, and those assessments include both written assessments, self assessments, but they also include videotape assessments that they’re each videotaped and are coded for their videotape and communication skills.

We have a program implementation assessment, that is, was the program delivered actually the way it was supposed to be delivered? And a treatment and integrity assessment, that is, did the control group accidentally somehow get some of the treatment? And we’re going to assess both the short- and long-term effects of this. We’re going to assess both immediately after they get the program, as well as these people are part of our recruit study, so we will continue to follow them and continue to see if this training program had an impact on their interactions with the public, both six months out and 12 months out and as far as the study goes a long.

Mastrofski: No study of police organizations would be complete if you didn’t at some point talk to the top leadership of the department and find out what they think ought to be going on and what they’re trying to accomplish. We are therefore going to be interviewing the top leadership in each of the 24 sites, and the questions that we ask will be open ended; we expect to get detailed accounts. I understand that we’ve already completed one — and are about to complete another of these — and got a lot of information. We think that we may have to go back in some cases because the responses may take awhile to get — it depends on how loquacious the chief is. We also may go on, depending on what information we get from the chief, and interview others in the top leadership as well.

The chief serves many functions in a police organization — the CEO of the organization — but certainly what must be primary among these is actually governing the organization, and so that’s really what we’re going to focus on. We are interested in the directions that the chief is trying to take the organization and what priorities the chief has. We’re also interested in how the chief tries to get things done and what things are available to the chief to accomplish those goals to implement.

And finally, of course, we’re interested in matching up the consequences to the chief’s objectives. So breaking this down just a little bit, we suspect that we may hear many of the same themes in general from each of the chiefs, but we think in the particulars, particularly with regard to priorities — after all the essence of leadership is to make hard choices — and so you can’t have everything, particularly in today’s economy. So we’re going to be very interested in the kinds of things that they emphasize in terms of priorities. We also want to learn whether the chief sees himself as someone who is continuing along the same trajectory as his or her predecessors or somebody who’s taking or establishing or going in new directions, new turf or even backtracking and saying I’ve got to undo some things that are being done.

One of the things that we really don’t know much about in the literature is tracking individual departments and the leadership over time to see the pathway of leadership and how it goes and how much backtracking there is and so on. We do know that those things happen.

And finally, of course, we’re interested in the kinds of things that influence the chiefs’ decisions, and here are some of the things that we’ll be looking at. With regard to the kinds of authority and resources that are available to the chief, here are listed some of the things that we’ll be asking about — the scope of their decision-making authority and a variety of different resources that are available to them for what we presume would be their desire for effective governance. And no study of leadership would be complete without focusing on the individual as well, so here are some of the things that we’ll be looking at — as well as asking the chief to indicate — what special skills and character they bring to the job.

And finally we do know that it’s not an easy job, and so we’re very interested in the kinds of leadership challenges they have faced and that they anticipate facing over time in the future.  And of course these can be internal, things like dealing with what the rank-and-file want, collective bargaining units and so on. And external … a variety of different interest groups always interested in pointing the police in one direction or another, sometimes opposing the chief, and some of it takes effort to get almost anything done, particularly if it costs money.

And finally, we’re interested in outcomes. While we’ll ask the chief to assess the outcomes of his or her leadership — what they think the outcomes will be — we will have the advantage because of the comprehensiveness of this study of actually comparing what they tell us to what we learn from all these other instruments that you’re hearing about. So we think it’ll be an interesting kind of thing to see the world from a chief’s perspective but also double-check it from the others. Thank you. Dennis?

Rosenbaum: Yeah, just real quickly because we want to open this up here… You know one of the things about this project is we really are trying to push the envelope with regard to measurement of things that are important beyond traditional measures of police performance. And we’ve talked a lot about community-based measures but not done enough about it over the past decade. This is an oversimplified view — there’s the general public whose opinions matter, who have some contact with the police but not always directly; victims and all those others who call the police; and then those various folks who are stopped, cited or arrested who have what we call police-initiated contacts. They all have maybe different views.

These are the traditional measures — the big four — this is all we keep in police departments. We actually keep a little bit more than that, but this is the way organizations have been measured by traditional narrowly focused crime control outcomes. We need a new system that measures what matters to the public. If you remember 10 or more years ago, NIJ had a several day conference on measuring what matters and put out some important documents. It was great, but it never really resulted in actual new measurement systems. We’ve done various surveys over the years, but this project tries to systematize that to develop reliable measurement at the officer level — small geographic areas — and measure things that are important to the public.

Legitimacy … We could all talk about police legitimacy today — and you heard it, how we avoid Beer Summits at the White House, procedural justice, car side manners, all of these things are important. I should also give Steve Mastrofski credit; he has a very important paper on … What is the title of it, again? I forgot. Yes, Policing for People for the Police Foundation — it ties into this theme very well, and Wes on our team and others have done work. Tom Tyler’s work is important — bottom line — how do citizens feel about how they’re treated during these encounters? We are going to do that, and we’re already beginning that. We’ve put a lot of resources into developing surveys, and we even had the big room full of computers and servers and software programs, that I won’t talk about now, that measure all of these things listed up here. But go beyond procedural justice also. There’s a whole focus I want to say on victims — very important, the emotional needs of victims, informational needs of victims — police have encounters with victims. And so this project will systematically measure the quality of those encounters, and I want you to visualize this as not just being one agency, but eventually a national database where we can compare police departments of different sizes and types on the quality of their encounters with the public.

And so we’re doing that now with Web-based surveys.  We’re just about to start in one city in January. Automated telephone surveys, and eventually we’ll validate this with independent phone surveys to see about the response-rate biases, etc., that we get. And we’re going to focus on traffic stops, asking the drivers … we’re going to focus on crash drivers, and crime victims — various types — everything from domestics to burglaries. So you’ll find out how victims feel in a more systematic way.

Scrivner: Thank you, Dennis. Thank you to all of our panel. It’s an exciting project, and a lot is going on in your work. I’m going to save my questions for last or maybe not at all because I want to open it up to the audience. Craig Fraser, do we have a microphone? Or do you need one?

Craig Fraser: It’s interesting because I find this really fascinating, but there are five interesting things that occur to me. First of all, this is partially based on the experience of being in 200 departments doing studies over the last 15 years or so. One is the impact of unions, and I’m just coming from the last three days in the Northeast department where the union has an incredible impact. If you think about some of the external things that influence officers’ behavior from start to finish — the whole context. And it looks like you’re going to have jurisdictions that are going to be in the South where there are no unions so to speak of, but also some that are highly unionized — that’s going to be, I think, a tremendous impact.

Second thing, I think, the other difference between training structures where the departments do their own training and really have control and states like Kentucky where it’s a centralized training — I think that’s going to be important.

And something else — Lorie alluded to this too — and that’s the difference between where supervisors have no training and whether it’s formal training that’s set out at the outset. So I hope as you look at the departments you make sure that you include some where there is virtually no supervisor training and then you need to get the surveys the day that they’re promoted not the date that they just started training because sometimes training might not be for six months.

The fourth thing that occurred to me is looking at the sort of context of the broader governmental structure because we see this frequently. We see that there is a difference between a mayor, a strong mayor, sort of the Northeast and Midwest that have a police commission, ones that have a city manager, and I didn’t see any mention of sheriff departments — I did see that you have Hillsborough County, but I think that you want to make sure that when you’re looking at that … the impact of being an elected official in a sheriff’s department I think that’s important.

And the final one that would be really interesting as a variable is looking at the impact of the diversity of people in the organization. I suspect there’s a real difference in officer’s outlooks and maybe in organizational performance between organizations that are very diverse in terms of gender, ethnicity and so forth. And Gary and I were talking about how some departments that have almost no females at all. So those are sort of five things that would be really interesting that as you get the data to look at combining back and forth.  But this is exciting stuff; this is going to be a lot of fun.

Scrivner: Very significant feedback. For those of you who don’t know Craig, he’s with PERF. And he is the guru of management studies there so he’s been out in the field a lot; I’m sure that has framed your comments. Anyone on this panel want to respond? Gary?

Cordner: Just to pick up briefly on your point three — see how many can remember what point three was.

[Laughter.]

It’s even more complicated than you said because you’re absolutely right — some people become sergeants that don’t get sent to training for six months. In some places, they send officers to sergeant training, and then they don’t get promoted for six months, as well as the places that don’t do any at all.

Scrivner: Anyone else have any comments?

Fridell: Well I want to make sure that it is emphasized that even though you saw a list of large cities, one of the values of this platform is that we are and will continue to expand into the medium and smaller agencies to get at exactly the supervisors that have been on the job for a year without training and so forth. So I think that we have the Kentucky agency and then each of us have smaller sites — I’ve got one that has, I think, 16 total personnel because policing research has not been able to look at the small and medium, and that’s the real value of this project.

Fraser: [Inaudible.]

Rosenbaum: I just want to add, all of your points are outstanding by the way, and only when we get the full national sample will we be able to make some of the comparisons. Part of the project now is to develop the measures around these things. What is the role of unions and their influence? The diversity thing is a big focus. We have several young scholars that are not here today at different universities on the team that are looking at the tolerance for diversity as well as the diversity. So within the organization, how well are women, minorities and others are integrated into the organization and feel comfortable in that organization? So we’re looking at a lot of measures of that as well.

Scrivner: OK, yes …

Audience Member 1: You mentioned that some of the sites or jurisdictions would be intensive; I’d like to know what defines “intensive”? And also you said that for those sites you would be using, or there was be a possibility you would be using, performance work statements, so I’d like to understand the value or importance of using that type of information for those particular sites and how they are defined.

Cordner: Thanks, I’m actually going to punt that in a second to Dennis or Lorie. By contrast, Kentucky is not an intensive site partly for the reason that Craig gave — my recruits and sergeants going through the training in Kentucky come from like 400 different agencies. They come there; they get their training, and then they go back to their agency. So we haven’t tried to get the agreements from all the different agencies that they work for, that we could then do 360 and get work records and all that. Someday down the road that would be great, but that would be enormously complicated. So the intensive sites are more like Chicago and L.A. where we’re working with an individual agency that also runs its own training. So I’ll let them talk about what they’re doing.

Rosenbaum: Well, one of the things I didn’t mention because there is another subcommittee on existing police records, so how can they be used in more innovative ways? Police collect a lot of data, as you know, some of which probably are not that useful, but some of them have not been thought about in different ways as indicators of organizational excellence or not. But at the individual level too, performance evaluations may not be that useful because, as you know, some of those are pretty standard across the board, but in the cases that are more intensive, we will actually be having our own instrumentation where we ask their supervisors to evaluate them on dimensions that we think are important. And so it will be new data — confidential data — that may have some variation among officers rather than the standard five on a five-point list. And also the officers — it’s a voluntary consent, but once they consent to participate in a long-term study, we will also have access to some of their records — citizen complaints, various things like that.

Scrivner: OK, there’s one question in the back and then two here. Can you state your name please?

Lyndon Diaz: My name is Lyndon Diaz with the Bureau of Justice Statistics. I’m wondering if you’re looking into mentoring or shadowing and things like that. And another thing is with police public contacts, in the Bureau of Justice Statistics, we do the National Crime Victimization Survey and there’s third-party victimization — there’s always witnesses and things like that, so I’m wondering are you’re looking at that too?

Rosenbaum: First, on the mentoring thing, yes … There’s a classic study in 1987 by Rosenbaum showing that in Detroit when we did a randomized trial training of the officers, that the effects wore off after four months in the field. So don’t forget everything you learned in the training academy and hope that you don’t have Denzel Washington as your FTO — for those of you who saw whatever that movie was … But I think, the answer is we are very much looking into the idea of expanding out with regard to innovation and rethinking FTO and all of that work in the field and the mentoring. Emotional intelligence is one of the variables that we’re looking at in police organizations — the idea that you can give people good supervision — your first-line supervisors as well as your FTOs. How they interact with these officers is going to be critical, especially in these larger departments, but I think in all of them. So we want to document that as researchers — what kind of supervisors and FTOs they had and get their views as well. But we also, in terms of reform, we’re thinking of expanding out the training that Professor Shuck spoke about in terms of combining it with FTO training, or not, in a randomized trial, to determine whether it has a supplementary effect. In terms of the third-party victimization, our surveys of the community have mostly focused on one individual victim encounter at this point and their experience with that police officer. There are lots of issues there, but I won’t go into that, but down the road we certainly could expand.

Scrivner: Is there another question here?

Linda Hammond-Decker: Yes, Linda Hammond-Decker from the Bureau of Justice Assistance. I’m interested in knowing in the leadership portion of your research, whether or not you’re looking at the impact that the chief’s selection process has on the stability of the agency and the turnover rate of leadership. Also, the impact of the government type structure — strong mayor versus city council, etc., on the leadership portion. Is that something that’s being looked at?

Mastrofski: To answer your first question, we certainly are asking the chiefs about the process that they underwent for selection. Ideally, we would get that from a number of sources including, for example, if they used headhunters and so on, and the government were willing to allow us to interview them — that’s something we could pursue. Right now we’re getting that mostly through the leadership survey — what they went through, what do you think people were looking for and so on … We can probably also get that by talking to some of the community leaders — the mayor, the city manager …

And your second question, are we going to look at the structure of the city government? That’s easy to do since we’re going to know that. I think in addition to sort of structural things what we find is that, a city manager government we think operates a certain way, a strong mayor government we think operates a certain way, but in fact there is great variation within them. So more importantly perhaps than the structure — we certainly will know that — is the nature of the interactions they’ve had, the guidance that they get from those people. Guidance is sometimes another word for interference depending on who you’re talking to.

[Laughter.]

We use very technical language to allow a chief to say whatever is on his or her mind. But we certainly can look at those things.

Karen Bachar: Hi, Karen Bachar from the National Institute of Justice, and this is fascinating. I have a very tangential area that relates to this — well not so tangential to me — but victim services and victims. And what you’re talking about in doing interviews with the community, you’re talking about individual level people. You’ve also talked about the fact that law enforcement rules are increasing in terms of the responsiveness to people, including victims. A lot of law enforcement officers or agencies will partner with victims services organizations to refer, to help the victim. And I’m wondering if anything in your research design, when you talk to victims, is going to include that part of how that linkage was happening between the law enforcement — their sensitivity to victims — and how they use community resources to help improve the lives of victims.

Rosenbaum: We do ask. We certainly ask how sensitive they are to victims on all of those dimensions. We do also ask about whether they made referrals to particular services — it’s  a couple general questions at this point because we’re trying to keep it brief. We also, from the agency standpoint, ask police officers in the organization to what extent you’re involved in any partnering at the ground level rather than just assume what the police chief says is what’s going on — so we get a sense of that at the organizational level. But we’re certainly open to learning more about how we should be wording those questions and what are the key things we’re missing are.

McDevitt: Just to follow up, I think that’s part of the external part — the external linkages on the organizational survey — and that’s key in your work for human trafficking — what kinds of agencies do they believe are partners and what kind of agencies do they think are not good partners for the organization. So I think a lot of that can come out of both of those places — the community piece and also how the employees feel about different external linkages.

Cordner: And just to add one more point to that that’s probably obvious … In a typical study that we’ve done in the past, we would have sent one survey to the agency, and either the chief or some college student doing an internship would have filled out the survey trying to represent what the whole agency does. Well we’ll be asking some of the same questions; but we’ll be asking everybody in the agency and probably get a richer and a more complex answer.

Scrivner: We have one question in the back, and then we’ll have time for about one more question. And I think that’s going to go to Kris Rose, appropriately so. Your question, please?

Audience Member 5: A two-part question … With regard to survey participation by new recruits — is that voluntary or mandatory? And secondly, are there provisions for anonymity and confidentiality when survey questionnaires require psychologically sensitive information?

[Break in recording.]
                     
Rosenbaum: [Break in recording.] … themselves play a critical role. If they buy into this, and they give the officers time to do this kind of stuff — they seem to be very positive about that, especially — I don’t know if enough has been said by Jack and Gary and others — the small- and medium-sized departments are just very excited about this. The bigger ones are too, but it’s more complicated. Yeah, we’d love to do this — go talk to the lawyers, kind of thing. And so we’re working all of those details out. More time is spent there than the smaller ones we have, but they’re saying,“Hey, we want you guys to do this; go take time to do this.”

Scrivner: Well, I want to thank the panel and particularly for your enthusiasm. It’s very noted how excited everyone is about the project and about where it’s going. Thank you very much for your willingness to come here today and share with us the real world of policing and what we know about it now and to participate with us at NIJ. And I’d like to thank the audience as well for all of your participation and good questions. Sorry we were delayed in getting started; we’re running just a little bit late now, but we’ll get you out in time for your luncheon arrangements, so thanks again.

[Applause.]