Alternatives to Traditional School Discipline - Breakout Session, NIJ Virtual Conference on School Safety

 -Good day, everyone.

Thank you so much for investing your time with us today.

My name is Basia Lopez, and I'm a Social Science Research Analyst at NIJ and will moderate this day-one breakout three session.

As the other breakout sessions of this conference, this one also highlights a series of presentations on school safety projects funded by NIJ.

We have an amazing panel on alternatives to traditional school discipline where we will be talking about findings from four NIJ-funded studies under CSSI.

First, presenters will share information about their projects, and after the presentations have concluded, you, as the audience members, will have an opportunity to ask questions of the presenters.

Please use Q and A option at the bottom of your screen to enter your questions, and we will address them at the end of all four presentations.

So first, we will have our distinguished guest, Lama Hassoun Ayoub, with her presentation on A Randomized Controlled Trial of Restorative Justice in New York City High Schools.

Then we will have another distinguished panelist, John Engberg, who will be presenting on his project, Pursuing Equitable and Restorative Communities Initiative, A Piecework Restorative Practices Program.

Then we will have a team of our distinguished panelists, Kimberly Kendziora and Juliette Berg.

They will present on the Impact of the Safe Public Spaces in School Program on School Safety, Student Behavior and Discipline Events.

And our final speaker and distinguished guest, Naomi Goldstein.

She will be presenting on Dismantling the School to Present Pipeline While Enhancing School Safety, the Philadelphia Police School Diversion Program.

If you have technical issues or other questions, please use the chat option.

If otherwise, post your questions to Q and A.

Thank you, and I will give the floor to Lama.

-Good afternoon, everyone.

Thank you for joining our session.

My name is Lama Hassoun Ayoub.

Thank you for the introduction.

I am a Senior Fellow at the Center for Court Innovation.

I'm very happy to be here representing a large team to talk about this study, A Randomized Controlled Trial of Restorative Justice in New York City, specifically Brooklyn schools, high schools.

We are grateful for the support of NIJ for this project.

Make sure I can...

Okay.

This is the agenda for the presentation.

I'm going to provide some background on restorative justice in New York City.

I'll then talk about our research questions, the research design, process evaluation, the outcome evaluation, and it says "Questions" here, but we'll be holding questions until the end so...

First, let me talk about restorative justice, and you'll probably hear a lot about restorative justice during this conference.

I don't think I need to review the literature on zero-tolerance policies.

We all know or should know or someone else in this conference will mention that the '80s and '90s saw the expansion of zero tolerance and exclusionary discipline and that there were...

We know now that there have been significant inequities, and there continue to be significant inequities both in how those practices are applied and who is bearing the brunt of their consequences.

Restorative justice has been gaining ground, sometimes called restorative practices, as a solution to those things, and the number of schools and districts adopting it has certainly been moving faster than the empirical work supporting it.

There have been numerous qualitative studies and, recently and probably during this conference, some results from rigorous quantitative designs that are beginning to point...

beginning to paint a picture of what the impact of RJ is looking like.

But what really is restorative justice, or what are restorative practices? Although it's very common, I don't want to frame restorative justice as an alternative to exclusionary discipline, and many RJ practitioners and trainers will tell you that it shouldn't be framed in that way.

Instead, I want to use these quotes from...

And these are publicly available, they're not collected-in-our-research quotes...

from RJ practitioners to talk about what restorative justice is.

So New York City Restorative Justice Initiative defines it as "A set of principles and practices rooted in indigenous societies.

Restorative Justice can be applied both reactively in response to conflict, and proactively to strengthen community by fostering communication and empathy." So what we're talking about really is a whole school shift, shifting the mindsets of everyone in the school and challenging their collective approach to community and accountability.

RJ asked us to abandon everything we know about punishment as adults, about everything we've been taught about punishment since our childhood, not just in our schools, but also in our society, in our own families, with our own children.

It asked us to focus on relationships and community perspective, taking accountability and empathy.

Kay Pranis, who is a national trainer on RJ, a circle keeper and facilitator and someone who I've observed her trainings and also participated in them, says that, "We have raised an entire generation without the prerequisites for developing empathy and then were outraged when they seem not to care about the impact of their behavior on others.

We did not consciously decide to raise them without empathy, but that is the result of significant changes in our social behavior." And last, I'm going to quote Dr. Kathy Evans, who wrote "The Little Book on Restorative Justice in Education," who says, "Restorative justice can't just be a set of things that we do.

It has to be a framework for how we view teaching and learning." And so I just want you to keep these things in mind as I move on and talk about what RJ implementation looked like in our schools and what outcomes we can expect to see.

So New York City schools, I'm not sure if any of the other speakers will talk about this.

So I'm just giving kind of a brief overview, but similar to Philadelphia, in 2013 we started to see a decline in incidents and suspensions in schools.

In 2015, the Mayor's office and the DOE announced a series of initiatives to end exclusionary practices.

One of them was the expansion of restorative justice practices, and they've started in 2016 with District 18 which, at the time, was the highest suspending district in the city.

Now, New York City is a huge school district.

It's the biggest one in the United States, and within it, there are 34 distinct sort of neighborhood districts that are school districts, and District 18 is just one of those 34.

Our study is built off of what was happening in District 18.

So I'm going to start with the research questions, the first one being, "What is the impact of RJ in schools compared to standard non-restorative approaches to school discipline on student outcomes?" And I'll talk about what those are, what outcomes specifically, later.

Our second research question is, "What is the impact of enhanced RJ in schools compared to standard RJ training?" And so I'm going to define "enhanced" progressively throughout this project, but essentially what I mean is, "How does a well-invested and hopefully well-implemented RJ approach differ in outcomes from what is most often done which is a standard RJ training for a few school staff, but not necessarily additional resources or intensive support?" And research question three is really about implementation, and so "What are the challenges, best practices and lessons learned for implementing RJ in schools through a whole-school approach, particularly in schools"...

and people characterize this differently, but I just put it out there, schools that are really "struggling with historical and structural oppression?" So they're in neighborhoods with high crime rates, historical divestment from both the neighborhoods and the schools and things like overpolicing.

So as I said, District 18 is located right here in purple, in Brooklyn.

It includes the neighborhoods of East Flatbush and Canarsie, and the DOE offered basic Tier I RJ training to all the schools in this district.

They required five people and a principal to go to a 5-day training, and so everyone in the district or at...

All the schools in the district got some exposure to restorative justice training.

And so we specifically focused on high schools, and we removed any non-eligible high schools.

So those included a few schools who had already started restorative justice or something similar in years prior, and we also removed schools that were involved in other research studies.

We tried to work collaboratively with our research colleagues so that our work was not overlapping too much, and we ended up with a list of 10 schools, and we randomly assign them to either a treatment group or a control group.

Now, in our control schools, like I said, they got just the standard thing that was happening across the district.

DOE offered RJ training, and then they got ongoing support from this district-wide RJ coordinator, and they can...

They could do whatever they wanted sort of once they had that RJ training.

And our treatment schools, they were exposed to the same thing, but they got this very intensive restorative justice in schools team that I'm going to talk about in more depth when we get to the process evaluation.

I just didn't want to repeat myself twice.

So in addition to having treatment and control schools, we decided that since our control schools have a level of RJ exposure and could implement RJ successfully, potentially successfully in their schools, we decided to bring in a group of comparison schools from nearby Districts 17 and 23 with no RJ exposure.

So we worked with DOE actually very recently to identify schools that we knew had not undergone any RJ training and were not implementing it in their schools, but were from nearby neighborhoods.

So District 17 actually includes the other part of Flatbush, for example, and District 23 is also nearby, and so these districts have similar racial-ethnic demographics.

They have similar economic statistics to District 18, and you'll see that in a second when I get to the schools.

So ultimately we have five schools in the treatment, five in the control group and nine in the comparison group from those two other districts.

Here you can see what the sort of the school and student descriptions are.

So they're very similar to each other across the three study groups, right? So we have an average age of 16 across all three study groups.

The racial-ethnic breakdown of the students is very, very similar in these schools.

You can see that the all...

In all three study groups, our schools are predominantly Black students with some other racial-ethnic groups that are essentially minorities.

The schools are slightly more male than female, just a little bit, and at the bottom, you'll see that they are mostly...

consist of students who are economically disadvantaged.

They're all above 80 percent.

The comparison group does have the highest percentage of students who are economically disadvantaged.

So I know this might be a little bit confusing, but again, the treatment and the control groups were randomly assigned, and then we brought in sort of a matched comparison group from nearby districts to account for RJ exposure within the control group.

So here's our research design.

We have a process evaluation and outcome evaluation.

I'm not going to get into it too much right now because I want to get into what we learned from the process evaluation.

So first, this is what the implementation actually looked like, and so I talked about an RJ in-school team that was more intense than a standard approach.

And so each school in the treatment group had a full-time restorative justice coordination and support from two more floating coordinators that were across all five schools, as well as a project coordinator and a project director.

They implemented the program by conducting literally thousands of relationship-building circles which occurred weekly in the schools.

They conducted response circles or response approaches like mediations or circle-mediation hybrid when conflict emerged, and they spent a lot of their time also doing non-circle activities such as circle supports, so that is preparing people for circle, or climate support activities such as, you know, starting student clubs and supporting students in other ways.

We collected a lot of data, and during the process evaluation that will take years to publish on, but I want to focus in on the interviews that we conducted because I'm going to report a little bit about what we learned from those interviews.

We conducted interviews with 124 school staff.

Seventy-five percent of them were in the treatment schools.

Twenty-five percent were in the control schools, and this is a little bit about who they were.

You can see most of them were instructional staff so teachers.

We conducted interviews with all of the principals and other school administrators when we could, as well as other school support staff.

Most of our sample had participated in a circle, but about at least a quarter of them had not, sometimes more depending on whether it was a treatment school or a control school, and many of them had also independently facilitated a circle.

I'm actually going to skip over this in the interest of time, but this is sort of how implementation was going in the different schools, and you'll see the control schools did do a little bit of RJ, if not all, if not all of it.

School 7, in particular, implemented a lot.

So I'm going to focus on three system-change components that we learned about during our interviews for our process evaluation.

There's four system-change components, if you're familiar with the systems level change framework, but I'm going to just talk about three of them, decision-making and power, resources, beliefs and ideologies, for the purposes of this presentation.

You can look out for our publications hopefully coming towards the end of this year.

So decision-making and power, the key thing I want to focus on here is that with RJ and really any whole-school program that we evaluate or look at, we can see that principal buy-in is central, but our participants really identified principal resource allocation as a barometer of a principal's support and whether principals were putting resources, meaning money, classroom space, time into RJ, really indicated whether or not they supported it, and there is sort of an open question for the field about whether RJ can "actually shift school culture if power generally remains in the hands of one (or a small group) of school administrators within all school contexts?" So that's just something for us all to think about.

The second thing I'll quickly talk about is resources.

So resources are generally thought of as what I said earlier, like, "Okay, we have more staff." So that's a resource, but we also looked at social relationships as a form of currency and saw the ways in which RJ coordinators were really enabling stronger social relationships that then, consequentially, also allowed other school resources, such as information, to become more accessible.

And so RJ coordinators are not just in themselves an additional staff person who is providing an additional resource to the school.

They're also creating other things that are also resources.

One of those things is an RJ training infrastructure that helped ensure that many, many, many more school staff were trained in restorative justice.

Schools often rely on existing school employees to implement RJ.

So someone is dividing their time between teaching and trying to do RJ, and when they have an RJ coordinator, that burden can be relieved a little bit while also ensuring that all teachers and school staff get trained eventually for the whole-school shift.

And I'm going to...Next I'm going to go into beliefs which I think are particularly important.

I'm not well...

like, not trained on during most RJ trainings, but also really in any whole-school approach should be something of consideration.

So we learned a lot about existing beliefs that schools, staff and teachers had around discipline, behavioral standards, their professional identity and authority that influenced their buy-in and adoption in RJ, and these beliefs were actually similar across both the treatment and control schools, but they were more likely to have been changed in the treatment schools after their participation in RJ circles or trainings.

And so it's really our responsibility, at this point, from our perspective of trainings and professional development, to move beyond just RJ skills of sort of, "How do you do circle, and what does it mean to do circle?" to tackle these underlying beliefs and ideologies that exist within education and really begin during education training early on.

I only have one slide on the outcome evaluation, and hopefully other researchers today will present more on their outcome results.

We have data from the NYC DOE on all of our schools and students.

We literally got it in December, late December, and we're still...

They still have to send us a few more things.

But we're going to use multi-level modeling to examine the impact of restorative justice on student attendance, grade advancement or graduation, and student incidents and suspensions.

We might also look at the school-level climate measures that NY, that New York collects, but that's not going to be a center of our analysis.

And so we're currently in data analysis.

So I won't say anything more about that, but we have our NIJ report coming out later this year and an outcome, like a journal article on the outcome study also in preparation.

Thank you.

That's all I have for you today.

I have a few QR codes here if you're interested in learning more about this specific program.

They were covered by PBS NewsHour, and so you could click on that.

My e-mail is also on this page.

The RJ coordinators help start an award-winning podcast for the students in one of the schools.

So feel free to go to these websites and learn more about this work.

Thanks again to NIJ for supporting it, and I have a huge...

give a huge thanks to my team, Andrew Martinez, Elise Jensen, Lina Villegas, who are...

were instrumental in getting this all done.

And on to John.

-Hi, this is John.

Let me figure out where my presentation is.

There it is.

And put it into mode.

There we go.

Thank you very much, Lama, and thank you for giving a good introduction to restorative practices and restorative justice.

I'm also presenting on restorative practices as it's been implemented in Pittsburgh public schools, and we are somewhat farther along in our cycle than Lama is.

We were in the field with a 2-year program for the '15-'16 and '16-'17 school year, and in 2018 we published a big report and a much more easily digestible research brief on our experiences.

So you can find those by Googling the RAND Corporation.

Catherine Augustine is my colleague who was the principal investigator of this project and is the lead author on the report.

So as I said, we worked with Pittsburgh Public Schools when the call for proposals came out from NIJ for Safe Schools, and we worked with them to figure out what type of initiative they wanted to undertake to improve safety in their schools.

They felt that they would be best served by an initiative that would decrease suspension rates because they were feeling that the fact that kids were missing so much school and also having troubles reintegrating into school after being suspended was causing a lot of problems, a lot of unsafe situations, and so they chose the International Institute of Restorative Practices to implement a model for restorative practices, a whole-school model, and they worked with us on the evaluation.

We set up a randomized control trial where we took the 44 schools that they made eligible for the study, and we broke them into matched pairs and randomized one school in each pair into a treatment school and the other into a business-as-usual school.

The evaluation I'm going to be...

First, I'm going to say a little bit about the restorative practices program so you get some of the details of what it is in the Pittsburgh context.

Then I'll be talking some about the implementation evaluation that Catherine had the responsibility for.

It's based on a survey that she conducted of all the staff in restorative practices schools or case study schools which they did extensive visits with.

And then I'll finish up with an impact evaluation which was my primary area of the study, and that's based on comparison of the treatment versus randomized control schools, using data from the district on suspensions, climates, test scores, a number of other things.

So the restorative practices program that was implemented is called the Safer-Saner Whole School Change Model.

That's developed by the International Institute of Restorative Practices.

Some of the support that these schools had were 4 days of PD at the beginning of the year, 2 in the first year, 2 in the second year.

A restorative leadership team was made up of school leadership and some heavily committed school staff, and they had a teleconference with IIRP, and then we also had seven coaches assigned to the 22 restorative practices schools, and they made two visits to each school per year.

And this, as Lama mentioned, this is based on the notion that you really have to change your whole mindset about relationships within the school.

The idea is to improve relationships among the staff and the students, and the IIRP organization focuses on the word "restorative practices" to emphasize that it's not just a reactive effort to figure out what to do when a kid makes a mistake, but it's also a proactive method to make sure relationships are established up front.

Another bit that the schools engaged in were Professional Learning Groups.

Every staff member was supposed to be in a Professional Learning Group, and they were supposed to meet twice a month.

Here are the essential elements that are defined by International Institute of Restorative Practices that will also give you a sense of some of the activities that the schools intended to engage in.

Now I'm going to turn to some of the findings from Catherine's implementation evaluation.

She found that the professional development before the school year started were very much appreciated by the school staff, and they were well-attended, and the coaches were also a very valuable asset to the school staff, both for mentoring and for feedback.

The leadership teams that were supposed to be on a teleconference twice a month with the coaches, those were not as well received by the school staff, and the Professional Learning Groups, the groupings within the school of all staff, it turns out that twice a month was more of a burden than almost anybody was able to meet.

In spite of the...

some of the limitations of the implementation, we found that the buy-in was quite high, and unlike a lot of other programs that we've evaluated in our time, it didn't tail off as the program went on in time.

We found that it continued into the second year.

In fact, it became stronger, in many cases, into the second year.

We found that based on the survey, restorative practices staff reported that they used the components fairly often.

We asked them if they knew what the essential elements were, and many reported that they did, and we found that in middle and high schools, we actually had an increasing amount of utilization between Year 1 and Year 2.

And we also found that those that had attended the training at the beginning of each year...

Although it was mandatory, not everybody did attend...

and self-reported knowledge of the elements and frequent attendance of the Professional Learning Groups and interactions with the coaches, these were all positively correlated with reporting high use of the components.

One of the things that we...

surprised us, though, is that even though the staff said that their relationships did improve over time, both with other staff and with the students, the staff did not have the impression that the students' behavior changed over time.

We asked whether they thought the disciplinary sanctions were changing because of restorative practices, and the staff did report that students were less likely to be suspended for behaviors that they previously would have been suspended for.

So this suggests that, at least from the staff's perception, behavior is staying about the same, but suspensions are going down.

Now I'm going to turn to the impact evaluation that we did based on data from both treatment and control schools, data provided by the school district.

We did, in fact, find that there was a significant reduction in suspensions, pretty much regardless of how you measured suspensions.

Our primary measure was number of days suspended to account for the fact students get suspended for just a day or 2 for minor infractions and for longer amount of time for more serious infractions.

If you average over all the students, those that never got suspended and those that did get suspended, we found that on average, students lost about two-thirds of a day of school out of 180 days in the baseline, the year before restorative practices was implemented, but this went down 16 percent after restorative practices was in place.

We also found a significant reduction in the number of suspensions.

We found a significant reduction in the percent of students who were suspended, though this was only significant at the 0.1 level.

Interestingly, we didn't find a significant change in the percent of students that were suspended for serious offenses: violence or weapons.

Most of the action seems to be coming from reductions in suspensions for other types of infractions: insubordination, other types of more minor misbehavior.

We did find that the reductions in suspensions were much greater for African American students than for white students.

You'll see in the second set of bars here that African American students are suspended much more frequently than white students, but restorative practice has seemed to reduce that significantly for African American students but not for white students.

I'll also point out that it's...

that economically disadvantaged students, there was a great reduction of their suspensions as well, reducing the disparity, suspension disparity, by poverty, and we also found that it was the lower grade levels where the drop in suspension was the highest, students in elementary grade rather than middle grade...

middle school.

Interestingly, and of course this got some headlines, we also found that there seemed to be some reductions in achievement in the restorative practice of schools relative to the control schools.

We found this to be the case more in math than in ELA.

That's not on the chart here, and we found it to be more the case for African American students than for white students.

We...This concerned us a fair bit, as you might imagine, and we spent some time after our report was released trying to really dig into this, and I'll talk about that in a second.

The impact on climate was generally positive, as measured by a Teaching and Learning Conditions survey that the district gives to all teachers every year.

We found that reduction in both an overall composite of school climate was significant, and we also found the student conduct composite, and a number of the elements listed below of the student conduct composite were...

had a significant improvement in the restorative practices schools.

Why did restorative practices lead to reduced test scores? Well, we scratched out head and built this little model that said, well, maybe it's because if you're giving a lot of training on restorative practices, that's less time to give professional development on other topics.

It also could be that there's more class time spent on restorative practices, and it leads to better relationships perhaps but could also reduce test scores if it's less time you're spending on fractions or other academic topics.

It's also possible that if you're suspending fewer students, that could lead to more disruptions to class and could lead to lower test scores for everybody.

So we looked at school-to-school variation to see if among the restorative practices schools, the schools that...

where more of the teachers went to the training, were those the schools that had lower test scores.

More of the schools that reported relationship improvements, were those the ones that had lower test scores? More of the schools that suspended fewer students, were those the ones that had lower test scores? And basically we found none of this seemed to explain the lower test scores.

It seems to be that something else is going on here that led to test scores in the restorative practices schools going down relative to the controlled schools.

So we're not sure what to recommend based on this.

It could be that one thing that needs to be done is restorative practices needs to be integrated better with instructional practices, or it could be that it just is going to take some time for restorative practices to become second nature to the teachers and not get in the way of their instructional motivation.

We also asked a little bit about whether this program was sustainable over time and could be brought to school and the district, and there seemed to be a variety of opinions in the district about the extent to which it competed or complemented other efforts within the district, and many folks thought that it complemented things like PBIS and other efforts, behavioral efforts within the district.

So the hope is that it's not going to be difficult to scale and sustain.

So thank you very much, and I will turn it over to the next speaker.

-Thank you very much.

I hope you can hear me.

Good afternoon.

My name is Kimberly Kendziora from the American Institutes for Research.

I'll be presenting along with my colleague Juliette Berg.

We worked on this study of the impact of Safe Public Spaces in schools program, and we want to acknowledge some of our colleagues whose work is presented by us today: Larry Dieringer, who's the executive director of Engaging Schools.

Engaging Schools is the organization that developed and supports this intervention and Sarah Klevan and Jim Kemple at the Research Alliance for New York City Schools at NYU who led the implementation evaluation for this.

We also want...

You'll be hearing from Juliette a little bit later.

We want to acknowledge our principal investigators on this work, David Osher, our colleague from the American Institutes for Research who I think is in the audience today perhaps and Nancy Guerra who was our lead PI from UC Irvine, and again, just as with everybody presenting today, we owe a great debt of thanks to the National Institute of Justice which funded this work.

This grant was part of the 2015 CSSI cohort.

So what I'd like to do is I will tell you a little bit about the Safe Public Spaces in school program, and then I will talk a little bit about the implementation evaluation that the Research Alliance led, and then I will turn it over to Juliette to talk about the impact evaluation.

So just to start, by Safe Public Spaces in schools, we really are talking about hallways, cafeterias, stairwells, entry and exit areas, those kinds of spaces within the school building that seem to differentially contribute to unsafe environments in schools, and this intervention really focuses on those areas and provides a lot of training and support to schools to not only improve safety in those areas but to also improve climate and relationships, promote positive school climate, good student character and conduct and reduce use of exclusionary practices.

There are several components to this intervention to help you understand what it is that we're studying here.

The two...If you just look at the second row going across, there are seven components.

The two in white boxes really are the efforts to promote positive behavior.

One involves placing staff at identified hot spots in the schools, which are based on incident mapping of school safety events but deliberately staffing adult supervision for those areas and also classroom meet and greets, which is really posting teachers in the doorways to their classroom, greeting students by name between classes, and I should say these are middle schools in which we're doing this, so there is transition between classes, making personal comments to students, gestures, welcoming them.

The three green boxes in the middle reflect efforts to prevent potential problems in schools such as teaching staff how to deliver effective reminders and directives to students in public spaces, how to conduct effective hall scans during the school day and how to defuse students who are emotionally charged in the context of some potential disciplinary event, de-escalation basically.

Then finally, there are two intervention strategies that were part of the Safe Public Spaces intervention.

One involves how to handle high-impact incidents and the other, how to handle students who are persistently hall walking in schools.

All of this was implemented in schools over a 2-year period.

The implementation support for this involved a lot of review of data with schools, the formation of a School Climate and Discipline Team that really was the crux of this.

There was initial professional development supplemented by ongoing coaching and follow-up training sessions.

It was...This intervention was implemented in phases over time.

Really the School Climate Discipline Team was at the heart of it.

The folks from Engaging Schools had consult...

Each school had a dedicated consultant who visited twice a month when they would do walk throughs and look at data together.

They would do live coaching.

So it was a pretty intensive intervention for these schools.

I'm going to turn to looking at the implementation evaluation for this initiative, and this is really driven by just a few questions.

First, to what extent basically was the intervention delivered with fidelity? Second, what do school leaders perceive to be the successes and challenges associated with implementing Safe Public Spaces, and finally, how do control and treatment at schools differ with respect to the safety approaches and practices that are part of the Safe Public Spaces intervention? We have a little bit of a convoluted story to tell.

I won't go into the great details here.

There are 24 schools with middle grades that were recruited during the summer of 2016.

We included only schools that were not colocated, so Safe Public Spaces would be public spaces that were particular to a single school.

We tried to include schools that had relatively more discipline issues.

Some schools transitioned from a K-8 to K-5 and then weren't eligible anymore.

We had two intervention schools drop out after the first semester, and they were replaced.

At the beginning of the second year, we had two intervention schools and one control school drop out.

The intervention schools agreed to continue participating in the measurement component of all of this.

The control school we used the school record data that we were able to get.

So we had 12 intervention and control schools in year one and 11 in year two, except for we did include the school record data for all schools.

There are two streams of data collection for the implementation evaluation.

One is observations, like observations that were done twice a semester in year one and three times a semester in year two of the study.

The observation protocol did not try to include high-impact incidents.

We're just not going to see that during an observation, but it did include looking at the hot spots, the cafeteria and sort of a walk through the building.

The other stream of data collection were interviews, which we did with school administrators to learn about the things that we couldn't see as well as understand their experience of implementation.

In terms of data analysis, the observations were aggregated into three broad constructs relative to things that were important about the intervention.

One was adult supervision, the extent to which the adult supervision components such as supervision and meet and greets were taking place, visual reminders which is signs indicating hallway flow, rules being posted, those kinds of things and then student behavior.

We also observed for occasions of disruptive behavior events during the time that we were observing.

This is a lot of data which what I'll do is just focus for this slide on the change over time.

What we see are the broad constructs for the observation categories on the far left, the specific in the middle and then sort of details about whether it was sort of a yes, no, did we see it or not or how many staff or how many reminders, those kinds of things.

So the findings for the treatment and control schools are shown in the columns in year one and year two.

If the number went up by 10 percent or more from year one to year two, that's shown in blue.

If the number went down by 10 percent or more from year one to year two, that is shown in sort of the orange or gold.

Also, just so you know, the bolding reflects if there's a difference of 10 percent or more between treatment and control that favors treatment, if there's a difference there.

What you generally see here is that there were a number...

So the adult supervision and visual reminders are all implementation constructs, and these generally went down from year one to year two.

So unlike some of the results that we've heard about for restorative justice that got stronger in year two, these went down a bit both in treatment and in control schools.

The other thing you might notice is that there's pretty high levels of all of these implementation constructs going on in the control schools, which is important to note.

Behavior incidents were generally higher in...

In general, they were higher in control schools.

That's less true in the hall...

in the cafeterias.

So this slide presents sort of differences by condition sort of in summary.

Basically, adult supervision was consistently higher in the treatment schools.

Student behavior was overall higher in control schools, visual reminders higher, and this is across the 2 years.

So what we found from our interviews was that the high level, the intervention pieces of the intervention weren't very different between treatment and control schools at all.

The sort of dealing with high-end behavior was very similar in both places.

What was different were how they handled the sort of promotion of positive behavior and the prevention of behavior, and also the Safe Public Spaces schools reported collecting and using more safety data.

There were some schools, about a third so not a majority but some, reported staff buy-in as a challenge and also found some challenges in finding time to review data.

They did actually institute changes to how they manage their public spaces.

They used a lot more meet and greets, which makes things feel friendlier, and there was report of improved rapport between students and teachers.

So just to summarize, the intervention was implemented with fidelity, was implemented more strongly in year one than year two.

The implementation was better, stronger around promotion and prevention.

Again, control schools were doing a lot of this, especially for high-end stuff, and there was some challenges around implementation, of course.

So let me invite my colleague Juliette to turn her camera and on, and we will...

And she will walk us through the impact findings.

-Thanks, Kim.

So I'm going to focus on the impact evaluation we did.

So our question here was about, you know...

It was looking at whether we saw an impact in initiative schools on suspensions and incidents, and we looked at...

So we were relying on administrative data from the district.

We looked at both proportion of incidents overall and specifically in public spaces as well as proportion of high level so the most severe incidents, four and five, on a scale of one to five.

New York City rates the level of incidents.

So we looked at those overall and in public spaces, and then we also looked at suspension rates overall and in public spaces and suspension rates that resulted from four or five incidents overall in public spaces, and just tog et more specific about the...

what we characterize as public spaces.

Oh, sorry.

Yeah, you can...

I missed the one slide about sample.

So Kim talked a little bit about the sample.

We had 24 schools in the impact analysis.

We randomized...

We randomly assigned schools to intervention or control using matched pairs with data that, as I'll talk a little bit about later, a little bit later, was imperfect data and was from 2 years prior to intervention, and I'll show you how that may have affected our randomization, but the sample was a very diverse sample reflective of New York City public schools.

So to get a little bit more specific about the...

how we characterize public spaces so these are the categories that we included in the public spaces.

So when we are talking about incidents or suspensions from public spaces, this is what we're talking about.

So this was the just some basic descriptives about suspension and incidents rates for each of our different types of incidents and suspensions in the pre-implementation year, and I just want to point out a couple things.

So we...You know, originally this was assigned as a RCT.

We were going to do a simple impact analysis, but we did find some differences, some baseline differences in our outcomes between treatment and control schools.

So, Kim, if you want to just click a couple times.

You can see a couple of examples of that.

So we do see higher incidents and suspensions in treatment schools compared to control schools at baseline, and so, you know, basically we thought that just doing a simple impact analysis was problematic because of this.

I'll also show you in a little bit that there was kind of a different trend over time between the two main control schools.

So the randomization for whatever reason was not entirely successful, and so instead of just doing a straight impact analysis, we decided to explore using a comparative interrupted time series design with our treatment and control schools, and that's what I'm going to present on today.

So a comparative interrupted time series looks at the trend in outcome during the pre-intervention period for treatment and control schools, so that's the baseline trend, and then it looks at the extent to which the outcome deviates from the baseline trend post-intervention, and that's the deviation from baseline trend.

So the effect that we're interested in looking at here, so the effect of the initiative, is a difference between the average deviation from trend for the SPS schools and the average deviation from trend for control schools, and I'll show you a visual of this in a minute.

So this is basically what we found.

So we found that we had significant effects for suspensions across the board of the different suspension categories that we looked at that were pretty consistent so...

which is what this table is showing whereas for incidents, we didn't see a consistent trend in the significant effects so suggesting that, you know, we really don't see an effect on incidents whereas for suspensions, there's a suggestion that there was an effect on suspensions in the direction that this is favoring the SPS school.

So the negative coefficient indicates that the average deviation from trend for the SPS schools are larger for control schools meaning a trend of more suspensions than projected, and I'll show you what this looks like.

So these are the coefficients from the models, and then here's just a visual of that.

So this is using just basic averages.

So this is not model bases.

It's just basic averages, and it's really just to illustrate what's kind of going on here, and just as kind of a caveat, these are, you know, small, really small differences, but what you see here is that...

So in the pre-implementation years so the negative three, negative two, negative one, so you can see a trend where treatment school suspensions are going up.

Control school suspensions are going down for whatever reason, and you see that kind of small difference in the pre-intervention year.

Now, what you're seeing in the dotted lines, to the right of the kind of vertical dotted line, is the projected trends based on the slope from before so from the negative three, negative two.

So for example, for treatment, you're seeing, like, there was a slope going in the upper direction, so suspensions were going up, and for control schools, suspensions were going down.

So that's what you would have projected just based on normal things happening without any kind of intervention.

Now, what you're seeing in the dotted lines...

So the yellow dotted line is what we actually...

what actually happened.

So you can see in the treatment school, so those yellow dotted lines are treatment, and what actually happened is that suspensions were lower than the projected line, and then in control schools, you can see those blue dotted lines.

So in zero, they're one on top of each other.

They're in the same place, but the blue dotted line is indicating that suspensions were actually a little bit higher than projected in control schools.

So the result is that you get this negative coefficient, and you get the difference between actual and projected in control schools is three percentage points higher for control schools and two percentage points lower for treatment schools.

You can see that in the actual, like...

So the suspensions are still higher in the control schools at the end of the second year, and so if you look at just, like, a straight impact, you see that there's actually higher suspensions in the control schools, but you're seeing, like, not...

They're almost the same in this particular example.

In another example...So sorry.

The previous one was suspensions for level four and five incidents in public spaces.

Another example here is just suspensions overall.

If you look at the time one dots there, you see that, you know, suspensions...

All the way to the right, suspensions are actually higher in treatment schools than control schools.

So you look at that and say, "Oh, well, you know, the intervention didn't work," but actually, what you're seeing is that in control schools, the actual suspensions were higher than projected, and in treatment schools, the suspensions were about the same as what was projected.

So that's why you get the negative coefficient.

So it seems to suggest that engaging schools did potentially, you know, change...

alter the course of suspensions, but these are small differences, and it's a little bit of a weird trend because it's not, like, a straight, like, you know, suspensions go down in treatment schools and stay the same or grow in control schools without any absence of the intervention.

So just to conclude, there were small effects in suspensions, but it's not a clear story.

It's a little bit confusing, and in each of the types of suspensions, the story is a little bit...

The trends are a little bit different, so it's a little bit hard to interpret.

There was a lot of fluctuation in incidents and suspensions across the school years, which may be why we had kind of this failed experiment and with a lot of imperfect data, and so that makes it hard to kind of really project, like, what would have happened in the absence of the intervention, and then, you know, for whatever reason, like, there's a lot of policy changes happening in New York at that time.

There's a lot of things happening.

This is just one program among many other things happening, and so a lot of that fluctuation may have to do with other things that schools were doing, and the findings are sensitive to model specifications, which kind of makes us a little bit worried that, like, you know, it's just not, like, a very clear consistent story.

So overall, you know, we do find that the SPS intervention was well implemented in study schools but only achieved modest and kind of unclear effects for the student outcomes that we were interested in, and, you know, one reason is that, you know, like Kim mentioned, the metrics from the implementation study were pretty high in the control schools, so there was a lot of stuff happening in the control schools as well.

Like I said, New York, you know, was working on suspension rates at the time, still is, and trying to get those down, and so schools may have been doing a lot of other things, and we just don't know everything that they were doing.

The discipline data is really noisy, and, you know, there's just a lot of things happening.

If you look at the discipline data across schools within treatment group and control group, there's so much fluctuation and so many differences just across here and between schools.

It's just really noisy data, and when you're looking at this kind of average, you know, these average suspensions in the school, there's just so much noise there that it's really hard to kind of, like, understand what's going on.

And then as Kim mentioned, there was a slight drop-off in implementation quality in the SPS schools from year one to year two, so, you know, it might...

This might reflect a lack of commitment from the school staff, or maybe this is...

Maybe they saw that something good was happening, and so they kind of, like, lightened the touch, or maybe this was hard to implement.

We don't really know exactly why, but that could explain kind of, you know, a lack of, like, a strong impact.

So that's it.

Thanks, everyone, and I will pass it onto the next presenter, which I believe is Naomi.

-Great.

Thanks so much.

I will share my screen.

There we go.

So I am thrilled to be able to talk about the Philadelphia police school diversion program and results from our ongoing evaluation that's been supported by a series of grants from NIJ and OJJDP.

A lot of you heard about this and got a little bit of a teaser for this presentation today from Kevin Bethel and Abby Gray during the opening plenary, and so I'll try to give you this in a bit more detail since I've got some extra dedicated time to this as opposed to the two of them who had to fit in all of the amazing efforts going on across the School District of Philadelphia over the past 5 years.

So, okay.

So just to give a little bit of a preview about how this program came about, a big of part of this came about from Kevin Bethel who, back when he was Deputy Police Commissioner, really looked at the schools and looked at data and said, "What are we doing?" Right? We started with, in 1994, the Safe Schools Act followed by zero-tolerance state policies, and zero-tolerance policies in schools led to...Sorry.

I'm having trouble with this thing.

...led to the introduction of metal detectors in schools which meant they had to be staffed, so school officers were put into schools, and as we...

I assume most people on this call are aware, and many work on, once you've got officers in schools, you tend to see real increases in the rates of school-based arrests and suspensions, and so as Kevin Bethel and his team at the police department were looking at 10 years of data, what they found was across 10 years, we're seeing about 1,600 arrests in Philadelphia schools annually across the city, and accompanying that, you also saw about 32,000 suspensions across the district each year.

So with that framework, I'm going to try to...

I think Kevin is on this call, but he's...

I don't think has the ability to speak, so I'll try to channel him and his statement that I've always heard him say which is, he and most police officers did not go into policing to arrest kids, especially for things that often represented sort of normal adolescent misbehavior, and so his question became, how did they address that? And so what Kevin Bethel did when he was at the police department was created the Philadelphia Police School Diversion Program which he did in collaboration with the School District of Philadelphia and the Philadelphia Department of Human Services, and just to give you a little bit of a sense of what this program looks like is if there an incident that occurs in school that meets the threshold for calling the police, the police get called.

If at that point the incident is a qualifying offense, so something that falls into a long list of offenses but, you know, certainly does not include things like gun possession, does not include serious harm to a victim but does include lots of other types of offenses like carrying a knife into school that is not used, that carrying in small quantities of substances into school, so a whole bunch of things, things, though, that you see kids getting in trouble for all the time like bringing scissors or mace into school, so all of these things would qualify.

At that point, the school and the police check with an intake center about whether...

by phone because again, the whole premise of this program is, the goal is not to remove kids from school via arrest, to not disrupt their lives, not disrupt their schooling, not get them into the justice system if at all possible, and so checking to see if they have prior offenses.

If they don't have a history of delinquency findings, if they don't have an open court case at this moment, and their offense qualifies, then they're automatically diverted.

This is not a program that has discretion.

When Kevin Bethel designed this program, it was designed specifically to make sure that you weren't increasing bias that we know often occurs with diversion programs, so one of the things we know nationally is that when diversion programs and alternatives to arrest or alternatives to processing are implemented, they often disproportionately benefit white youth, so how do you do this without creating or enhancing disparities and ideally decreasing disparities? So trying to remove any subjective opportunity to figure out who should be diverted, so these kids are automatically diverted, and they can be referred for services in lieu of an arrest.

I want to be really clear: This is a program that automatically diverts youth, but the opportunity for services is voluntary.

If they choose to take services, Department of Human Services funded services to provide academic supports, mental health supports, social service supports.

That is something that is open to them free of charge, but if they choose not to do it, they are not then arrested.

There are no contingencies.

There is no turning back the clock that once a youth is diverted, they are diverted from arrest, and they have the opportunity to remain in school.

Also relevant is the school retains the decision-making authority on how to respond with whether they should be responding with exclusionary discipline or responding in some other way, so should there be a suspension, expulsion, disciplinary transfer? So these are two separate mechanisms that happen in parallel.

One is the arrest decision and referral for services, and the other is the school discipline response, and those, the fact that those are two simultaneous but not linked distinctions is going to be incredibly important as I talk about these results.

So I want to give you a sense of the goals of this program, so...

and where this came from, so just to give you a sense of schools, about 134,000 students with 1,600 arrests annually.

A disproportionate number of those arrests were happening to particularly black youth, and that the goals of the program, there were really four of them.

The two primary ones were to safely reduce arrests by 50 percent.

The goal was originally within 3 years.

To improve school outcomes, so keeping, again, this goes back to what Kevin Bethel was talking about earlier with keep kids in school and out of court, so both to improve their outcomes in terms of staying in school and hopefully graduating long-term, reducing racial and ethnic disparities in these arrests and school discipline rates and to provide kids access to services.

Again, this idea was that so much of the adolescent misbehavior that they were seeing very well may have been a reflection of youths' needs, right? The kid who comes to school with mace is typically there because they feel unsafe, not because they want to harm somebody, so trying to address mental health and social service needs.

So what I'm going to talk about today, though, for the sake of time is these first two primary outcomes.

So to give you a sense of how this study worked, we compared all youth who were diverted between 2014 and 2017, so those are going to be referred to as our diverted youth or our quasi-experimental group.

We actually have data through 2020, but because we want to...

One of the things I'm going to be talking about are 2-year outcomes, we want to...

and want to sort of get rid of COVID times so that we don't...

our data is not so messy.

And my apologies.

What is a presentation without a dog barking in the background? But...So comparing diverted youth to arrested youth, youth arrested in schools in the year before the program started.

Kevin joked earlier during the plenary that, what's a control group when you can...

can't have a control group when you're trying to do it across the entire school district at the same time in one day as he said.

So instead what we did was, we created a quasi-control group for this, so it was all youth arrested in schools in the year prior to the start of the diversion program who would have qualified if the program had existed at the time, so that was our group.

We also did some propensities for matching for those of you who are interested in sort of how we narrowed down further and created, made sure we had fairly equal groups.

And so in terms of our data, again, this was a collaboration between the police department, the school district and the Philadelphia Department of Human Services, and so they all provide us with de-identified data, so the evaluation team at Drexel, they provided us with de-identified data that could still be matched and merged across systems, so we ended up with a very robust cross-systems database.

In terms of our outcomes, I'm going to first...

One of the main outcomes we really looked at, we looked at arrests in the short-term.

We looked at recidivism over 2 years.

We looked at suspension outcomes, and that's what I'm going to talk about here.

So Kevin gave you guys a heads-up and a little bit of a teaser earlier about the really successful reduction in school-based arrest rates across the city, so again, remember, the first goal was to reduce rates of arrest by 50 percent within 3 years, and we saw that in the first year of the program.

Those numbers kept going down over 5 years to an 84 percent reduction in our last full school year, the 2018-2019 school year, so again, from almost 1,600 arrests annually down to 251, and just to give you a sense, some of that reduction was through the diversion of youth, so this lighter blue is the number of youth diverted, so the question became, "Well, what are all of these other differences? Where did this reduction also come from?" And one of the places it came from was also what we saw as a reduction in the serious behavioral incidents reported in schools, and this was really important.

There were about over 1,000 fewer serious behavioral incidents reported in the first 4 years with over 2,000, a 34 percent reduction, in year five, and this is important for two reasons.

One, it doesn't do any good if you're reducing arrests, but there's still chaos at the schools, or you can imagine if the rates of serious behavioral incidents go up, it means that arrests were really reduced at the expense of public safety or at the expense of school safety, and so one of the things that we see here is that it seems like that reduction in arrest rates did not compromise school safety in terms of serious behavioral incidents, and this is also really important because this was initially principals' primary concerns in the implementation of this program.

Okay, so those were really descriptive findings.

Thinking about the 2-year recidivism rates, we can compare diverted youth to the quasi-control group of youth who would have qualified for the diversion program had it been in place at the time, and what we found was over 2 years, no matter when we looked at this, a significant reduction in recidivism rates, and this is recidivism in school or in the community.

So at 2 years, you'll see a reduction there from about 32 percent down to about 27 percent.

Importantly, though, what I want to point out is that that is a significant difference, but perhaps just as importantly if not more importantly is to keep in mind that 100 percent of those kids in the quasi-control group now have an arrest history, right? Every kid who was arrested in school has a history of arrest that can present long-term collateral consequences.

In contrast, the diverted youth were spared that history of arrest and all of the negative long-term consequences that go with it, and so if we look at recidivism 2 years out, at 2 years out, 73 percent of diverted youth remain arrest-free.

They have no history of arrest.

So in terms of sort of grand impact on these kids' lives, it's potentially really tremendous when you think about the effects of arrest on access to higher education, public housing, opportunities to enter the military.

There are really a broad set of benefits to these kids.

Okay, so now shifting gears and thinking about outcomes in terms of exclusionary discipline, what do we see happening in schools? Again, one of the goals of the program was not just to prevent the arrested kids but also hoping that kids would stay in school long-term, and we're still looking at that in terms of progress towards graduation and graduation rates, but in the...sort of over the course of 1 year, what are we seeing in terms of suspensions and expulsions? And what we see is initially, you'll see this first set of bars.

Initially, as we think about, as we asked about suspensions related to that initial incident that referred youth to the police, we don't see a significant difference, but what we do see is a substantial and significant decrease in suspensions within 1 year following the incident for other incidents that might occur, and we also see a significant decrease in suspension and disciplinary transfers across the year, across 1 year from the date of the incident, so again, it's not necessarily impacting the immediate outcomes in terms of school discipline, but it is having significant differences over time at least for that first year.

One other thing that I want to point out is something that Abby Gray raised during the plenary when she was talking about how suspensions across the School District of Philadelphia have gone down to such a small number that we can barely look at them as outcomes anymore, and while that's true for the district as a whole, that's obviously not true for kids that have had contact with police, that this is not a rare incident.

In fact, it's much, much more common for these kids to have suspensions that not to, so just want to sort of highlight that this is still a group that's really running into both, that's really running into school challenges even when they're being diverted.

Just want to give people also a little bit of a flavor of the needs that these kids are coming in with and what kinds of services that they're being linked to through these voluntary community-based services, and this doesn't, I'm sure, come as a big surprise to most people on here, but you've got certainly, you know, basic access to things like food, clothing, school uniforms which is one of the things that we hear from kids about the reasons they may skip school and be truant, stable housing, transportation, but also things like utilities in their homes.

I mean, the number of kids who don't have access to electricity in their houses, and so the program has helped to get them access, families access to that, but also mental health needs, certainly language and cultural barriers both among kids and families and insurance, so really a wide range of challenges that I think are not uncommon among kids in urban schools in general but are particularly pronounced among this group.

So in sum, I just want to note that what these results reveal to date are certainly successful progress towards the primary goals of keeping kids in school and out of court and preventing long-term recidivism.

We've seen that this is being accomplished without compromising safety, so meaning the decrease in serious behavioral incidents, and again, decrease in recidivism in school and the community among diverted youth, and lastly, what I haven't talked about today but is a big part that I think we need to discuss in the future is how police, school district and Department of Human Services, how the data interacts to really shed light on integration, the sense that these kids, they don't exist in siloed systems, that they interact with families.

They interact with schools.

They interact with police and communities.

They interact with public service providers, social workers.

Many of them interact.

Also, one of the things we're looking at in our NIJ data is involvement in the child welfare and foster-care systems, and so how does all...

What's the interplay between this in terms of both creating heightened risk for these youth but also the ability to intervene and help them where needed? And I want to pull one quote to end on from a youth.

This was somebody who was in the program very early on, and this quote was in a newspaper that he gave, so there's not sharing any personal information, but just the idea that this program was really created to give a second chance, the idea that kids mess up, and this program is there to give a second chance, to give them help that they need and to prevent them from getting locked and trapped in a system that we know creates negative outcomes for them.

So on that note, I'm going to wrap up as I know we are moving onto some questions, so thank you all so much, and if anybody has any questions, here is my e-mail address.

You can feel free to reach out to me directly for more information.

Basia, did you want to jump in with questions? -Yes.

Hello, everyone, again.

First of all, I want to thank our panelists for sharing this important information with us.

It is evident that these projects are imperative to producing scientific evidence about alternatives to traditional school discipline.

I'm looking at our Q and A, so we do not see any questions, so I want to remind our audience that you still have a time to enter some questions for our panelists.

Please feel free to do so right now.

Also, if you think of a question afterwards, please post it on our discussion board that is available on our conference site.

So I'm going to give us a few moments to let the audience post some questions on our Q and A.

In the meantime while we are waiting, how about if some of you might want to provide us with some conclusive thoughts on the alternatives to school safety in no particular order? Just if you would like to share some conclusive thoughts for the audience, you're more than welcome to do it.

-Sure.

I'm happy to jump in.

So I think just hearing across all of these, all four of these talks and projects, I think I had a few takeaways.

One was, I think everybody's work is based in the premise and existing research showing that traditional approaches to dealing with student misbehavior like arrest, like exclusionary discipline, are really harmful not just to youth but also to school climate and school communities.

We know that when kids are arrested in school, we know when there are suspensions and expulsions, kids actually feel less safe and less secure in school, and that the perceptions of schools as being sort of dangerous and nonprotective places is...

really occurs, and so I think all of this work is embedded in the premise of, how do we find more prosocial, positive, supportive ways to help kids knowing that that's not only going to have...

that that's going to have positive impacts both on kids' long-term, in terms of completing school, in terms of their long-term well-being, but also in terms of its impact on the broader school community.

And so I was really inspired by all the other research that was presented during this sessions around the efforts to do that through restorative justice, to do that in public spaces especially with noninstructional staff who often aren't reached through sort of traditional intervention mechanisms that really target teachers and administrators.

And so those are my takeaway thoughts.

-Thank you very much, Naomi.

So there are some questions that are coming in right now, so I will still give an opportunity to others to provide some conclusive thoughts if you wish so, but let me go with this question here, and this one is actually directly to Naomi again.

So, "How are you dealing with privacy issues in working with data across systems? Is there concern that people can be identified even if their names have been removed from records?" -Great question, so this is obviously one of the biggest challenges, and I'm sure everybody else on the call has dealt with this as well in terms of dealing with individual-level student data or individual-level youth data from police, and so that's one of the things we've been really taking into account.

We make sure that not only are leaving out names, but they're leaving out identifying information, so even though age is a critical factor in some of this research, we get month and year of birth, but we make sure we don't get date of birth.

We get zip code, but we don't get addresses, so there's lots of things that we're doing to try to remove, to deal with protecting that, and again, all the information we get is identified by a number that's across systems.

The agencies that worked with each other have an MOU that they can share information, so they're able to coordinate back and forth with each other to make sure that they're inserting the same ID numbers, and if they're able to identify the youth, that's fine, but they never get...

They never see the original individual-level data for each other other than the names.

So so far, we have really not had any issues or concerns with that, and certainly when we present it elsewhere, we're presenting information at the aggregate level.

-Very well.

Thank you very much.

There are some...

There a couple, three more questions, so let me go with this one here.

This is no...

So any one of you can answer it, so please feel free to jump in.

"A common criticism of disciplinary alternatives like restorative justice is that they are race-neutral or colorblind and fail to address underlying racial bias.

Do any of you have suggestions for how to address underlying racial bias within the alternative framework you assessed?" -I can try to jump in and answer this, although it's a very difficult question, obviously.

So speaking for the specific restorative justice program that I was evaluating for the project in District 18 in Brooklyn, so racial equity was a primary concern in the implementation of the program, and I think in a few weeks, the program staff are actually going to put out an RJ implementation guide that you'll...

hopefully I can share that centers that, and I think that it was to be an intentional aspect of the restorative justice program or else it ends being a sort of second...

It sort of gets pushed to the side.

Dr. Kathy Evans, who I quoted at the beginning of my presentation, says that RJ has three central priorities.

One is relationship-building.

The second is repairing harm, and the third is creating more equitable environments, and so if those three things are not centered in restorative-justice approaches, it becomes more challenging to address the racial inequities that we're seeing in the traditional approaches, and I'll just say at the end is, I think that the implementation of the RJ program that we evaluated was pretty unique in that, for example, the schools were predominantly students of color, and the RJ coordinators were also predominantly students of color.

In fact, two of the...sorry, were predominantly people of color, and, in fact, two of the staff members I believe grew up in the same neighborhoods where they became restorative justice coordinators, and so they were able to develop very strong relationships with the students, and they tackled very what would be considered challenging topics in their RJ circles including Black Lives Matter and police brutality and sort of all the things that were at the forefront of the minds of the young people in the schools.

And so I don't know if that really answers the question, but I do think there is a way within restorative justice to be intentional about racial equity, and hopefully that's something that will be shown to be more effective and will be used more within RJ.

Bye.

-Thank you very much.

So I see nodding of the heads here as well in agreement with you, Lama.

Thank you.

It's already 2:16.

We have one more question that I would really very quickly like to touch on.

For those of you who have to disconnect, you can always access the recording of this session to listen to the last answer, so this question asks, "Could you please speak to how school faculty, administrators and communities of color receive messages about policy, philosophical shift from punishment and enforcement?" Any thoughts on this from you? -I can jump in.

-Oh, go ahead.

-Go ahead, Naomi.

-Does somebody else want to go? -Okay, so, you know, I think this happens in a variety of ways, and I think this, in many ways, can be very unique to cultures of different school districts, police departments, agencies, but I think when I've seen it done effectively across partner agencies, it's often been done using multiple mechanisms so often done through large group training, sometimes often through repeated messaging through professional development, through e-mails, which always are of sort of limited scope.

But I also think a really important piece that often gets sort of understated is the value of changing policy, that if you're really looking to have an impact, and I think, Lama, you mentioned it in terms of the racial equity piece, but just leaving things up to individuals to make a difference and to do it differently sometimes...

often isn't enough, that often there needs to be a real specific policy change with really clear mechanisms and criteria for how to shift behavior.

One of the things that we know is that even when people's beliefs and attitudes change, their behaviors often don't, and so trying to make sure that you can bridge that gap between training and practice, and often, that can happen through lots of different mechanisms from hands-on ongoing coaching to ongoing repeated training to reporting of information that happens to real clear policy change that structures decisions differently, and so the places that I've seen sort of get word out and actually change what people do try to do it from multiple perspectives.

-Terrific.

Thank you so much.

I will have to start wrapping it up.

Sorry we couldn't answer all the questions.

If we did not answer your question, you can post it on the discussion board on our conference website, and our panelists will address it when they access it at some other time.

Again, thank you very much for investing your time with us today.

Thank you to all the panelists for these wonderful presentations and sharing all this information with us, and the next session starts at 2:30.

That is a roundtable discussion on impact of COVID-19 on school safety to which you are all invited cordially.

The next breakout sessions will start at 3:30 Eastern Time.

I hope you join.

Have a great rest of the day.

Thank you.

-Thank you.