Advancing Justice Through Science: Tracing Allegheny County’s Journey - 2024 NIJ Research Conference

With that I will pivot.

While I frame this opening plenary I will invite our panelists to come up on stage. We didn't plan to have this research conference in Pittsburgh. Pittsburgh found us, our wonderful contractor who I didn't even think found us. Can we thanks Saxman1 please? And global encore, thank you. And the staff of the Wyndham Hotel. Thank you. Clearly my nerves got the better of me and I skipped by those important thing is, but I didn't forget. Are wonderful conference contractors came to us and said we have a good spot in Pittsburgh. I said, that's amazing because Pittsburgh is home to a great city into Allegheny County. I've known the county well because I've had occasion to do research in partnership with folks in the criminal justice and human services area and I thought, well, this will be our opening plenary because they can tell a story, and it is a story that really embodies all the priorities I went through, the inclusive research, racial equity, getting from evidence action, all of it. I'm so delighted today to welcome our panelists, thank you for being here. I will invite you to introduce yourselves and share a little bit about who you are and how you fit in with the city and County to promote safety and justice through data in research and innovation. We will start with Erin.

So happy to have a conference in Pittsburgh pickup I am the director of our department of human services. You might not think human services should be a strong partner and criminal justice, but these issues are really intertwined. Mental health, substance use, criminal justice involvement. Our role has been prodding information and reform and building the data infrastructure to support its.

I am Sabrina on board. I'm an economist and work in the rants, Pittsburgh office. By the pleasure of working with the county on several research projects. The one project was (Indiscernible) looking at the end act of providing public defenders at bail hearings. The sources of racial disparity at all the key stages of Alleghenies criminal justice system.

Hello, I am not from Pittsburgh, I am from New Zealand. But I've had the pleasure of almost a decade at relationship. A research professor and I led some early work and data-driven decision-making including Allegheny family screening tool used in child welfare and is now being scaled down -- out across many parts of the state in the US and other countries. It's been a pleasure because one of the things I love about Allegheny is not just because it has amazing integrated data but scales out really well.

Good morning I am Janice Dean, the current deputy director of criminal division in Allegheny County. I am (Audio cutting in and out)

I'm a judge and Allegheny County, criminal division but prior I served as an assistant district attorney and had the privilege of becoming director of the Allegheny County Public defender's office and it was in such disarray when I took over because everyone I would meet with tell me there's no way to turn the office around. It was just imperative that I could become involved with Erin Dalton a very intelligent woman and Janice Dean they had the answers and money.

Thank you to all of you. I want to turn it to Aaron, if you can trace the history of the county and all these various efforts to ground us in the story.

Is an easy assignment. I will try and highlight a couple things. And I want to say that I started my career at NIJ, so it was a homecoming from each. And learn what it meant to try to do work in jurisdictions and how hard it was for those jurisdictions to implement reform. Was excited to have the opportunity when it came to Allegheny to come to a place where innovation is happening in partnerships underway and data infrastructure starting to be built. Lucky for me. Three things to highlight starting with -- the jail collaborated was started with relationship for a long time between the human services director and Warden of the Allegheny County Jail and the department. We had second chance act funds and that was important to us to do really strong evidence-based reentry work. There was a large scale evaluation happening. A multi-site, randomized controlled trial. And I figured the timeline would be about five years and I think it was the thing I want to highlight is when work is really important, to make sure the evaluation will support the local needs. We were fortunate enough to have local funding and foundation dollars to invest in an evaluation for us. We participated in the urban Institute conducted both the process evaluation and outcome evaluation that I think is really critical for us putting in place reentry locations. Fidelity to the model and reduction in recidivism.

Another thing to highlight is a place where Allegheny County struggled. Today we have excellent data but when we think about the justice reinvestment work, this was the Bureau of Justice assistance funding and it needed key processes to improve. If we just did a couple things a little different. But we didn't have the data. I'm sure I can convince myself and the feds that we did. But it wasn't making sense and it wasn't there are integrated. But we had to innovate. Back against the wall, how will we do this work and fell back on some work I saw a NIJ. Incredible homicide reviews and jurisdictions throughout the country, death reviews and child welfare. We did pretty elaborate case reviews, criminal course -- court case reviews and going through papers for weeks and months. We set the next set of reforms that helped Allegheny take the next steps of innovation and validated those with the administrative data. But I think it's important to note that you can't always have everything you need and there are still ways to innovate and reform.

Lastly, wanted to highlight, because I think you'll hear more about the risk assessment work and what's been going on in Allegheny, places where government can't always lead. We like to think -- (Audio cutting in and out) we know what the problems are and can analyze and implement but sometimes it's better for outsiders to come in and help push and prod us. University of Pittsburgh Institute of politics led by incredible people, the former Chancellor of the University, Fred Tina, Clinton administration, work beside us for years pushing and prodding and holding us accountable to reform, including spearheading with local foundation funding the evaluation of the racial disparities in the criminal justice system that was mentioned this embodies a lot when he spoke about, Nancy, in your opening comments. Mixed methods with a ton of stakeholder and lived experience involvements, a tricky project as you might imagine. With the two nationally known in their own kind of entities having to work together. The quantitative side was led by ranch and qualitative was led by RTI. They worked massively together which is a credit to them as well. That some of the next work we have to do. Recently MacArthur put out in the safety justice initiative findings that suggest Allegheny County is the only jurisdiction that has both reduce the jail population and violent crime at the same time and we should be proud of that. But those of us that also know that racial disparities in the jail went up during that time. You have to be working on all three of those things at the same time if you want to have results. That's a little bit of the history at a high level.

Thank you for that framing. Aaron, as you know, I dipped in and out of this trajectory of Allegheny County over the years. Serving on the Allegheny County Jail collaborative advisory board. We work together on the local justice reinvestment we wrestled with that data and came up with a creative solution. (Audio cutting in and out) Just bore witness to all the innovations you have done over the years. But there's one I want to hear more about around the bail study. enzyme might invite Samina and Judge Elliott.

Before we got involved with the RCT project the county piloted providing public defenders at bail hearings during business hours. There came the opportunity that they would expand this pic of the idea was some additional defenders overnight, weekends and evening hours. This is an opportunity where Erin Dalton came to us because she saw an opportunity that we could do an RCT study. We were happy to get involved. This required a lot of work on the county's part to make it work. What was happening is they were public defenders being hired and not enough to staff all the shift pick only staffed half. Were able to develop the work schedule with the public defender’s office where a shift where the public defender shows up is similar to where they don't show up she get nice balance control groups. Once we developed it goes out to the public defender’s office to implement and they did such an amazing job and it's not an easy task. You need to get people to stick to that schedule.

What was the outcome?

The study came out about a year and a half ago and this is the only causal evidence we have on the intervention in the past 30 years. We find it has beneficial impacts for the pet -- defendants. Reduces monetary bail or pretrial detention. The best evidence we have is currently about half of counties in the US are not providing a defense representation. To do a rigorous evaluation is helpful for policymakers across the country.

You are the defender at the time and experiencing the innovation for an experimental design. How did that work for you? What worked well, what you a little discomfort? How did you experience it all?

Initially because of the data that we were getting and the information one of the things that went early on was a large number of our clients were being detained in jail because of their inability to post bond. As a result they sat in jail for extensive periods of time. When I began to understand that we had people in jail waiting to plead guilty or resolve their cases to get out of jail the question became why were they ever in jail in the first place. As a result of that we began in our own little silo to try to identify ways to intervene sooner. I got a lot of pushback from stakeholders because they had never been done before. We never represented our clients at the preliminary arrangement where bail was set. (Voice trailing off) One of the most strident opposition I got was from the public defender’s office because we were asking them to do additional work that wasn't part of their responsibility. Change is hard. We ultimately got involved but the question became where will the money come from and where we hire these people? When I spoke with Erin and Janice they came up with the idea of the study I thought it was great. My attitude was we will do the study and if it supports our efforts than the county will finance it going forward. It was a win-win. And we found a putting public defenders at the preliminary arraignments, the initial stage where the bail is being set and by bringing in social workers to work with the attorneys, it resulted in a 26% reduction in jail admissions. We never thought that would be the case but it was impactful right out the gate. One of the most challenging aspects was staffing the shifts and identify (Voice trailing off) there was no way to staff every shift because arraignments are done 24 seven. But with the information we had a targeted time will be new traffic at arraignments was its highest and that became our focus.

You did the experiment and showed it was successful and had a desire to impact. Did the end goal, was that achieved? Did the county fund it?

They did because they took over the responsibility of staff paying for it. Two or three positions.

That's the generating the evidence. I love it. Janice, you've been in so many different roles and I've known you for a long time ago every time I meet you you are wearing a different hat. You had so many different opportunities to see at least types of innovations work or don't.

What is the secret sauce in your perspective?

(Audio cutting in and out) I think a lot of this has to do with trust. There is a lot of things that have happened and of and reference doing case reviews and not was probably 15 years ago. We took three cases and pick different things. A specialty court case, a regular track case and on the side. We would just dig into that case. You have to have a bond of trust. We had a group of 10 of us and we were saying things that weren't super popular. We were talking about offices that made mistakes at that time. I was at pretrial.

Just getting to the point where you can sit down and talk about the stuff honestly and really take a good look at your own functions. We use data a lot for that but there's a lot of stuff that just seems to happen. You get people that are doing one thing we want another thing. Change is really hard. A lot of the data we have when we show people you have that data lines and statistics which ends up being -- especially from the judiciary, sorry, judge. But they sort of think they instinctively know stuff. Will be started really putting the data it really started with the case reviews and it worked out really well for us.

I like that example because it tells us that you don't have to be in a jurisdiction where you have the most sophisticated data. There are ways to dig into problems systematically and empirically. The thing we haven't touched on enough is Allegheny County also is very well known not just nationally but internationally for blending data across different sources. And, Rima, I think you have a lot of experience in that in supporting the county but learning what's happening elsewhere. Can you ship more around the data sharing efforts and what value they supply?

Sure. I got involved with Allegheny almost a decade ago when we developed the Allegheny family screening tool which is to screen children, child abuse hotline calls. Lot of people in the room they not know that one in three American children are investigated for child abuse and maltreatment before age 18. One in three American children are investigated. Are child welfare system is huge and touching lots of children. One into black children are investigated. Allegheny wanted to help solving the problem using the data. What we did was rather than have humans make those decisions we replace it with the data-driven tool which used the full depth of the system.

And what you saw is the amazing integrated data they had was a great starting point but now this tool which is shown to reduce and Allegheny the racial disparities 50% alcohol screening. That was such an amazing effect that it has been adopted in a whole bunch of counties in Colorado and LA with our help. And what we are showing is you don't need the amazing integrated data. You can do this just with child welfare data. Is a valuable lesson. A lot of my work starts and Allegheny because of the amazing integrated data and the support they give to folks like me to roll things out pick up what we are trying to help other counties learn is whatever you have you can usually do a better job than just human intuition.

Hold that thought, we will come back to that. You described a form of a risk assessment tool. And Allegheny County has been a leader in risk assessment around pretrial. But we haven't talked about that. What I'm interested is this tension that exists with risk assessment in this day and age figure risk assessment amount issues, should we detain someone or release them on their own recognizance? Because there was a risk to public safety. Those decisions used to be made by you like you based on their own independent knowledge and there was wide variation in those decisions that were disparate in nature.. We had risk assessment tools that were designed for systematic way of identifying risk factors. So these decisions are more consistent for individuals. That has come under scrutiny for perpetuating or exasperating biases. That brings us back to the issue of racial disparity. But you just described is an example of a risk assessment tool that helps close that gap.. Discuss. Maybe I will turn it back to you, Aaron.

Maybe I will turn into Janice. It's past the microphone time. Just to talk a little bit about the pretrial work and Allegheny and then we can go from there.

Prior to 2007 we had no risk assessment, no data going to the court and I think a couple people have said we arraign a new charges and arrest warrants 24/7, 365. Prior to 2007 the judicial officer that was making that decision had information on the current charge and whatever the police officer told them.. Sometimes there was information the court knew if they were arrested two days ago and they remember them. Back around 2006 we had an evaluation done of our criminal justice -- the criminal court process. When the evaluation came out, one of the top three things they said we needed to fix was pretrial and needed to get some information to those judges. And information that was the same in every single case. We started investigating and we didn't really know anything about pretrial. We didn't know if it was one word or two words or hyphenated. It's not hyphenated, by the way. (Laughter) I did learn that in my 20 years. We start looking at that and get someone to come and they tell us we will need a bunch of data to do this risk assessment. We were late, great, can we get it from the state?We had a state system for common police cases but not for lower court cases. We spent almost a year with thousands and thousands. Prior to COVID we did about 30,000 case filings. We had years and years of file to extract the data and it almost took us a year and a lot of interns and staff sitting and looking through files and Hash tagging stuff on a yellow tablet. That's how we started. We had the first risk assessment in 2006. It was locally validated. I think the journey we had it revalidated again when we were keeping track. At the time when we were going through all of the folders to find this stuff, one of the things that I pushed with our administration was we needed a case management system for pretrial to collect the staff to the next go around.

The infrastructure.

I didn't know what it was called then but said we needed a case management. I didn't want to go through it again. That's been our journey and in our case management system has evolved as we found things that we were not collecting in 2006 that we would like to collect now and 2024 and even in 2020 we revalidated in 2020.

When revalidated be looking at issues of disparities?

In the 2020 revalidation we looked at racial disparities, gender bias, and the reports is on our courts website. We were neutral. We were over inflating gender and females and their risk to either reoffend during pretrial phase or fail to appear. We did account for that.

When we are getting to revalidated again because what we would like to look at, in our 2020 revalidation, couple things we removed that we had in a prior was arrests. We removed all the arrests.

To my point.

Arrest data was predictive but also racially biased.

We had a researcher look at that, the benefit we gained wasn't enough to keep it in so we removed it. We also looked at failure to appear in two different pockets. We were looking at forever and now it's two years or more or less than two years. Its way to different. The next iteration we will look at criminal history based on time. That is sort of the next pretrial for.

Judge, you were a public defender when these were rolling out. How was your experience with a risk assessment tool?

A lot of people were adamant that they were biased and would result in people becoming detained. And nothing is perfect. There issues and things you could debate. But the one thing I did know is I didn't trust my colleagues to just make the decision on their own. That had not worked well. With that being said it was important to give those judges as much information as possible. Good bad, just looking at a file in talking with an officer. And it has improved over the years. But there was a lot of pushback.

I remember a chart, Aaron that we collaborated in creating that looks at variations and judges decisions by court. We weren't allowed to name the judges by name, they didn't want to be identified for being an outlier with one direction or another. But I can see this in my mind’s eye and those bars were all over the place. There was such a huge variation in judicial decision-making by judge in court. We know things have gotten better for sure.

I want to pivot a little bit because in all these examples you have given us there is at least one thread that runs through them. That is the importance of research partnerships.

Every step of the way. You have tremendous in-house experts to be sure. But you are also intentional about reaching out to the research community. Like Samina.

I have a question that I want each of you to answer. What makes for a good research partner?As a researcher you might have an idea. As a practitioner, imagine you do as well. Aaron?

I think it's humbleness all around. It's a lot of times the case, not all but sometimes researchers get it wrong. Knowing how to interpret the data is a critical piece. Without the expertise particularly of people like Janice that know the system inside and out every which way it can be wrong. Researchers that are willing to come as you would expect, is not always the case, collaborate in that relationship particularly in interpretation but even in design. And we have excellent examples of that here. They also have to be willing to take it from the practitioners who will often have critical feedback, some warranted and some unwarranted and just working through that, I think those are two key pieces.

And persistence with some reluctant partners.

Been there. Who else wants to go? What makes for good research?

I feel if I my colleagues it's a willingness to turn up on the ground in whatever meetings and shadow people and understand where the data originated. A lot of folks think you can sit at your office and you research because estate or research. It's more demanding than qualitative research sometimes because you really have to see how the data was entered, by whom and what the incentives were into the data. And have good partnerships and note the folks like Janice really well. I would plead for people just because you are doing data, to also be very actively on the ground with the folks.

And Joan assume that what you see is what it is.

I remember Janice, those early days where the data was so horrible. But there were three fields with almost identical names that were measuring three different things entirely. Or, worse yet, the courts over here versus probation over here and jails all had a different way of calling the same thing something else. How do you merge that all? It's a huge challenge and requires a lot of assistance -- persistence and patience. What is your advice to make a good researcher?

As a practitioner I have learned something new every day. I've been doing this for 37 years. I appreciate talking with the researchers. We've had -- I can just think of our last project we did on the racial equity. It takes a lot of time. We spent hours upon hours. The thing was 180 pages and we got it three days before to vex its.

Sounds like DOJ land.

It takes a lot of time you have to make it a priority bigger honestly, when that came out it was the middle of summer, we had a bunch of stuff going on, we were meeting every other time we turned around and then reading this hundred 80 pages. Both practitioner and researcher you have to make time for that. It's not baked into your duties.

When it matters and you want to make a system better you just have to find the time. We have plenty of practitioners that were upset about some of the findings. They didn't get it they didn't believe it.

Just working through that. Again, some people will be more understanding than others and some will be more open for what you feel like your life's work is being criticized.

Just making sure, and he gets back to relationships.

Building the trust, making sure when I tell you that your decisions stink, and I give you a reason why.

Pour judge, is getting picked on.

I'm right-handed, I just go that way. I will give you a reason for it but we have to discuss it. I won't tell everyone else about it. I won't talk about it when we are in that room we hash it out there and then move forward.

No gotcha research.

I can't stand that.

What else makes for a good researcher?

I think it's the relationship. For myself, we knew we were doing a whole host of things very poorly. We wanted the data and as much information because I always believed it would guide our efforts going forward and save time. Does require you to have difficult conversations. But you have to be receptive and humble enough to take the criticism. Your committed evil value that information. I heard a lot of my colleagues and judges that will say researchers can say what, they always quote the book.

Hear that researchers?

It's easy to come to that conclusion if you are not willing to change. That's the pushback. Now that you have the information you are required to do something. I value the information, I don't always agree and when I don't we hash it out. But I would rather have the information to guide my efforts than just spinning my wheels and do what we've always done because in -- we know it doesn't work well.

I will just echo what everyone said. With the racial disparity project that's always a confident -- controversial topic. Because we already did the RCT study we worked with the stakeholders and there was trust built their and that was helpful. Rand did the quantitative work but it was mixed methods and something that we were talking with all the stakeholders about the process. Once we had initial results and we spoke with the analyst to make sure we are interpreting right and meetings with just the analyst and then we had huge meetings. Like what Janice said you can't do all the work involved stakeholders upfront and then dumped a report with recommendations that people have to implement. We would have loved more time to go over that but I appreciate the infrastructure that Allegheny County has had these huge meetings with different stakeholders.

In that community engagement is important and researchers play a role.

Definitely. It just breaks the whole trust process if you're just going to involve early on and then publish something they are responsible to implement and you don't give a chance to give feedback it just looks badly on research partnerships.

Let's spice it up a little bit.

True or false bigger randomized controlled trials are the best method to test innovation.

Discuss.

I will say false to be controversial. I think there is a role for CTs and the method and a role for people who just summarize the research and just take from it what they can and then try to do it in their local context. There is evidence that RCTs, there is no RCT on RCT with the best research is Elizabeth (Indiscernible) did a paper where she followed a bunch of RCTs and show that just because an RCT result was positive it didn't increase Caleb at all. That is just because a project not a positive effect from an RCT, if you went back 10 years later and asked was that into renovation taken up, there was no more chance that the positive RCT findings were taken up by other counties or say than one that didn't find.

That is fascinating. I will guess it has a lot more to do with how the findings are shared with the field and how the research was generated to begin with. Erin, true or false?

Certainly not an RCT purist. There are lots of ways to evaluate like I said in the beginning, depending on the question, which methodologies will work. When it comes to the cooking of the books and they thought that statistics can be made up or pushed interactions that researchers want to be posted, I find the RCT much more easy to explain. It's clear with the treatment and control groups are, the study makes sense logically. When you get into explaining quasi-experimental design?Has anyone tried to explain difference and different findings?

Have you ever heard a researcher go so far as the state difference and difference? No they just say dif and dif.

Because everyone knows what that means. There can be benefits, with greater adoption. It can be beneficial to explaining those findings.

I have a theory that finding was very compelling I been using it a lot by the way. Is the case because I think people have this notion that the rigor carries more weight than it does. And makes people less inspired to want to explain it because it so easily explained. You are both right. Anyone else want to chime in?

I totally agree with Erin with RCTs, is because it's so much easier to explain to practitioners. But you don't -- RCTs only work insert situations, especially the criminal justice field. Only certain interventions that will support the RCT. You can't arbitrarily take things away. You can do an RCT if not practical. You want to make sure you are using other things.

I have personal experience. The Allegheny family screening tool that we use and we have two replication RCT sites that show a great effect reduction in arm safety another quasi-experimental. None of those work to help me convince new states and counties to adult. I don't think. They might on the margin but sometimes states and counties tell me I can't adopt this tool because I'm too busy doing a new innovative on restructuring, which has zero evaluation. But someone sold it to them because of the fantastic narrative that the intervention has. You don't have a narrative like this. I got evidence and the have a narrative.

We have to wrap the evidence in a compelling narrative. That's the challenge.

I will say this. I'm not a researcher, this is an away game but as a practitioner the RCT was helpful because it allowed me to take the information and go to our County manager and say look at the numbers and information, they didn't have much information and we implemented. For me it was perfect.

When they first approached us with the RCT I'm thinking we should not give certain people things and other people in. Especially if looking at them.

The issue.

Makes me nervous with that.

Unless a scarce research -- resource but it's still hard.

Even though we couldn't cover the shifts I was nervous about studying that because we were not giving people something. If you got arrested at 9 AM you had the benefit of an attorney at a hearing where your liberty was at stake. But if at 9 PM he didn't have that. I was nervous about that. We weren't ever going to get full coverage. It just worked out.

Timmy it was a price I was willing to page because without that experiment and research you couldn't make the case to increase it. It was still better than what we had previously. For me it made sense.

Even with limited resources you are allocating them more equitably than before.

Based on the information.

One of the exceptions, the only people that RCT Alta was those that decided to take the program before the RCT. (Laughter) My point is, for folks like you who decided that it's a good idea, you do an RCT and it confirms your prior and you scale it out. All I am saying is, if someone came to you, Judge, with an RCT from another state or county, with that RCT have convinced you to take their stupid idea?

Findings from an RCT from somewhere else. That the implementation science question to unpack.

I get it. I guess it depends who is the messenger. If Aaron and Janice, I'm all in.

Some entrusted. Okay. That's consistent with the literature. Let's just be clear about something. We touched upon it. I think there are so many successes in Allegheny County we have elected make sound like it was all sunshine and roses. We know it wasn't. I would like to invite Janice to start, what's the opposite of sunshine and roses?

Challenges.

Janice, do you remember that unicorns I leave it on slides?

Aaron would come out -- we would talk and come up with a new idea and then trying to sell it to our group whether jail collaborative -- the thing in Allegheny County is the people doing the work sit on every single group. Those people get invited to every single thing. Aaron would always put a unicorn and rainbow on that this was going to be --

Usually the worst thing ever trying to convince people.

To do something they didn't want to do. One of our big challenges in Allegheny County we have 133 police departments in the County.

It's crazy. One of the challenges is being able to consistently and with fidelity whole the police into the conversations. And one thing we have been missing for years is a consistent voice from the police.

And a lot of changes in leadership over the years. It's hard to get the buy-in from the agency.

And even when we had some consistent participation, it never makes it down to the line staff.

Or managers.

Exactly. We are thinking things are happening and going well and then there is lots of stuff that says it is not. I think we can do a better job here of bringing people into the circle of trust or whatever you want to call it. And I think that is a voice we need. I would also like to bring in the voice of politicians, which we have historically not done. We just pushed it to the side.

We mentioned unicorns and I think it's pretty obvious that Allegheny County is a unicorn in its own right and that you really stand head and shoulders above most any other county in terms of degree of innovation, using data and data integration to solve problems that intersect, that span not just the justice, but the human service system of sharing your knowledge with the fields.

If I'm sitting in the audience from another place I'm thinking that is all well and good but we don't have the data infrastructure. We don't have the degree of collaboration. What is in it for me? What can I take away when we’re hearing from a unicorn of a County? Who wants to take that up?

I can start. I think I already said this a little bit the people just have to start where they are. It wasn't always perfect, but we were still committed together to moving forward and finding a path for innovation and starting where you are we didn't get here in a year, not in six years. This has been 25 years of sustained commitment to the principles and that's how long it takes.

And I think people can hurtle over us. To starting where you are. One of the things in the system in particular, I work in child welfare and mental health and a bunch of other systems, you only note you are making improvements if it's painful in the system. People are uncomfortable. People are angry. People are upset and sometimes friends yelling at one another about these things. The system, at least here, maybe that's not true everywhere, to make reform hurts and is painful. And we are in the midst of that and a judge is calling and someone is calling on the other line and that is happening, we celebrate this moment as moments of progress.

And that is something people just have to get used to in the system. To push through it feels horrible.

Pain is a precondition for progress.

In this condition it just is.

I don't know if others feel that way but I do.

I think for me, I was a public defender and public defenders want to talk about the Constitution and what's mandated and why it's the right thing to do. Those reasons are good but everyone does not show the belief and view. One thing I think we did well in Allegheny County is a commonality. The things we needed to do I can find some benefit for probation. A lot of times you don't know who is in the Java because of the data we learned 50% were in for violation detainer but it wouldn't be in an imposition of additional jail time. I thought we can get this people out of jail quicker and probation reduced in the likelihood -- lesson likelihood they would end up in jail. We can help you shrink your caseload. When you say to probation officers we will reduce your caseload and you can focus on the people that can warrant your attention it is a win-win. So there's a collaboration.

I don't think it was helpful going around telling everyone about the Constitution. Some people know and other people don't care. It was important finding that commonality in common ground.

I witness the culture here in the county around these issues and it's remarkably collaborative, but not without pain. And I hear how everyone goes to all these different meetings. When do you actually do the work? There's so many conversations. Is that one of the challenges of trying to balance your time around being collaborative and doing the day-to-day, much less doing the innovations?

I agree. There is actual work that needs to be done. Trying to get that done and still wanting to push things forward, I think you have to prioritize and make the time.

We have been lucky here that our administration gets that and they allow us to do that. Not that I wouldn't be allowed to come here but I go a lot of different places and do a lot of different things. There is no opposition. We get our stuff done, but we also share a lot with other places. We have the Allegheny standard arrest procedure program and attempt to get all of our police departments to file their charges in one place so we have one place to look for them, which worked. And we wrote that here.

You had over 100 different law enforcement agencies in the county. 132. 42 school districts, three housing authorities.

I think that's where having the internal analytic infrastructure can help. It is in comparison to others is outrageous. The 50 person analytics team and part of their job is to do the criminal justice related work is not to say you don't need the practitioner's time, but we have a whole team whose job it is to go and that's where the internal analytics capacity.

The investment in your own infrastructure.

You have had a huge role in not just spurring the success here in the county but taking it nationwide and internationally. How are you using death helping others scale this up?

The cynicism I have with the RCTs because I feel researchers feel the last piece of work is the publication. I think it's almost irrelevant. I think the way --

I heard applause. I agree.

I think as a researcher and passionate about figuring out whether something is good for a community or bad. And what I said at the start of Allegheny work is if it's bad for the community my job in the future is a stop others from doing it. If it's good for the community my job is to help others do it. It's clear for me, once I have seen the evaluation and so on I think as a scientist I am convinced that objectively this is a good thing without the unintended adverse effects people may have been worried about. It's my job to be scaled up. People, and say this was fantastic they say I love what you are doing and I like how can he do that?I'm like I don't know, go talk to someone. We set up a US-based LLC that actually helps agencies adopt and deploy the same methods that we used here for child welfare for homelessness. And it is working and it's very hard. People want to hear a narrative --

Having toiled as an applied researcher for two decades, how do you make ends meet? I was at a research Institute where the project is over and number funding new go to a new project. Folks in the academic space, they are not incentivized to stop doing what they are doing to get tenure or be promoted to go off and spread the gospel in these other places, right?Who pays for you to do this?

I have to hustle.

You are called to this?

The privilege of being an active is that you follow the thing that mean something to the community you live in. It's not -- I don't care what incentives they place in front of me, I really don't. This is a calling and it should be a calling. Is a privilege to serve our most vulnerable to science and we should do it in a way that's true to the calling. We are one and hundreds of people and thousands of generations to do this. (Applause)

I believe that Allegheny County, or at least department of human services is also supporting your work in this space. That's a great exemplar in and of itself. That you are investing and spreading your experiences and successes not just nationwide but across the world. Which also deserves a round of applause. (Applause)

I know we have a lot more to say but I also promise that we would allow members of the audience to ask some question. We will raise health life a bit, it takes a minute, be patient. We have Mics set up in the aisle. If you have mobility issues and would like to ask a question just raise your hand and we will get a microphone to you. While people are trying to get the gumption to ask a distant, it always takes at least one to start. We will do a lightning round.

If there is one more soul of information to share with the audience, what would it be?We will start with you, Judge.

You have to work hard and be receptive. With disagreements you have to work but you can't just disregard the information.

I think if you get an outcome or some information that you were not expect that your whole system understands that you are ready to pivot and go in a different direction.

Use whatever data you want, don't wait for the perfect integrated data set. You will have 100 improvements to make with the data you have now.

Just to highlight, the fact that Allegheny County has we centralize a lot of the data streams from the other agencies it allows you to do so many research projects that are more difficult in other areas big a lot of times you will find the course own a piece in the jail owns a piece. And makes it difficult. Being able to centralize your data allows you to do a lot more.

Start where you are. You should start by building integrative data. I cannot imagine making the decisions we have to make, the investment decisions without the information we have. I cannot imagine a day where we would have to go back and do it without it because much as you start where you are, you should start.

That's great. I will add one from the group that I heard, you do not underestimate the importance of relationships and establishing strong relationships. That is not a one and done. It takes hard work, it's painful, but it does pay off. I think we have some people. I can barely see. If you are behind the we will start on this side and then we will go back and forth.

Hi, my name is Monique Kelly. I work at the Department of Veterans Affairs. This is my first foray into this field of forensics and crime. The VA, historically, has shied away a little bit from the overlap of veterans with crimes. They have focused on health factors associated with crimes such as trauma, trauma informed care, justice outreach programs. Things that are addressed after the fact.

I really curious and interested about the department of human services and in Allegheny County and the overlap with the VA, and any kind of programs associated with those data.

My role in the VA -- I'm sorry to interrupt but we have half a dozen people that want to ask questions. Can we stop at that they are interested in knowing what the county is doing around homelessness and veterans and the intersection maybe with the justice system?

Sure.

Let's be snappy with the response.

We do have a veteran score and I think that's worth highlighting. And I think it is another place where we have veterans efforts on the ground. But it is hard to cross that boundary and work with the federal system and integrate and do that care together. I think that is a frontier in the work. When it comes in the criminal justice system the veteran score, it's my favorite.

NIJ as funded courts for the purpose we need more knowledge.

Real quick. We collect information on veterans. That's one of the things in the pretrial stage we collect so we can direct the veterans to different programs we have. And we -- the court works with the VTC's and VJO.

Let's move to a question on this site.

I am from coaches. I'm really interested and curious come Allegheny County is integrated health systems which is fascinating. But now we move into a moment where Medicaid and Medicaid systems are moving inside the jails through statutory and regulatory changes. And curious how the county is thinking on how to leverage these huge gigantic data sources and how NIJ is thinking about what the data can do to drive and help us understand the underlying causes maybe in the justice system to begin with because of underlying health causes.

That was for both of us, perhaps. Sethi 1115 waivers, people can Google that, are the most exciting thing happens in a long time. This will allow for better care inside the jails because they will have to be a Medicaid standard and the jails will have to come to that standard.

We are preparing quickly. The sponsor the states and the states have to decide how to work with the counties, work immediately and we are trying to push for that. And preparing the infrastructure, a lot of which we have to run a Medicaid system inside those facilities. We are excited and to sip a not and hoping to provide more and better care as more of entitlement. Right now we have to prioritize the highest risk of the most vulnerable inside the jail. But if this becomes more of entitlement that means services for all.

Thanks. NIJ is supporting research in this space, that's my answer.

Good morning, Ginger Barrett with the office of violence against women of the Justice Department. I been following the (Audio cutting in and out) screening tool at a professional curiosity. But also as a foster parent that sees child welfare decisions up close and personal every day. My question for Rima and Aaron is about any kind of iterative process and when you make decisions about revising the data points or the weights they are given.

Just to get everyone else on speed the Allegheny screening tool is a predictive risk model based on machine learning approaches because essentially the weights are determined by that statistical method. We updated and change it based on the protectiveness and quality of the data coming in. It is not a human deliberated set of weights.

With that said, we have, over time, decided not to use certain data in the model. There's always a little bit of push and pull in that. But in that particular model we want the best model. Not all of our models, willing to make sacrifices, but with a child safety decision we want the best model we can have.

My quest with the King County Department of adult and juvenile detention in Seattle, Washington. When one implements a new pilot program you spent lots of staff time, resources getting it in there. Then you do the evaluation and the evaluation comes back negative. Either no positive change or a negative change. How to get people to stop doing it based on the evaluation? (Applause)

I have a painful story. The first thing I worked on when I got to Pittsburgh was a violence prevention program, a street outreach program. And I was stupid enough to think a NIJ funded evaluation really early on in the implementation of a program was a good idea. And different researchers conducted that pickup it was a strong evaluation design and found no effect. There is no logic model to suggest it did harm but it found hard. The researchers throughout the entire ethnographic portion of the evaluation as gone native, which is a separate thing we can discuss on a different day. But that program lost its funding. And they do nothing but go back and create versions of a similar thing.

I learned some hard lessons and I think the question is a good one. It's hard to get people to stop think about one of the hazards in a place like Allegheny County were people by the data, I couldn't talk my way out of that one. The findings were clear and it went away.

That's a real lesson.

Lots of mistakes on my part.

We learned at NIJ that we should fund evaluations of things that are brand new, maybe formative evaluation to have those programs on the research partnerships be well-positioned for an impact evaluation. But going right impact is just setting the programs up for failure. Thank you.

Who is next?

Good morning and thank you for this, it's been fascinating. My question is sort of related to the gentleman that just spoke. Curious if you have anything else to add. We talked about when the research supports a practice and maybe the chief supports like you have some buy-in. But the folks on the ground like the street level folks sort of continue with business as usual. What are our thoughts on potentially incentivizing implementation, especially with folks on the ground? And if we have any kind of positive attitude toward that, what are some of the ways that we can actually do that?

I think that getting people to filter all the way down to the people that are doing the work, when spending so much time at this level, looking at everything. You have to talk to them and you have to meet with them constantly. I've had -- we can incentivize some things and I figured the incentive would be your patient because that's just my opinion. (Laughter)

I think it's just meeting with them and talking with them. And I think people want to truly know why you are doing this and what that research is. You have to share that with them. We have something that we started back in 2006 called the change committee. I've done it in every job that I went to and I have people in all of the departments from clerical, supervisors, line staff. In any time on looking to make a big decision, I run it by that group. And they are doing the work every day and they can tell me that is a stupid idea. If you do that, this will happen. I make sure I am running all that stuff through. And that gives people the opportunity to participate and feel vested. And it also gives the limited staff that's on this the opportunity to go back and tell everyone else what is going on.

It is important to communicate what is in it for them. And may not be a clear incentive like you will make more money. But it could be your workload will get lighter. And I think we heard that from you, Judge, earlier in this conversation.

We just have three more and we will stop here.

I am project teams from the Yale school of medicine and Elm city Compass. A terrific panel, thank you. You covered big picture issues as well of the details of building and sustaining data-driven approach. Have you involved and how can you involve community members? Especially those with lived experience in sustaining this kind of approach?

I will start by saying one of the key things we built with the integrated data, which might be a surprise to folks is data for people themselves. Right now if you live in Allegheny County you can gain access to your own integrative record figure with the idea that who better than people to manage their own care. When you are willing to do something like that and give data back to individuals directly I think that is the first sign. When we implemented big, new changes to the system we got advisory boards, community engaged advisory boards, homelessness model comes to mind and decision-making. We met with people experiencing homelessness on how it would change for them and how the system may be easier. They don't have to call every day and say yes, I'm still almost and still have substance abuse in the history, all those invasive questions. But there's some agency in answering those questions that will be lost when you have a data-driven model.

As we have gone, we have really tried to involve people in the process. Starting with giving all the integrative data back to people themselves.

As a researcher I go to lots of committee meetings and so on. My summary of what I think the community wants from people like me is to know our values and don't need to know the science. They need to know they are heard in the problems they have with the way I am taking things on. Note my value is in listening to them and thinking about what they told me and how it should change how I do my science. They don't want to know the insights of the science.

We have two minutes left and two questions. Can each ask your question and then we will try to answer both in two minutes?

I can start. Hello I am Aaron quick. One of the student scholarship award is an PhD student. (Applause) (Laughter) Thank you. My question more is related to kind of the future. As we start to take on the resource positions and policy positions, how do you suggest that we navigate this pipeline of research using our findings to inform data and then getting past? Specifically when our research and policies are dismantling racial disparities, gender disparities. Many times you have a political ideology that prevents it from being passed. How do we navigate that and find interest convergence without foregoing implementing policies that really dismantle disparities in our systems?

Thank you. Unless question.

I am Colleen O'Toole, a prosecutor in Ohio. I've been doing a lot of research. There is research on the role of the prosecutor and the fact that they are a linchpin to everything that happens. My office prosecutes every juvenile case in Everett neglect dependency case, every felony and misdemeanor. The sole issue of how I charge, when I charge, and what kind of restorative justice or not. Is our lot being done there? I see public defender and research and wondering what they are doing around that?

One question about the researcher as passionate to make change especially around issues of equity and justice. How do I speed that up and accelerate it and get those findings. And a question on the role of the prosecutor and what we are doing. I will answer the second one first from NIJ's perspective. We funded a lot of research in the space because we recognize that so much rests in issues around plea-bargaining and the court working group, what goes on behind those doors and who has control of these decisions and how they get made, and getting better data. We are investing a lot in that space. But I welcome you all to address that, or the first question around getting the evidence to making those policy reforms.

Our chief deputy from the District Attorney's Office is on all of our working groups. I don't know. This was the group (Audio cutting in and out) we have lots of people that aren't here on the group. And Warden. There's plenty of people, a bunch of Aaron folks that serve a lot of committees. They are definitely involved in everything we do.

Say it like it is, Judge.

But not nearly as involved as they should be.

All right. Thank you.

Going to the first question and where the judge is that. It's about finding common ground. Of the findings in the critical racial disparities report, we have tables that place against one another these findings and how big of an impact we think they can have and how hard it would be to implement. You have to balance those in a quadrant and work on the ones we think you can make reform today and set the stage for the ones you can make reform on in the future.

Very good. Let's give this panel a round of applause. (Applause) Thank you so much. Before anyone goes anywhere, if you are NIJ staff, please come to the stage to take a picture. For the rest of you, you are off to enjoy a lovely 30 minute break before you find your panel room. We will see you around. And look at this. The URL. Get your phones out.