High-profile acts of violence strain the relationship between police departments and the communities they serve.[1] NIJ and its agency partners are working on ways to mend these relationships and help each side heal.
In support of its Transforming Victim Services Initiative (Vision 21), the Office for Victims of Crime (OVC) awarded a competitive grant to the International Association of Chiefs of Police (IACP) in 2016 to oversee the training and technical assistance for the Collective Healing Initiative. The initiative uses community-informed strategies to improve ties between law enforcement agencies and the public prior to and in the wake of police-involved shootings and other traumatic events.
NIJ funded RTI International (RTI) to conduct an evaluability assessment and baseline study of the Collective Healing Initiative in partnership with OVC in line with Vision 21’s promotion of evidence-based practices.[2] RTI both assessed the program as a whole and assessed the individual demonstration site agencies in Baton Rouge, Louisiana; Houston, Texas; Minneapolis, Minnesota; Oakland, California; and Rapid City, South Dakota.[3]
RTI concluded that the Collective Healing Initiative is not yet ready for an outcome evaluation because the program as a whole lacks well-defined program elements. They followed up this finding with considerations and recommendations for a future cross-site evaluation.
RTI determined that despite the program sites having shared objectives, the individual sites’ varied goals and implementation plans make them too dissimilar to compare. Therefore, RTI developed site-specific logic models that could produce data to support a process evaluation for each program and ultimately inform the development of measurable objectives and indicators of a site’s success. The site-specific logic models included detailing implementation activities and their potential outputs, outcomes, and impacts.
Overview and Objectives of the Collective Healing Initiative Program
IACP selected the five city police departments through a competitive process to serve as Collective Healing Initiative demonstration sites based on their differing sizes, demographics, income profiles, and crime rates.
IACP provided training and technical assistance to the police departments and their community partners to design a program to identify and address specific needs in their cities. Although the programs were customized for each location, they shared the following three objectives:
- Boost officer and agency wellness and resilience.
- Enhance victim services and assistance.
- Improve community-police relations and community wellness.
Evaluability Assessment
The RTI research team used a mixed-methods approach to determine if the Collective Healing Initiative was ready for an overall evaluation and for individual site evaluations. The team gathered data through in-person demonstration site visits and semi-structured interviews on five program-planning and implementation topics:
- Type and extent of police department partnerships with local organizations and agencies.
- Approaches used to develop the demonstration site program.
- Community perceptions of the program.
- Police department perceptions of IACP’s training and technical assistance.
- Future evaluability of the Collective Healing Initiative overall and locally.
RTI concluded that the Collective Healing Initiative was not yet at a point where an overall outcome evaluation would be meaningful because it rated “low” across all evaluability criteria in its current state, including agreed-upon purpose/goals, programmatic actions/interventions, fidelity to program, availability of baseline data, availability of process and outcome data, sample size, and causality.
RTI provided detailed considerations and recommendations for a cross-site evaluation at a later date.
Except for the program in Rapid City, South Dakota, local Collective Healing Initiative programs were also not ready for a site-specific outcome evaluation. This was due to early-on changes in programmatic focus and partnerships, which led to downstream delays in demonstration site planning and implementation.
RTI recommended process evaluations for local program sites to better understand how the sites developed their collaborative teams, how they selected program activities, what the programs look like in practice, and how to measure program success. For demonstration site process evaluations, RTI recommended using site reports and visits, web-based surveys, and IACP’s self-assessment tools to collect data about:
- Amount and type of training and trainee characteristics.
- Satisfaction with and perceived value of training.
- Number and type of new services, and characteristics of the victims or police officers who are using the new services.
- Number and type of community events, and participant characteristics.
RTI also recommended several cross-site measurable outcomes that address the Collective Healing Initiative’s three objectives, including:
- Improved trust between police department and partners.
- Increased job satisfaction among officers.
- Expanded network of victim service providers.
- Increased communication between police departments and communities.
These data will also inform future outcome evaluations and help site evaluators determine when a demonstration site can expect tangible outcomes.
Recommendations for Future Collective Healing Initiative Program Sites
For future community-based programs such as the Collective Healing Initiative, the RTI team recommended building site-specific logic models, developing measurable outputs and outcomes, and identifying potential data sources at the proposal stage. The team suggested the following mechanisms to improve the feasibility of a rigorous evaluation:
- A research partner as part of the program’s training and technical assistance team to develop an evaluation framework that applies to all demonstration sites.
- An evaluation of demonstration sites to identify opportunities for small cross-site evaluation partnerships around shared program criteria.
For a cross-site evaluation of a community-based program, RTI recommends using an external evaluation team to be funded in phases depending on the outcome of an early program-wide evaluability assessment. This phased approach would ensure that evaluation activities were chosen based on program needs.
About This Article
The research described in this article was funded by NIJ grant 2017-VF-GX-0006, awarded to RTI International. This article is based on the grantee final report “Evaluability Assessment and Baseline Study of the Supporting Collective Healing in the Wake of Harm Program” (April 2020), by Kelle Barrick, Elizabeth Tibaduiza, Caitlin Dean, Amanda Young, and Merissa Gremminger.