Since the tragedy at Columbine High School in 1999, one of the worst school mass shootings in U.S. history, school administrators and researchers in Colorado and elsewhere have been searching for solutions to identify and mitigate the impact of bullying and school climate on threats to school safety. Fortunately, numerous studies over the past two decades have focused on developing and evaluating evidence-based prevention programs to improve the safety and well-being of American youth at school.
The prevention of school-based violence can be especially important at the middle school level, as children experience key physical and developmental changes during this time. Some adolescents become particularly vulnerable when faced with the pressures of peer influence – often experiencing a decline in bonding and socialization and beginning an antisocial trajectory that can lead to poor school performance, aggressive and delinquent behaviors, and even subsequent dropout. Establishing a positive middle school climate can help students successfully navigate adolescence, resulting in better outcomes for students and safer school communities overall.
Recent research suggests that a comprehensive approach to school safety offers potential to keep schools and students safe, integrating scientific evidence and best practices for strategy implementation.[1, 2, 3, 4] With that in mind, researchers from the University of Colorado Boulder’s Center for the Study and Prevention of Violence partnered with educators in Colorado middle schools to implement and evaluate the Safe Communities Safe Schools (SCSS) model.
The goal of the model is to use a comprehensive framework at schools to prevent and reduce behavioral incidents, address mental and behavioral concerns, and increase prosocial behavior through (1) developing a functioning school-based team, (2) building capacity around data use, and (3) selecting and implementing evidence-based programs. Researchers examined the ability of schools to implement the model as designed, as well as whether the model affects school climate and student behavior at school.
Specifically, the study explored research questions in three areas: readiness, implementation, and associated outcomes. Researchers asked:
- Do schools have the readiness, or the ability to increase their readiness, to implement the model?
- Is the model being implemented as intended?
- Is the model feasible, acceptable, and effective when implemented schoolwide?
- What is the effect of the model on school climate, safety, related behavioral and mental health indicators, and academic outcomes for youth and staff; and how do these impacts vary in association with implementation?
The project included educators, students and their families, and the broader community from 46 middle schools in Colorado. Researchers studied the outcomes of two cohorts (10 schools and 36 schools, respectively) after years one and two using a randomized, controlled trial with staggered SCSS model implementation. Evaluations employed both qualitative data from open-ended questions on surveys and from focus groups and quantitative data from staff and school climate data, attendance and truancy rates, suspension rates, and academic achievement data.
Assessment of Readiness, Implementation and Feasibility, and Program Effectiveness
School readiness
Results from the study indicated that schools did indeed have sufficient initial readiness to implement the SCSS model. Moreover, readiness increased (in some areas) during the study period.
- Of the 60 schools that applied for the study, 46 met the readiness criteria and agreed to participate.
- For 6 of the 20 readiness subscales, including staff capacity, researchers noted a significant increase in readiness levels.
Implementation and feasibility
Most of the participating schools implemented all three core components of the SCSS model with fidelity in the first year and about half of the core components in the second year.
- About half (48 percent) of all treatment schools completed all core components.
- Implementation challenges included: turnover of school leaders and school-based coordinators, degree of relevance of evidence-based program components for the student population, program fit (i.e., appropriateness for the school), and perceived value as assessed by program staff and researchers of the program during implementation.
Program effectiveness
The model yielded limited impacts on school climate, safety, behavioral and mental health indicators, and academic outcomes, with outcomes varying by degree of fidelity to model implementation. Researchers reported mixed outcomes, in that:
- Total suspension rates and in-school suspension rates decreased significantly in year one but did not change significantly in year two.
- Academic achievement, as demonstrated by test scores, increased significantly in reading in years one and two, but math scores did not change significantly.
- Staff perception of school climate did not change significantly.
- Attendance and truancy rates showed no significant impacts.
- Student perception of school climate was negatively impacted, with significant outcomes for two peer norms and two types of behavior, all in the unexpected direction.
Implementing Evidence-Based Programs Takes Time
Evidence-based preventive efforts have the potential to reach the highest-risk youth, to promote prosocial behavior, reduce teen suffering, and increase health equity. [5] Given that some components of the SCSS model were implemented as intended and were acceptable and effective, the study suggests that it is feasible within a two-year period to support a school in developing a functioning school-based team and use data to implement an evidence-based program. However, researchers determined that full implementation in schools generally takes two to four years.
Because of the time needed to fully implement the program, the study may have benefited from a longer follow-up period to assess the effects of the model on school climate and associated behavioral indicators. In addition, researchers noted that in the future it could be beneficial to conduct evaluations that assess intermediate outcomes that may impact longer-term outcomes, such as students’ perceptions of safety, increased attendance, and academic achievement.
The Value of Readiness Assessments and Implementation Data
Readiness assessments provide actionable information for program implementers, guiding decisions about training and technical assistance that can build implementation capacity. The multi-phase readiness approach used here, which included conducting feasibility visits and using criteria based on readiness to select schools for comprehensive implementation, offers lessons that are translatable and applicable for other comprehensive schoolwide efforts.
To effectively bring the SCSS model to scale schoolwide after programming is introduced, implementation data can prove invaluable. The implementation data in this study demonstrated that the SCSS model helps to build team members’ readiness and capacity to implement the framework, but this increased knowledge may not extend far beyond the core team. Several factors likely contribute to this issue, including turnover of school leadership and staff and concerns about the relevance, fit, and perceived value of the evidence-based program.
The limited desirable results described here highlight that this kind of comprehensive, schoolwide intervention produces change one piece at a time, rather than in all areas at once. The field could benefit from a deeper understanding of what the most effective scope and sequence for comprehensive change efforts is, as well as of what the ideal implementation change process looks like.
About This Article
The work described in this article was supported by NIJ award number 2015-CK-BX-K002, awarded to The Regents of the University of Colorado.
This article is based on grantee report “Assessing readiness, implementation, and effects associated with a comprehensive framework designed to reduce school violence: A randomized controlled trial,” (PDF, 25 pages), by Allison B. Dymnicki, Beverly Kingston, Sabrina Arredondo Mattson, Elizabeth Spier, Susanne Argamaso Maher, and Jody Witt.