Confidence in our nation's criminal justice system rests on several core beliefs. First, we believe that most justice work is fairly routine, following a predictable path that makes errors rare. Second, we believe that in the rare instances an error does occur, it is the result of simple negligence or individual misconduct, which "the system" can readily detect and fix. Finally, we believe that processes are in place to ensure that similar errors do not happen again.
The truth, however, is that these beliefs may be largely unfounded.
Since 2011, NIJ has been investigating the feasibility of using sentinel event reviews (SERs) in the criminal justice system. Put simply, the theory is that when a bad outcome occurs in a complex social system — like our justice system — it is rarely the result of one person's mistake. Rather, multiple small errors combine and are exacerbated by underlying weaknesses in the system.
One of the godfathers of this theory of human error and organizational processes is James T. Reason, a renowned British researcher whose work has been used to improve safety in medicine, nuclear power, financial services and aviation. Dr. Reason writes, "We cannot change the human condition, but we can change the conditions in which humans operate."
Indeed, it is this principle that lies at the core of NIJ's Sentinel Events Initiative (SEI). The goal of the Initiative is to change the conditions — or culture — in which criminal justice practitioners operate. Just as medicine and aviation have used sentinel event reviews to instill a "culture of safety," NIJ's Initiative explores a routine, culture-changing practice that would lead to greater system reliability and, hence, greater public confidence in the integrity of our criminal justice system.
Sentinel Event Reviews: The Basics
A "sentinel event" is a negative event or outcome that:
- Signals underlying weaknesses in a system or process.
- Is likely the result of compound errors.
- May, if properly understood, provide important keys to strengthening the system and preventing similar adverse outcomes in the future.
In criminal justice, a sentinel event could be the conviction of an innocent person, a police-citizen encounter that unexpectedly turns violent, the release from prison of a dangerous person, or even a "near miss" that could have led to a bad outcome had it not been caught in time. An SER brings together all of the system's stakeholders (law enforcement, crime laboratory personnel, prosecutors, defense lawyers, judges, corrections officials, victim advocates and others, depending on the event) to review the event and determine — through a deliberative, transparent, nonblaming process — how and why it happened and what can be done to prevent a similar outcome in the future.
From Then to Now
NIJ's work began with a research question posed by Visiting Fellow James Doyle, a criminal defense lawyer from Boston: Could SERs, which have successfully been institutionalized in fields such as medicine and aviation to improve outcomes, be adopted in the criminal justice system?
In his two-year fellowship at NIJ, Doyle performed what social scientists call "key informant interviews," talking with criminal justice practitioners and researchers from around the nation. Doyle's vision — and the positive reactions he received from boots-on-the-ground practitioners, top executives and others throughout the federal, state and local justice systems — helped launch the SEI.
The SEI seeks to answer three empirical questions about using SERs in the justice system:
- Can it be done?
- Does it help decrease error, increase effectiveness and produce other public safety dividends?
- Can it be incorporated into the routine activities of state and local justice processes and sustained over time?
To date, NIJ has reached a number of significant milestones in its work on the SEI. First, NIJ brought together criminal justice experts and potential early adopters to vet the concept. Second, the Institute published Mending Justice: Sentinel Event Reviews, a special report that discusses Doyle's two-year "reconnaissance" and includes commentaries from former Attorney General Eric Holder and 16 highly respected criminal justice practitioners and researchers. In 2014, NIJ funded two research projects and followed that up with an NIJ-supported pilot, or "beta," project in three jurisdictions to test the first empirical question: Can it be done? And in 2015, NIJ funded two additional research projects to dig deeper into the best ways to bring SERs into the justice system.
See "A Glimpse at Ongoing Research Projects."
Testing the Concept: The Beta Project
In 2014, NIJ asked jurisdictions from around the country to volunteer to perform an SER. Through a competitive process, it selected three sites: Milwaukee, Philadelphia and Baltimore. NIJ provided minor logistical support, but no funding, to the beta sites.
"It was very rewarding to see the courage and commitment that the beta teams in these three forward-leaning jurisdictions showed," said Katharine Browning, a social scientist who heads up NIJ's SEI. "They are true pioneers."
Each site designed and conducted its own review of an error (a sentinel event) that had occurred in its justice system. Earlier this year, the sites successfully completed their reviews, providing empirical evidence of the feasibility of adopting SER in the justice system. A summary of the findings from the beta project — Paving the Way: Lessons Learned in Sentinel Event Reviews (pdf, 24 pages) — covers issues such as:
- How do you choose the right negative event, case or outcome to review?
- Who should be on the SER team?
- Who should lead — or serve as the facilitator of — the review?
- What does "nonblaming" really mean?
- How do you manage the need to share sensitive data and information with others?
- How do you measure impact and outcomes?
Katherine Darke Schmitt, a policy advisor to the Assistant Attorney General in the Office of Justice Programs who is working with the NIJ team, visited the three sites and interviewed members of the SER teams. Her assessment revealed three overarching themes when it came to the sites' choice of a negative event to review: mitigation of legal risk, the age of the event and the need to have broad system participation.
"Perhaps the single most important procedural question facing SERs in the criminal justice system is whether criminal or civil actions regarding the event have been resolved," Darke Schmitt said. "It doesn't mean an SER can't be done if such actions are still pending; it only means that the stakeholders may need to take actions to mitigate any existing or potential legal risks."
Leading an SER
The leader — or facilitator — of an SER in a jurisdiction operates a bit like a project manager. Whether the facilitator is an independent, neutral convener (such as a researcher from a local university) or someone from within a key criminal justice agency, this leadership position requires significant time and energy.
In two of the beta sites, the facilitator was an outside researcher; in the third site, the facilitator came from a criminal justice agency. In all three sites, the facilitator was a fully engaged member of the review team.
Although it remains an open question who is in the best position to lead an SER in the justice system, Maureen McGough, a policy advisor at NIJ and member of the SEI team, said the beta project suggests a variety of workable alternatives.
"That said," McGough added, "some key traits emerged for the role of facilitator: She or he must be intellectually curious and well-informed about the philosophy of SER, possess strong facilitation skills, and be both trusted by the other team members and able to hold them accountable for their participation."
The "Blame Game"
One of the key components of SER is a systemwide process that is less concerned with fixing blame on one person or agency and more focused on getting to the root(s) of a bad outcome. In her commentary in NIJ's special report, Mending Justice, Jennifer Thompson addressed this issue head-on.
Thompson, a sexual assault survivor who lobbies for nonblaming learning-from-error reform in the criminal justice system, titled her piece "The Blame Game." She writes, "[B]lame and fault have never answered the big questions, such as, 'How did this [error] happen in the first place?'"
Of course, determining blame and fault is central to what the justice system does. By its very nature, the system is adversarial, which may make the core "nonblaming" component of an SER a particular challenge. Some members of the beta SER teams noted that they were natural adversaries in their day jobs and had to work hard to overcome a reflexive defensiveness.
But, as NIJ Deputy Director Howard Spivak points out, systems such as aviation and medicine have been able to overcome their similar tendency toward assigning blame. Spivak, a physician who was involved in bringing SERs into medicine following child deaths, noted that NIJ's SEI is grounded in the same principles that led to "culture of safety" reforms in other high-risk fields.
In aviation, for example, the National Transportation Safety Board uses an SER approach to analyze airplane crashes and near-misses and publishes the results online.[1] A summary of these reports also appears in magazines, such as Flying.
"The result," said Doyle in Mending Justice, "is an aviation community — including manufacturers, aviators, airlines and regulators — that is informed about the current lessons of recent errors."
In medicine, the accrediting body for hospitals, the Joint Commission, requires the reporting of sentinel events, defined as "unexpected occurrences involving death or serious physical or psychological injury, or the risk thereof."[2] Hospitals also are required to conduct a "root cause analysis" of each event. The Joint Commission compiles these analyses and periodically — and publicly — issues "Sentinel Event Alerts." In addition, the Institute of Medicine and other medical communities use SER to learn from errors.[3]
"At their core," said Doyle, "the safety reform movements in medicine and aviation depended on laying aside a tradition of 'blaming and shaming' and moving toward a sentinel events approach for reviewing and learning from errors and near misses."
See "Successful Models for Change and Challenges for the Criminal Justice System."
SER: Not Just Another Review
Members of the SER beta teams pointed out that we seem to be living in an era of organizational fatigue, which may make it more difficult to convince jurisdictions to institute SERs. Therefore, it is important to understand that an SER is different from other types of review processes that may already exist in state and local jurisdictions' criminal justice systems, such as after-action reports, task forces, independent monitors, commissions, inspector general reports, internal affairs investigations and performance management systems like CompStat. As noted earlier, one of the most significant ways in which an SER is different: It is not about assigning blame.
John Hollway, associate dean at the University of Pennsylvania Law School and executive director of the Quattrone Center for the Fair Administration of Justice, facilitated the Philadelphia beta site's SER team. Hollway said that it cannot be emphasized enough that SERs are not performance evaluations.
"Their purpose is learning, not punishment," he said, adding that personnel and discipline issues are handled through separate processes, which is something that other fields — such as medicine and aviation — have worked out, including by making the results of a review inadmissible as evidence in litigation.
Measuring Impact and Outcomes
In an era of having to do more with less, the institutionalization and sustainability of the SER approach in the criminal justice system will require demonstrable results — not only in policy and practice but eventually in the workplace and systemwide culture. To succeed, SERs must come to be regarded as a good use of people's time and energy, which, of course, will require proof.
"Therefore," said Thomas Feucht, NIJ's senior science advisor and member of the SEI team, "our second research question is to determine whether the SER approach produces measurable outcomes. In other words, does the SER approach improve system functioning, help prevent errors or achieve other public safety dividends?"
No one maintains that measuring progress toward achieving outcomes will be easy, but proponents of bringing SERs into the criminal justice system note that the medical field has established metrics for patient safety, and the aviation field has done the same with respect to airplane crashes and near-misses — so there is confidence that similar metrics could be developed in the justice system.
Despite the challenges in measuring the success of an individual SER — let alone a learning-from-error cultural shift that may accrue over time — the beta project SER teams suggested these possible metrics:
- Were recommendations for policy or process changes produced?
- Were the recommendations presented to decision-makers?
- Did SER team participants value the process?
- Were similar potential sentinel events averted in the future?
As NIJ continues to explore the use of SER in the criminal justice system, it will be important to work with early adopters and other champions of the process. Could multiagency commissions or coordinating councils, such as local criminal justice commissions, be supporters? Could risk managers play a crucial role? In the medical field, for example, insurance companies began to see the value of SERs in improving outcomes. If a city or county's risk managers are a logical counterpart to health insurance companies, could they serve as change agents if they believe that using an SER approach in the justice system could result in cost savings?
Conclusion
Recent events on the national stage have highlighted the need for a science-informed approach to making system improvements that go beyond placing blame and disciplining rule-breakers. In the wake of events in Ferguson, Missouri, President Barack Obama established the President's Task Force on 21st Century Policing, which recommended SERs as a way to improve public confidence in the legitimacy and accountability of law enforcement.
Millions of people work in individual criminal justice agencies across the U.S. — law enforcement, crime laboratories, the prosecution and defense bars, judges, corrections, victim advocates and service providers — and, as beta team members noted, they rarely have the opportunity to participate in this sort of review of error.
"It is very hard to step back and take a 30,000-foot view," said one beta team member. "But it is incredibly important that we do so."
For More Information
Learn more about implementing SERs in the criminal justice system in Mending Justice: Sentinel Event Reviews (pdf, 68 pages).
Learn more about NIJ's beta project in Paving the Way: Lessons Learned in Sentinel Event Reviews (pdf, 24 pages).
About This Article
This article was published as part of NIJ Journal issue number 276, published December 2015.
This article discusses the following grants:
- "Case Deconstruction of Criminal Investigative Failures," grant number 2014-IJ-CX-0037.
- "A Sentinel Events Approach to Addressing Suicide and Self-Harm in Jail," grant number 2014-IJ-CX-0030.
- "Instilling a Culture of Continuous Learning From Criminal Justice System Errors: A Multi-Stakeholder Sentinel Event Review of Process in Philadelphia," grant number 2015-R2-CX-K040.
- "Sentinel Events: A Sustainable Model for System Change," grant number 2015-R2-CX-K041.