Sentinel Events Initiative: Looking Back to Look Forward

DNA exonerations of wrongfully convicted defendants have thrown a new light on the problem of error in American criminal justice and have revealed a gap in our system's design. We lack a feature that medicine, aviation and other high-risk enterprises see as critical: a way to account for tragic outcomes that no one intended, learn lessons from these errors, and use these lessons to reduce future risks.

Can the criminal justice system develop this capacity for "forward-looking accountability"?[1] Can we accept error as an inevitable element of the human condition and study errors (or "near misses") in a disciplined and consistent way? Can we share the lessons learned from these studies to prevent future errors? Can we focus on minimizing future risks instead of on blame Fifor the past?

To explore these questions, NIJ has launched a "sentinel events" initiative. A sentinel event is a significant, unexpected negative outcome — for example, a wrongful conviction, an erroneous release from prison or a cold case that stayed cold too long — that signals a possible weakness in the system or process. Sentinel events are likely the result of compound errors and may provide — if properly analyzed and addressed — important keys to strengthening the system and preventing future adverse events or outcomes.

As part of its sentinel events initiative, NIJ convened a roundtable of experts in 2013 to discuss the applicability of a sentinel-events approach — a non-blaming, forward-thinking, all-stakeholders approach — to improving criminal justice outcomes. The eclectic and veteran group, drawn from all criminal justice stakeholders, agreed that NIJ should step across the threshold and begin to critically test the viability of a sentinel events approach.

How Things Go Wrong: The Wrong Patient, The Wrong Man

One way to see the learning opportunities presented by criminal justice sentinel events such as wrongful convictions is to examine contemporary medicine's encounter with its own version of the problem: "iatrogenic" injuries to patients, or harm caused by medical treatment.

The criminal justice system is haunted by the fact that it sometimes convicts the wrong person; the medical field is haunted by the fact that it sometimes operates on the wrong patient. When modern medical researchers began to look carefully at wrong-patient events, they uncovered surprising insights. For example, one intensive examination of a wrong-patient surgery discovered at least 17 errors, including that the patient's face was draped so that the physicians could not see it; a resident left the laboratory assuming the attending physician had ordered the invasive surgery without telling him; conflicting charts were overlooked; and contradictory patient stickers were ignored. But the crucial point for the researchers was that no single one of the 17 errors they catalogued could have caused the wrong-patient surgery by itself.[2]

The researchers' analysis showed not only mistakes by individual doctors and nurses but also latent systemic problems. Communication between staff was terrible; computer systems did not share information. When teams failed to function, no one was alarmed because of a culture of low expectations that "led [staff] to conclude that these red flags signified not unusual, worrisome harbingers but rather mundane repetitions of the poor communication to which they had become inured."[3] Deviations from good practice had become normal — and tragedy resulted.

The findings showed that the wrong-patient surgery was an "organizational accident." No single error can cause an organizational accident independently; the errors of many individuals ("active errors") converge and interact with system weaknesses ("latent conditions"), increasing the likelihood that these individual errors will do harm.

These insights can apply to "wrong-man" convictions and other criminal justice system errors.

Many things have to go wrong before the wrong man is convicted. Yes, the eyewitness has to choose the wrong man from a photo array, but law enforcement has to decide to put him into the photo array in the first place as well as design and administer the array. Forensic evidence on the crime scene may have been overlooked or — even if properly collected and tested in the laboratory — distorted in the courtroom presentation. Cell phone, Metro card or other alibi information may have been ignored or considered insignificant. Tunnel vision — augmented by media hysteria or by clearance-rate and caseload pressures from above — may have overwhelmed investigators and prosecutors.[4] Poorly funded or untrained defense counsel may have failed to investigate alternative explanations or execute effective cross-examination. No single error would have been enough. The errors combined and cascaded — then there was tragedy.

The right answer to the question "Who is responsible for this wrongful conviction?" is almost invariably: "Everyone involved, to one degree or another." Those involved either made a mistake or failed to catch one. And "everyone" includes not only cops, forensic scientists and lawyers directly involved in a case but also legislators, policymakers and appellate judges far away from the scene of the event, as they helped design the system and dictate the conditions under which those directly involved work.

The range of criminal justice sentinel events extends far beyond wrongful convictions. It encompasses "near miss" cases that at earlier points seemed solid and cold cases that stayed cold too long. It includes wrongful releases (because of legal or technical shortfalls) of dangerous or factually guilty criminals. Sentinel events also could encompass failures to prevent intimate partner violence within at-risk families or situations where catastrophic cost constitutes the harmful event, such as incarcerating a non-dangerous geriatric person long past the point that the imprisonment serves any purpose. In fact, anything that stakeholders can agree should not happen again could be considered a sentinel event.

From Blame to Comprehension

In criminal justice, we traditionally take a single-cause approach to error that assumes those responsible are "bad apples." Someone must be to blame for the error, so the impulse is to find and discipline that person: charge him, sue him, fire him or, at the very least, shame him and exhort him to do better. This is what people typically mean when they call for "accountability" in the aftermath, for example, of the exoneration of an innocent person.

But by focusing exclusively on ascribing blame, we drive valuable reports of errors underground and leave latent system weaknesses unaddressed. Practitioners do not want to be blamed, and they do not want to blame colleagues — thus, nothing gets reported.

This can affect agencies as well as individuals. In a blame-oriented environment, it's likely that when a sentinel event cannot be buried completely, the pressure intensifies to keep it in house or to try to shift the blame to someone else's "house." But because no individual house can ever fully explain an organizational accident, weaknesses that might be studied and understood instead remain latent in the system, waiting for the next "perfect storm" of case facts or processes to come along. Searching for a single cause prevents us from understanding how complex systems fail through the confluent, cascading errors — active and passive — of multiple contributors from many houses.[5]

Even in situations where we identify a bad apple — a corrupt or incompetent forensic scientist, for example, or a prosecutor who buries plainly exculpatory evidence — the single-cause approach is incomplete. Standing to the left and the right of the bad apple are the officials and practitioners who hired him, created his work environment and failed to catch his mistakes. We never ask the critical question, "Why did this horrific decision look like the best (or, perhaps, the least bad) decision to the bad apple at the time?"

From Stovepipes to Common Ground

To reduce the chronic risk of organizational accidents, the medical and aviation fields bring all stakeholders together to examine — in a non-blaming manner — sentinel events; they then share the lessons uncovered.

If we decide to view a sentinel event in criminal justice as an organizational accident to which everyone's house contributed, we — like those in medicine and aviation — must consider an all-stakeholders approach. All practitioners and stakeholders should come to the table to analyze known errors and near misses — not by searching for a single cause or blaming a single bad apple, but by appreciating and describing an event's complexity.

As Dr. Lucien Leape, a professor at the Harvard University School of Public Health and a pioneer in the patient safety movement, noted in the medical context, "Efficient, routine, error identification needs to be part of hospital practice, as does routine investigation of all errors that cause injury."[6] The National Transportation Safety Board follows a similar practice in the aftermath of an airplane crash.

But as Leape added, "The emphasis is on routine. Only when error is accepted as an inevitable, although manageable, part of everyday practice will it be possible to shift from a punitive to a creative frame of mind that seeks out and identifies the underlying system failures."

System Repairs

NIJ's sentinel events initiative explores this idea in the criminal justice field: Can — and should — criminal justice develop a commitment to regular, routine, risk-oriented review of known errors and near misses conducted by experienced practitioners and other stakeholders?

To discuss the applicability of such an approach, NIJ convened a roundtable of experts in May 2013. The roundtable, which included nationally recognized experts from law enforcement, prosecution, defense, courts, crime laboratories, the victims' community, risk management and the research community, was a first step in maturing the concept of using a sentinel events review process in the criminal justice system.

The group noted many challenges to a sentinel events approach, including some that are obvious: inertia, unfamiliarity and the adversarial basis of American jurisprudence. Other challenges were more subtle: the balance between incident liability and risk management, the role of internal disciplinary processes, and the current state of confidentiality protections.

Noting that these and other challenges could not be resolved by a one-size-fits-all program, the roundtable participants agreed that each jurisdiction would have its own unique features.

But there was also a conviction among participants that these local challenges were not insurmountable — that "if you want to learn something, try something." The participants returned again and again to the importance of shifting focus from blame for past mistakes to understanding future risk. And most of the roundtable participants came away convinced that there is room for taking a step forward and testing the prospects.

Read the full summary of the roundtable.

An Ambitious Goal by Modest Means

In practice, the world of criminal justice operates as a vague ecosystem — a swamp or a pond, where something (funding, for example) dumped in on one coast has mysterious and unanticipated effects on the far shore. An effort to adopt modern medicine and aviation's experiences to criminal justice will enter this swamp or pond from a different angle.Such an effort can be at once both modest and ambitious: modest in the investment and the degree of federal involvement; ambitious in that it seeks to change a culture to one that routinely concentrates on improving the reliability of the criminal process for the victims, the accused and the public. To accomplish this, NIJ's sentinel events initiative would ensure that the effort be subjected to rigorous scientific evaluation right from the start.

For More Information

• Read the full summary of the roundtable.

About this Article

This article was published in NIJ Journal No. 273, November 2013.