Cost-Benefit Analysis of Criminal Justice Reforms

Policymakers, philanthropists and others interested in what works in reforming criminal justice policy and practice are concerned traditionally with whether new approaches have better outcomes than business as usual. But funders at all levels increasingly see themselves as investors and are concerned not only with outcomes but also with costs and benefits. They ask whether the investment of additional resources is worth the added costs — and whether they will see those benefits down the road in their budgets. A cost-benefit analysis (CBA) can help answer these questions, but its application to criminology can be tricky.

Consider this example: A court sentences a person who is involved with drugs to community-based substance abuse treatment instead of incarcerating him. After several months of fits and starts, he is clean and in recovery. One evening, he is standing outside a subway exit as you leave the platform. Before going into recovery, he may have walked up to you, punched you in the face, and grabbed your phone and purse to support his drug habit. Now, however, he simply goes about his business.

We know how much the treatment program that he went through cost. We can ask him about his work, income and family life and solidly estimate how they have benefited since he went to treatment and not prison. And although we do not know exactly what sentence he might have received, we can use sentencing grids and past court behavior to estimate whether he would have been locked up that evening and what those costs would have been.

Yet we are far less certain of what harm you would have experienced had you been beaten and robbed by the person convicted of a crime outside the subway: Would you have gone to the hospital? Missed work? Suffered from post-traumatic stress disorder? What about the harm your family and neighbors may have experienced because of your victimization?

Criminal justice cost-benefit researchers could argue about which of these costs and benefits even "matter." Should the criminal justice system be concerned about improving wages and income? Should it count the benefits you experienced because you were not victimized, even though they do not show up in a budget? How can a CBA even generate estimates about hypothetical events?

This article explores these and other important considerations when analyzing the costs and benefits of crime interventions. It also examines NIJ's recent Multisite Adult Drug Court Evaluation to demonstrate how CBAs can:

• Include a wide range of potential costs and benefits, including those related to crime, drug use, education, employment, family functioning and mental health.
• Calculate — in dollar amounts — the difference or "net benefit" between drug court participants and a comparison group of people on probation across a variety of outcomes.
• Improve the accuracy of cost and benefit estimates while showing how variable those estimates really are.

The article closes with some general recommendations for improving CBAs of criminal justice reforms.

The Market for Crime

Basic economic theory says that the price of a product or service will be determined by how much demand there is for that product or service and how much manufacturers are willing to supply. The higher the price, the less consumers will demand, but the more manufacturers will be willing to supply.

Although this basic idea works in theory, the market often behaves inefficiently in practice, for a variety of reasons. For example, a monopoly might cause a price to be artificially higher than it would be in a competitive market. CBA was traditionally used to determine whether the benefits of correcting these inefficiencies were worth the cost.

Many government activities exist within a clearly observable market. Public health researchers can study the direct costs and benefits of health care reforms. Transportation planners can predict accurately the effect new tolls have on driver behavior and thus can reliably total the costs and benefits.

There is, however, no market for crime.[1] No one chooses to be victimized. And although people can alter their chances of being victimized by changing routine activities, all victims are unwilling participants in the exchange of the crime "good." Not surprisingly, most cost-benefit work in law and economics focuses on the few areas in which there is something of a more defined "marketplace" — for example, when examining whether changes in sentencing practices and the costs of more imprisonment are offset by crime reduction due to incapacitation and deterrence.

Cost-Benefit Analyses in Criminology

CBAs in criminology are usually part of an impact evaluation, which looks at how a new program affects outcomes for participants. Most applied criminology CBAs count the costs of new interventions, translate participant outcomes into dollars, and compare those costs and benefits to business as usual.

When performing CBAs in criminology, there are three important issues to consider:

Alternative Explanations, or Counterfactuals

Early CBAs in criminology simply counted costs and benefits and compared them to each other, without considering whether there were alternative explanations for the results. Consider our successful treatment client. To put a value on his recovery, we need to know whether he would have been in prison or on the street without treatment. We also need to know how much of his recovery was due to the treatment. What else happened in his life between sentencing and the evening at the subway that might have affected his behavior? This process of developing an appropriate "counterfactual" is critical to generating rigorous CBA results.

Whose Benefits Count?

For most consumers, return on investment is the most critical bottom line. But earlier CBAs of criminal justice reforms make clear one inconvenient truth — most of the benefits of reform will fall to individuals outside of the criminal justice system.

Consider our person convicted of a crime in treatment. In that example, his recovery prevented a criminal victimization. Had he committed the act, he may have been arrested, gone to court and been sentenced to prison. Avoiding those specific events, however, did not yield direct benefits to the criminal justice system. For costs to be recoverable, enough of those convicted of a crime must succeed in treatment to drive crime rates down so low that we need fewer police and corrections workers. In practice, reforms are rarely of this scale.

The question then is whether to include nonrecoverable benefits — such as avoided harm to victims — as benefits. There is a strong scientific basis to do so;[2] however, researchers and advocates should be judicious when discussing nonrecoverable benefits because policymakers and those in charge of budgets tend to be skeptical of so-called "soft" numbers. Nevertheless, including such benefits reflects a more honest account of a reform's effect.

Variable Estimates

Researchers commonly report CBA results as a cost-benefit ratio, which compares average costs to average benefits. However, using a cost-benefit ratio can hide how variable an outcome really is. People often misunderstand ratios to be facts, but they are actually estimates of the average outcome within a broader range of plausible outcomes.

There is also "uncertainty" to consider; this includes those important factors — such as the recovering person's intrinsic motivation to change — that cannot be included in a statistical analysis. To indicate the uncertainty of an estimate, researchers use confidence limits to reflect measurement error and variance or they provide a range in estimates. For example, the results of a telephone survey may be reported as 60 percent in favor of a ballot item with a 3 percent margin of error or as an estimate between 57 percent and 63 percent.

The typical criminological study evaluates enormously complex human behavior. Efforts to quantify an intervention's effect never will yield exact results, and studies that use seemingly precise cost-benefit ratios can be misleading.

NIJ's Multisite Adult Drug Court Evaluation

To examine these issues more closely in an applied CBA, we turn to NIJ's Multisite Adult Drug Court Evaluation (MADCE), an unprecedented study examining the effects of adult treatment drug court programs.

MADCE involved process and impact evaluations as well as a CBA. To date, most drug court evaluations have relied on recidivism as the sole measure of impact. MADCE, however, measured both short- and long-term outcomes — for example, crime, drug use, education, employment, family functioning and mental health — and captured the role of court policies and practices, perceptions of those convicted of a crime, and interim compliance with program requirements.

Researchers at the Urban Institute, the Center for Court Innovation and RTI conducted three waves of interviews with nearly 1,800 drug court participants and a comparison group of individuals on probation from 29 U.S. jurisdictions. Additional data included drug tests, administrative records on treatment and recidivism, court observation, interviews with staff and other stakeholders, and budget and other cost information.

Researchers collected comparable information from program participants and the comparison group. This allowed them to control for characteristics that might offer competing explanations — or counterfactuals — for why the behavior of the drug court participants changed relative to that of the comparison group of those on probation.

What the MADCE Impact Evaluation Found

The impact evaluation found that adult drug courts significantly reduce participants' drug use and criminal offending during and after program participation. Drug court participants reported less drug use (56 percent versus 76 percent) and were less likely to test positive for drug use (29 percent versus 46 percent) than the comparison group of those on probation. Participants also reported less criminal activity (40 percent versus 53 percent) and had fewer rearrests (52 percent versus 62 percent, but not statistically significant difference) than the comparison group of those on probation. Differences in employment, schooling, community service and other outcomes were not statistically significant.

A full description of the MADCE design and results can be found on the NIJ website.

Measuring the Costs and Benefits of Drug Courts

When performing a CBA, researchers can take a top-down or a bottom-up approach to estimating costs. The top-down approach divides the total budget for the service by the number of people served and assigns the same value to each person.[3] In the bottom-up approach, researchers first identify the unit cost for the service (e.g., the cost of a counseling session) and then multiply it by the number of units an individual receives; the result is a person's individual cost. All individual costs are summed to arrive at the total cost. MADCE's CBA used a bottom-up approach, drawing from individual interviews and administrative data.

As for benefits, criminal justice reforms can lead to reductions in criminal offending and improvements in other outcomes. This results in:

• Cost reductions associated with investigating, arresting and supervising persons convicted of a crime (sometimes recoverable)
• Reductions in harm to victims (rarely recoverable)

Table 1 shows the outcomes that MADCE measured. In addition to offending, the evaluation examined social productivity outcomes, which include wages, educational attainment and payment of legal obligations such as child support — all positive outcomes for society.

The last two categories in the table — service use and financial support use — are more ambiguous in that they may lead to greater cost or greater benefits. For example, an effective drug court should lead to less acute care for participants, such as detox. However, an effective court also could mean more use of relapse prevention treatment services, although hopefully at a declining rate over time. Receipt of welfare funds would decline if a program met its goals, but the potential impact of changes in the receipt of disability payments is unclear. Nevertheless, these outcomes represent a real use of resources resulting from a drug court program and thus were included in the CBA.

Adding Up the Costs and Benefits

The MADCE researchers identified all services provided and all outcomes experienced by each person and then converted relevant benefits into dollars. They weighted a recidivism event by the price of crime to victims plus the price of processing the case. Researchers repeated the same process for other outcomes, if data were available.

Rather than directly comparing costs and benefits, the researchers instead summed the costs and benefits for each participant. This included positive outcomes (such as wage increases), negative outcomes (such as lost wages), and all costs associated with treatment and criminal case processing. The resulting measure could have a positive or negative value, depending on the participant's own experience.

This approach has several important advantages. First, because new costs and benefits are hard to predict, this approach allowed researchers to easily calculate the variance for an estimate and show how certain or uncertain that estimate is.[4]

Second, the researchers did not have to determine subjectively what constitutes a cost and a benefit. In traditional studies, events that require new spending are considered costs, and activities that reduce spending are benefits. These designations are arbitrary and often lead to controversy. For example, should prison or jail time for drug court participants be counted as a cost or a benefit? What about post-disposition jail or prison time for the comparison group of those on probation? By summing the costs and benefits together, the MADCE researchers were able to combine all drug court outcomes without subjectively defining them as costs or benefits.

Finally, as mentioned, the researchers opted for a bottom-up approach, using individual rather than aggregated data. Aggregated data — dividing total costs by the number of participants — yield only an average, which is then assumed to be the same for all participants. Individual data, on the other hand, allow researchers to estimate a range of plausible values and describe variations in costs, benefits and outcomes among drug court participants and the comparison group of those on probation.

What the MADCE CBA Found

The MADCE CBA used these calculations to estimate the individual costs and benefits for both the drug court participants and the comparison group of those on probation for each of the outcomes listed in Table 1.

Table 2 shows the net benefits for each outcome category. Although cost-benefit analysts use the term frequently, "net benefit" can be slightly misleading — it is the difference between total costs and total benefits. Thus, net benefits can be positive (i.e., drug court participants used fewer resources overall than the comparison group of those on probation) or negative (where the total end result is negative).

The total net benefit ($5,680) is in the positive direction and is substantial,[5] but there are not savings in every category, and the overall difference is not statistically significant. Looking at specific outcome categories, there are substantial and statistically significant savings in crime and victimization ($11,566). However, these savings are offset by significantly higher service use costs (−$8,135) because drug court participants accessed more drug and alcohol treatment.

The MADCE researchers recognized uncertainty throughout their analysis to help highlight important conclusions that might otherwise be obscure. For example, if they had conducted a top-down analysis of the data, they would have arrived at a statistically significant cost-benefit ratio similar to what has been found in previous research. However, by including uncertainty throughout their bottom-up analysis, the researchers found that the results are not statistically significant.

This latter point is critical. The MADCE researchers estimate that drug courts produce about $1.50 in benefits for every dollar in costs — this is similar with much of the current research literature. However, the researchers do not find that difference to be statistically significant once they account for additional factors, such as the range of victimization costs. The severity of crimes avoided ranges from low-level misdemeanors to violent felonies, and the associated victim costs range from very low to very high. Therefore, the confidence limits surrounding average cost estimates are wide, making it difficult to confirm whether the range for drug court participants differs from the range for comparison group of those on probation, especially for violent felonies, which are relatively rare. Thus, prior studies that produced only a single cost-benefit estimate may overestimate the effects of drug courts.

A careful examination of MADCE's impact evaluation and CBA, however, paints a clear story: Drug courts prevent many petty crimes and a few serious crimes. In fact, the CBA results showed that those few serious crimes drive much of the drug court effect; if we remove those outliers, the benefits of drug courts barely exceed the cost. This finding suggests that although drug courts may reduce recidivism among many types of persons convicted of a crime, drug courts that target persons committing serious crimes with a high need for substance abuse treatment will produce the most effective interventions and a maximum return on investment.

Improving CBAs in Criminology

The CBA performed in the MADCE study demonstrates that criminal justice reforms can have tangible, positive benefits, including fewer crimes and more savings in victimization costs. It also shows that some reforms can lead to additional costs, such as increased drug and alcohol treatment services. However, increased costs that achieve important objectives — such as keeping drug addicts in treatment — should not be used to argue against reform. No one would argue that we should not reduce school truancy because more kids would go to school and require more resources. And some of the positive outcomes shown in the MADCE study — such as improved family life — simply cannot be measured.

Over the last decade, the criminology field has seen a rapid increase in the use of more sophisticated statistical analyses. But when it comes to CBAs, much work remains to be done. More sophisticated CBAs that examine each category of spending and savings could yield important information about a program's success or failure. For instance, significant savings in public safety costs may require significant investment in treatment costs, but potential benefits may be missed if you look only at overall estimates.

An intervention can directly benefit persons convicted of a crime in dramatic ways. It might prevent or ameliorate health problems, save lives by preventing overdoses, or reduce criminal behavior and community supervision violations that affect income and families. To really use CBA to improve public policy, stakeholders should consider expanding the range of included benefits. Broad measures of a program's impact — including on victims and clients — provide much more useful information than studies focused solely on returns on investment.

NIJ Journal No. 272, posted May 2013
NCJ 241929

For More Information

• Read more about the MADCE study.
• Learn about the basics of performance measures and program evaluation.