Science supports corrections agencies and the larger criminal justice system by delivering precise, reliable processes capable of generating consistent, repeatable outcomes.
The National Institute of Justice is tasked by Congress with producing real-world benefits, through scientific innovation, for justice agencies, their stakeholders and the public.
One important way NIJ accomplishes this goal is through program evaluation. Program evaluation methods, themselves the product of rigorous science, have established not all science-based program and practice innovations are equally reliable. With the pace of scientific discovery around the world accelerating as never before, justice agencies seek assurance the particular science underlying an existing or contemplated program or practice is sound, and the program or practice, if properly implemented, can work as intended.[1]
NIJ has an established, evidence-based online resource to help justice agencies find and refine reliable solutions.
CrimeSolutions: A road map to justice system practices and programs
To help agencies know what works, what doesn’t, and what’s promising, NIJ’s CrimeSolutions has established a process for identifying and rating programs and practices that aim to:
- Prevent or reduce crime, delinquency or related problem behaviors.
- Prevent, intervene, or respond to victimization.
- Improve justice systems or processes.
- Assist persons convicted of a crime or at-risk populations of individuals with potential to become involved in the justice system.
Rated programs and practices are included in an online clearinghouse.
The terms “program” and “practice” have particular meanings in CrimeSolutions. A “program” is a specific set of activities, carried out according to guidelines, to achieve a defined purpose. Program profiles on CrimeSolutions tell us whether a specific program was successful in achieving justice-related outcomes when it was carefully evaluated.
A “practice” is a general category of programs, strategies or procedures that share similar characteristics, in terms of the issues they address and how they address them. Practice profiles tell us about the aggregate results from multiple evaluations of similar programs, strategies or procedures.
CrimeSolutions employs a standardized process to evaluate programs and practices in order to determine both (1) the reliability of the science methods used to evaluate effectiveness, and (2) the level of effectiveness, if any, of the program or practice under evaluation.
This article discusses key aspects of the purpose, design, benefits and limitations of the CrimeSolutions resource, providing links to key program web resources.
Of perhaps greatest utility, it explains to the reader where to find and how to take advantage of CrimeSolutions when searching for or studying a program or practice, or deciding whether to implement it.
Overview of CrimeSolutions
The CrimeSolutions clearinghouse identifies justice system programs and practices that have been submitted to rigorous scientific evaluations. The CrimeSolutions evaluation process closely examines programs and practices for scientific evidence of effectiveness. If the evidence is strong and clear enough, a program will be rated “Effective,” “Promising,” or “No Effects.” If the evidence is insufficiently strong and clear, however, CrimeSolutions reviewers will withhold an effectiveness rating. Those unrated programs or practices are deemed to be emerging, inconclusive or unclear.
Table 1 shows, as of May 2021, totals of rated programs and practices and subtotals for those focused-on corrections.
Programs and practices selected for CrimeSolutions review are first identified, screened, reviewed and rated, through a standardized process. See How We Review and Rate Programs form Start to Finish and How We Review and Rate Practices from Start to Finish.
Two certified program reviewers assess each screened program and practice with objective scoring instruments. Reviewers examine scientific evaluations of programs. Broader practices are reviewed by reference to meta-analyses that synthesize existing rigorous evaluations of those practices.
The importance of evidence-based practices and rigorous science
At the core of CrimeSolutions’ effectiveness ratings is the reliance on available, rigorous evidence. Programs and practices are considered evidence-based when their effectiveness has been demonstrated by causal evidence demonstrated in high quality outcome evaluations. The use of scientific methods yields causal evidence by ruling out alternative explanations for observed change.
The more rigorous the scientific method used to evaluate a program or practice, the more certain an effectiveness rating will be. The science must be sufficiently strong and certain before CrimeSolutions will assign an effectiveness rating for a given practice or program.
The so-called gold standard of evaluation methods is the randomized controlled trial, or RCT. In RCTs, participants are randomly divided into treatment and control groups. Carefully controlled and measured division of participants ensures that, to the extent possible, the only difference between the groups is one receives the experimental treatment and the other does not. As an evaluable method, the randomized controlled trial is uniquely capable of isolating and measuring the effect of experimental treatment. Beyond randomization, RCTs must be well-coordinated, with provisions to ensure, for instance, there is no contamination between the treatment and control groups.
In some studies, a rigorous comparison is either not feasible, or not needed to establish the answer to the scientific inquiry. A randomized trial would not be feasible when, for example, denial of the treatment to a control group would be unethical, or the subject of a study is retrospective. And generally, RCTs rely on relatively large sample sizes to demonstrate measurable effect.
The distinct value of CrimeSolutions for different professional audiences
Having objective scientific evaluations of available corrections programs and practices can offer immediate benefits for a variety of stakeholders, including academic researchers engaged in program evaluations, practitioners, policymakers and trainers.
Some of those CrimeSolutions benefits, by audience group, are:
Corrections professionals may be able to improve program or practice effectiveness by one or more of the following actions:
- Familiarizing themselves with rated and evaluated programs in their field.
- Replicating a program or practice found in CrimeSolutions.
- Adapting a program or practice from CrimeSolutions.
Policymakers can shape funding decisions by:
- Creating incentives to use evaluated and rated programs and practices.
- Creating incentives for ongoing innovations and creation of evidence-based programs and practices.
Program trainers can improve training by:
- Developing training materials for evaluated and rated programs.
Researchers can improve justice programming and become more informed on criminal justice research by:
- Consulting CrimeSolutions evidence standards to strengthen evaluation designs.
- For programs rated “Promising” by CrimeSolutions, focusing on using rigorous evaluation designs to build the body of evidence and potentially increase confidence in program effectiveness.
- For programs or practices rated “Inconclusive” by CrimeSolutions, researchers may have an opportunity to build the program evidence base sufficiently for a subsequent CrimeSolutions evaluation to establish, with the requisite degree of scientific certainty, a program or practice is promising or effective or has no effects.
- Improve the available body of knowledge: Where an existing program or practice has been rated “Promising” by CrimeSolutions, there is incentive for independent researchers to conduct additional evaluation work, using a more rigorous scientific design where applicable, such as an RCT, to test or re-test program strength and effectiveness.
Limitations of CrimeSolutions
Anyone interested in tapping the benefits of CrimeSolutions’ ratings of program and practice effectiveness should bear in mind certain practical or possible limitations of the resource. They include:
- The fact a program or practice works in one setting, with a particular population, does not mean it will be equally effective in every setting, or for different populations. Adopters of programs or practices rated Effective or Promising on CrimeSolutions should be prepared to evaluate scientifically the program or practice once applied in their own environment.
- Programs often have mixed results for various outcomes – some positive, some negative. CrimeSolutions assigns ratings to programs based on the preponderance of evidence from up to three studies that have assessed outcomes related to crime, delinquency, or victimization prevention, intervention or response.
- Although CrimeSolutions is a rich resource, with more than 700 posted programs or practices, not all criminal justice, juvenile justice and victim services programs and practices have qualified for CrimeSolutions rating. Agencies should take care to select programming well matched to their needs.
CrimeSolutions’ eight-step process for reviewing and rating programs and practices
CrimeSolutions follows the eight-step evaluation process to determine whether a given program or practice will receive a rating:
- Preliminary Program Identification — programs and practices identified through literature searches and nominations from the field.
- Initial Program Screening — programs and practices reviewed to confirm they fall within the substantive scope of CrimeSolutions.
- Expanded Literature Search — expanded search to all evaluations and research and program materials that may be of interest to reviewers.
- Initial Evidence Screening — identified studies reviewed to determine whether the criteria for “evidence” of effectiveness has been met.
- Selection of Evidence Base — senior researcher selects evaluations or meta-analyses that follow rigorous study designs and methods.
- Expert Review — two certified reviewers perform an evidence review, using a scoring instrument, that assesses the quality and strength of the evidence, as well as the extent to which the evidence indicates the program or practice achieves its goals.
- Study Classification — each study is assigned one of five classifications on the quality of the study.
- Effectiveness Rating — CrimeSolutions assigns evidence ratings addressing program or practice effectiveness. For practices, ratings are assigned by outcome; one practice could receive multiple, differing ratings, based on how effective it is in addressing various outcomes.
Evidence ratings falling into three classes:
- Effective — Strong evidence to indicate the program achieves justice-related outcome(s) when implemented with fidelity to the design.
- Promising — Some evidence to indicate that the program achieves justice-related outcome(s) when implemented with fidelity to the design.
- No Effects — Strong evidence to indicate the program had no effects or had harmful effects.
Not all programs and practices that make it past the initial review receive a rating. Many evaluations are determined by reviewers to be inconclusive, and for those programs or practices no rating is assigned.
Examples of corrections practices rated by CrimeSolutions
The following are two examples of correctional practices rated by CrimeSolutions:
Conclusion
We created CrimeSolutions to help criminal justice, juvenile justice and crime victim service professionals better understand crime and identify program and practice solutions that address the unique needs of their communities.
CrimeSolutions helps justice professionals who are not social scientists improve the effectiveness of programs. The systematic, independent review process and evidence ratings are intended to give practitioners access to social science evidence that is otherwise difficult to obtain, and serve as a basis for gauging the quality of evidence. In short, CrimeSolutions strives to help practitioners answer the questions: has it worked, and in what context?