No matter the source of funding for their program — government or private foundation — managers everywhere are feeling greater pressure to demonstrate their programs' effectiveness. And they must do so using scientific methods, not anecdotes about individuals who benefited. Funders want to see data and other hard evidence to justify continuing or expanding a program.
One of the first tasks in gathering evidence about a program's successes and limitations (or failures) is to initiate an evaluation, a systematic assessment of the program's design, activities or outcomes. Evaluations can help funders and program managers make better judgments, improve effectiveness or make programming decisions.[1] Evaluations can describe how a program is operating, show whether it is working as intended, determine whether it has achieved its objectives and identify areas for improvement.
Having a plan for the evaluation is critical, and having it ready when the program launches is best.
Evaluation Plans
An evaluation plan outlines the evaluation's goals and purpose, the research questions, and information to be gathered. Ideally, program staff and an evaluator should develop the plan before the program starts, using a process that involves all relevant program stakeholders.
The benefits of an evaluation plan
Having a plan helps ensure that future evaluations are feasible and instructive. Putting the plan in writing helps ensure that the process is transparent and that all stakeholders agree on the goals of both the program and the evaluation. It serves as a reference when questions arise about priorities, supports requests for program and evaluation funding, and informs new staff. An evaluation plan also can help stakeholders develop a realistic timeline for when the program will (or should) be ready for evaluation.
Creating an evaluation plan
Partners and stakeholders use evaluation plans to clarify a program's purpose, goals and objectives and to describe how program activities are linked to their intended effects. To this end, stakeholders should consider developing a logic model. A logic model visually depicts how a program is expected to work and achieve its goals, specifying the program's inputs, activities, outputs and outcomes.
Figure 1. Sample Logic Model(figure forthcoming)
The evaluation plan should develop goals for future evaluations and questions these evaluations should answer. This information will drive decisions on what data will be needed and how to collect them. For example, stakeholders may be interested in the extent to which the program was implemented as planned. Determining that requires documentation on program design, program implementation, problems encountered, the targeted audience and actual participation. Or, stakeholders might want to know the program's impact on participants and whether it achieved its objectives. In this case, program staff should plan to collect data before implementing the program so an evaluator later can assess any changes attributable to the program.
Types of evaluations
A program can benefit from multiple evaluations over the course of its design and implementation.
The type and timing of evaluations are important. Evaluation is more difficult and less meaningful after the program ends, because stakeholders cannot use information gathered from the evaluation to alter the program's implementation or to justify continued funding. Conducting certain evaluations, like outcome evaluations, is difficult when a program is too new because program elements, strategies or procedures often still are being adjusted and finalized.
| Formative | Process | Impact or Outcome | |
|---|---|---|---|
|
Questions the type of evaluation can answer |
|
|
|
|
When to use the type of evaluation |
During the planning stages or beginning of the program's implementation so revisions can be made before the program starts |
In the early stages of the program's implementation to provide initial feedback |
At the end of the program's development, when the program is stable and unlikely to change in fundamental ways |
Plan for Evaluation from the Start
When designing a program, it is easy to focus only on the immediate decisions that must be made to implement the program and make it operational. But evaluating a program can be challenging or impossible if stakeholders do not plan for evaluation during initial program development. Having evaluation in mind when designing a program can help ensure the success of future evaluations.
Choose the questions you want to answer and know what information you need to answer them
Stakeholders need to know the questions they want an evaluation to answer and build the capacity to collect data to answer those questions. For example, if stakeholders want to know what changes resulted from the program, baseline data should be collected before the program begins. This is especially important if the evaluation will use surveys or interviews to assess baseline opinions or behaviors, because asking respondents later to recall prior opinions or behavior may produce biased results. By thinking this through in advance, stakeholders can ensure they conduct any necessary pre-tests before the program begins and establish a method to collect data over the course of the program. Furthermore, planning for a future outcome evaluation — even if the immediate goal is a process evaluation — can be beneficial because at some point many stakeholders will want or need to answer the question "Does it work?" Partnering with an experienced evaluator can help stakeholders identify potential evaluation designs and decide how to collect the required data.
Determine the timing and resources needed
Stakeholders should consider the time and cost of an evaluation effort and build them into the evaluation plan. A general rule of thumb is to budget 10 percent of the total program cost for evaluation. Although completing a process evaluation may require only a few months, a large-scale outcome evaluation may require years and a substantial financial outlay. If stakeholders want the evaluation's results to help improve the program or justify continued funding, they need to make sure the evaluation is completed before the program is slated to end. This is particularly critical for programs that rely on grant funding, which are usually active only for a set period of time.
Document critical information
To help ensure the evaluation is instructive and meaningful, program staff should document the program's design, purpose and objectives so an evaluator can compare them to the program's actual implementation. Without that documentation, an evaluation is unlikely to produce enough meaningful information to justify its cost and level of effort. Having an evaluation plan in place from the beginning with clear requirements for documentation can help ensure that the needed information is actually collected.
Remain flexible
Despite the best planning, stakeholders cannot anticipate all aspects of a program's operation before implementation, so an evaluation plan should be responsive to program changes and shifting priorities. As they get new information, stakeholders may find some goals unrealistic or some data impossible to collect, access or track. They should revise the evaluation plan as necessary and document each change, justification and decision point.
In turn, stakeholders should be aware that some evaluations, particularly outcome evaluations, might require staff to operate a program differently than usual to rigorously assess the program's effect. For example, evaluators might ask staff to refrain from altering the program's operation during the evaluation period or to select participants in a different manner, perhaps through a randomized process. Partnering with an evaluator in the early stages of program development and implementation can help program staff understand what may be required of them to successfully evaluate the program later.
Special Challenges in Evaluating Multisite Programs
Implementing and evaluating a multisite program can be challenging, especially when sites are given latitude to implement the program in ways that suit their specific needs, because goals and designs will vary by site.
When writing an evaluation plan, stakeholders must consider whether sites will be implementing the program uniformly or will have flexibility in their design. If each site has a different strategy, stakeholders need to take that diversity into consideration and note it in the evaluation plan. Each site should create its own documentation, including a timeline and list of goals and objectives, and sites may require different evaluation strategies. Addressing differences across sites in the evaluation plan and monitoring their progress over time helps ensure each site is fully operational and has the necessary data and functionality for future evaluations.
Evaluability Assessments
Programs without evaluation plans in place can experience significant challenges during evaluations. If a program does not have an evaluation plan, an evaluability assessment can help determine whether the program can be evaluated and whether an evaluation will produce useful results. A program with an evaluation plan also can benefit from an evaluability assessment, which can gauge how well the evaluation plan was put into action and its effectiveness in preparing the program for an evaluation.
An evaluability assessment analyzes a program's goals, state of implementation, data capacity and measurable outcomes. It can save valuable time and money if it shows the program cannot be evaluated because evaluability assessments cost significantly less than actual evaluations. The evaluability assessment also can provide stakeholders with valuable information on how to alter the program structure to support future evaluations.
Design It So It Can Be Evaluated
The key to developing a program that can be evaluated is to have the goal of future evaluation in mind when designing the program's documentation, goals and implementation. Stakeholders also must continually monitor the program's progress and verify that relevant data are being captured, particularly if the goal is to conduct an outcome evaluation. Although evaluation is not always easy and can sometimes be an imposition to program operations, having an evaluation plan is invaluable to making such efforts as feasible and successful as possible. Program staff should, whenever possible, partner with a university, experienced researcher or sister science agency to help construct the plan. Having an evaluation plan in place will help ensure that future program evaluation is feasible and financially viable and that its results are instructive to program staff and stakeholders.
For More Information
Read a chapter by Finn-Aage Esbensen and Kristy N. Matsuda in Changing Course: Preventing Gang Membership (pdf, 12 pages) to learn more about program evaluations and why having a well-designed evaluation is critical to determining a program's effectiveness.
About This Article
This article appeared in NIJ Journal No. 275, posted May 2015.