Program Evaluation and Performance Measurement at the EPA
Performance measurement and program evaluation can both help identify areas of programs that need improvement and determine whether the program is achieving its goals or objectives. They serve different but complimentary functions:
- Performance measurement is an ongoing process that monitors and reports on a program's progress and accomplishments by using pre-selected performance measures.
- Program evaluation, however, uses measurement and analysis to answer specific questions about how well a program is achieving its outcomes and why.
So, performance measurement data describes program achievement, and program evaluation explains why we see those results.
What is Program Evaluation?
Program evaluations are individual systematic studies conducted to assess how well a program is working and why. EPA has used program evaluation to:
- Support new and innovative approaches and emerging practices
- Identify opportunities to improve efficiency and effectiness
- Continuously improve existing programs
- Subsequently, improve human health and the environment
What Types of Program Evaluations are there?
Program Evaluation
Program evaluations can assess the performance of a program at all stages of a program's development. The type of program evaluation conducted aligns with the program's maturity (e.g., developmental, implementation, or completion) and is driven by the purpose for conducting the evaluation and the questions that it seeks to answer. The purpose of the program evaluation determines which type of evaluation is needed.
Design Evaluation
A design evaluation is conducted early in the planning stages or implementation of a program. It helps to define the scope of a program or project and to identify appropriate goals and objectives. Design evaluations can also be used to pre-test ideas and strategies.
Process Evaluation
A process evaluation assesses whether a program or process is implemented as designed or operating as intended and identifies opportunities for improvement. Process evaluations often begin with an analysis of how a program currently operates. Process evaluations may also assess whether program activities and outputs conform to statutory and regulatory requirements, EPA policies, program design or customer expectations.
Outcome Evaluations
Outcome evaluations examine the results of a program (intended or unintended) to determine the reasons why there are differences between the outcomes and the program's stated goals and objectives (e.g., why the number and quality of permits issued exceeded or fell short of the established goal?). Outcome evaluations sometimes examine program processes and activities to better understand how outcomes are achieved and how quality and productivity could be improved.
Impact Evaluation
An impact evaluation is a subset of an outcome evaluation. It assesses the causal links between program activities and outcomes. This is achieved by comparing the observed outcomes with an estimate of what would have happened if the program had not existed (e.g., would the water be swimmable if the program had not been instituted).
Cost-Effectiveness Evaluation
Cost-effectiveness evaluations identify program benefits, outputs or outcomes and compare them with the internal and external costs of the program.
- What is performance measurement?
- How do we determine good measures?
- How is performance measurement different from program evaluation?
What is performance measurement?
Performance measurement is a way to continuously monitor and report a program's progress and accomplishments, using pre-selected performance measures. By establishing program measures, offices can gauge whether their program is meeting their goals and objectives. Performance measures help programs understand "what" level of performance is achieved.
How do we determine good measures?
Measurement is essential to making cost-effective decisions. We strive to meet three key criteria in our measurement work:
- Is it meaningful?
- Measurement should be consistent and comparable to help sustain learning.
- Is it credible?
- Effective measurement should withstand reasonable scrutiny.
- Is it practical?
- Measurement should be scaled to an agency's needs and budgetary constraints.
How is performance measurement different from program evaluation?
A program sets performance measures as a series of goals to meet over time. Measurement data can be used to identify/flag areas of increasing or decreasing performance that may warrant further investigation or evaluation. Program evaluations assess whether the program is meeting those performance measures but also look at why they are or are not meeting them.
For example, imagine you bought a new car that is supposed to get 30 miles per gallon. But say, you notice that you are only getting 20 miles per gallon. That's a performance measurement. You looked at whether your car was performing where it should be. So what do you do next? You would take it to a mechanic. The mechanic's analysis and recommendations would be the program evaluation because the mechanic would diagnose why the car is not performing as well as it should.
You need performance measures to know whether your program (or car) is performing where it should be, and you do a program evaluation (or go to the mechanic) to find out the reason why it is not meeting those expectations.