Basic Program Evaluation

Intrinsic Value

We need to examine and celebrate the work that we do. Evaluation is a systematic way to identify the strengths of our programs, and share substantive information with others. Evaluation has many intrinsic benefits, allowing internal staff to examine their programs and results.   If you have a sense of the intrinsic value of evaluation, you and your staff will be motivated to conduct an evaluation. Ask yourselves these basic questions – what do you want to know about your own organization, what do you want to be able to show others?

Evaluation has numerous extrinsic benefits: sharing information with partners, stakeholders, funders, and the general public. Evaluation helps us confirm the success of our program, and allows us to review the quality of our programs and services. Program evaluation can help us identify strengths and weaknesses in the effectiveness of a program, identify areas that need improvement, share findings, and make decisions for future programs. Evaluation helps you show your program intervention has a positive impact in the lives of your clients.

Familiar Steps

We already do many activities that fit directly into the process of Evaluation. The process of evaluation overlaps with processes such as: annual planning, periodic reports, client record keeping, and setting strategic goals. In annual planning, for example, we set specific goals, and review them at a later point in time. That process is a form of Evaluation. Program Evaluation will feel approachable once you realize that you are already doing many processes that overlap with a systematic Evaluation of your program.

Dr. Cohen’s webinar Performance Measurement – Using Data to Improve Quality notes that Program Evaluation often overlaps with strategic planning.  To conduct an evaluation, begin by deciding one item that you want to study. Dr. Cohen provides many areas of activities common to most organizations:  

• Community and professional training
• Intervention processes
• Provision and receipt of services
• Outcomes monitoring

Within each of these areas, Cohen provides numerous example of constructs that you could measure. By reviewing the categories and examples he lists, you may be able to identify an area you want to examine for your own Center. For example, within “provision of services”, you can examine the percentage of clients receiving employment counseling within the first month after intake. Within “outcomes monitoring” you could examine the number of clients reporting improvement in housing stability, or reduction of somatic symptoms.

Start Small

In your planning phase, begin by choosing only one measure that you want to examine. A small-scale, manageable process will be more approachable and will fit into your daily routine more easily. For example, assessing client satisfaction may require adding one additional form to your case files. This questionnaire could be filled out by clients at a specific interval, such as six months after intake.

In his webinar, Demonstrating Client Improvement to Yourself and Others: Setting up an Evaluation System to Succeed, Greg Vinson notes the need to narrow your definition of a goal (objective or indicator) that you can use to measure an outcome. You need a narrow focus and systematic approach. Once you choose a measure, collect “just enough” data to examine your item. Focus on the data relevant to what you want to measure.  

At the planning stage, if you have clients with extraordinary needs or circumstances, you need to narrow your focus to measures which apply to the majority of your clients. In addition, choose your measurement tool carefully. Choosing a well-defined measure is critical to a successful evaluation. A poorly conducted measure can give the appearance that your program is not working, even when it actually functions very well.

Evaluating a Process

Process evaluation is a slightly different approach, examining the operation of a program, more specifically, the process by which it achieves its results, rather than the results themselves. Examining a process can reveal how implementation of the program differs from the original goals, or how results are delivered in an inefficient way. Program evaluation can help you show your colleagues how to replicate your program. For funders, general public, you can show the method by which you operate. Joan Othieno reviews process evaluation in her webinar Outcome Based Evaluation: Planning, Process, & Implementation.

Review Examples

Looking at samples in peer reviewed articles, helps us see how others have applied the concepts. The Evaluation publications list may help you identify articles relative to your own interests.

If you start small and examine one item, the results may lead you to make changes in your program or to study another item. You will find that evaluation will become a cyclical process and you can build on each successive iteration.