What is Program Evaluation?
Program evaluation involves the application of various research methods to assess the impact of an initiative, project, or program. For example, a program evaluation of a workplace employee learning initiative might involve pre- and post-test surveys and observations of meetings to determine whether the learning initiative was effective. Or, a program evaluation of a nationwide anti-drug campaign might involve a random survey of individuals across the country and site visits to assess local impact. These are just two examples among hundreds of others.
The goal of program evaluation can be formative--answering the question, "How can we enhance our initiative to reach more people?" Or, the goal can be summative--answering the questions, "Was the program a success, and for whom?"
We specialize in the evaluation of learning initiatives, especially programs that use technology to educate their audiences. Audiences can include anyone--from students in a school setting to adults in a workplace environment. If your program was designed to teach someone something, we can evaluate its impact.
Types of Program Evaluation
Several different types of program evaluation services include:
Process evaluation / Formative evaluation
Process evaluation involves analyzing program inputs, environmental factors, throughputs, and outputs. Process evaluation helps us to understand whether a program is implemented in the intended manner, who the initiative is reaching, and can help explain why a particular program did not have the impact that was expected.
Formative evaluation means that it is conducted early in the life of the program with the goal of continuous improvement. Formative data collected in the early stages of a program can lead to important course corrections that ultimately lead to greater program impacts.
Impact and outcome evaluation / Summative evaluation
Summative evaluation focuses on studying program outcomes and the impact of the program. The goal is to determine whether the program had the desired effect. Did employees learn anything? Did the initiative lead to better healthier patients? And so on.
Research Methods
We use many different research methods to perform program evaluations:
Experimental and quasi-experimental studies
Experimental studies or Randomized Control Trials (RCTs) involve the use of randomization to control for extraneous factors--enabling us to make stronger inferences about the impact of a program.
Quasi-experimental studies are those that do not include randomization but, if well-designed, can still be rigorous enough to make inferences about program impacts.
Field studies / Ethnographic methods
We conduct in situ research in a wide variety of settings, including workplaces, medical centers, retail stores, and schools. CEG has specific expertise in evaluating educational programs in formal school settings as well as informal learning environments.
Focus groups
Focus groups typically include 6-8 participants. Focus groups are useful for gathering data on what appeals to target audience members, what their needs are, and what they like or don't like about certain programs or products.
Interviews
We conduct one-on-one interviews with participants over the phone, during Web conferences, and in-person. Interviews help us to gather in-depth information from participants that cannot be gleaned from surveys or focus groups.
Pre- and post-tests
Pre- and post-tests help us to measure the impact of a program. In a typical research design, we administer pre-tests to a control group and a treatment group. Then, we expose treatment group members only to the program that we are studying. The next step is to administer the post-test to both groups. This design enables us to draw conclusions about changes over time that may be attributable to the program alone.
Surveys
We develop and administer surveys on the Web, in-person, and through the mail. Surveys enable us to gather data from large numbers of participants across a wide range of settings. Using Web-based surveys, sample sizes can easily reach into the thousands.
Usability and accessibility studies
We typically include usability testing and accessibility research in studies that evaluate the impact of a technology-based program or initiative. For example, if we were studying the impact of a hospital's new employee assistance program, we would be sure to evaluate the readability, usability, and accessibility of the program's documentation--including manuals as well as materials that are on the Web--as part of the process evaluation. These methodologies enable us to determine whether the ease-of-use of the program materials is preventing the program from achieving its full potential.