Fact Sheet: Number 4 (1995)

Evaluating HIV/STD Education Programs

Because of the question of accountability, evaluation of community programs, in general, and HIV/STD education in specific, has become increasingly important. To implement any community or school educational program, financial assistance, educational materials, and human resources, for example, are required. Often, the main sources of support for such programs are government programs, private donors, volunteers, or foundations. These sources are concerned about the effectiveness of educational programs. That is, they are interested in knowing to what extent the objectives of the programs are met. For example:

• Do these programs make a difference?
• Is that difference significant enough to warrant the resources spent?
• Do these programs achieve other unintended objectives?
• What are the weaknesses of the programs?
• How can the program be improved?
• Should the program be terminated?
• Are the programs up-to-date?
• Do the programs serve significant, yet unmet, community needs?
• Are the programs being implemented according to the plan?

The above are basic questions about community educational programs that need to be answered properly. This is what evaluation is all about.

One generally accepted definition of evaluation is as follows: evaluation is a systematic investigation of the worth or merit of an object which can be a program, project, or instructional materials.

Selected key questions about evaluation

Who should be served by an evaluation? An evaluation can and should have more than one client or audience. For example, evaluation of a school HIV education program can serve the students, school personnel, parents, and community. Each of these clients has an interest in the success of this particular school HIV/STD educational program. Thus, evaluation should address various audience concerns.

Who should do the evaluation? Obviously, evaluation of a community HIV education program should be done by an individual or a team of experts who are both familiar with evaluation methodology and the content of the program. The evaluation literature suggests two groups of evaluators who may conduct the evaluation. The first group consists of internal evaluators with the second being external evaluators. While the internal evaluators may be more familiar with the content of the program, the external evaluators may be more objective and have greater experience. Each type of evaluator has its own advantages and disadvantages. Generally speaking, external professional evaluators are preferred for making important decisions based on evaluation results, although greater financial resources are usually needed for their assistance.

What steps and procedures are involved in conducting an evaluation? Depending on the purpose of the evaluation and theoretical framework, different procedures can be employed. For program evaluation, one needs to take the following general steps:

1. State the objective of the program
2. Develop or adopt appropriate measurement devices
3. Adopt an appropriate evaluation design
4. Collect data
5. Analyze and interpret the data
6. Make recommendations based on the data

By what standards should an evaluation be judged? The program evaluation standards recommend the following:

1. Utility standard. This standard deals with the extent to which an evaluation produces and disseminates reports that inform relevant audiences and have a beneficial impact on their work.
2. Feasibility standard. This standard deals with the extent to which resources allow an evaluation to be conducted.
3. Propriety standard. This standard is concerned with the extent to which the evaluation has been conducted in a manner that evidences uncompromising adherence to the highest principles and ideals, including professional ethics, civil law, moral code, and legal issues.
4. Accuracy standard. This standard is concerned with the extent to which an evaluation is truthful or valid in what it says about a program, project, or material.

Back to Top

Why Evaluate HIV Education?
The main objective of all HIV/STD education programs is to help individuals make intelligent decisions with regard to the prevention of HIV/STD. The Division of Adolescent and School Health of the U.S. Centers for Disease Control and Prevention (CDC) provided the following five guidelines for assisting those individuals responsible for evaluating school-based educational programs. Specifically, these guidelines focus on evaluation procedures to help accomplish two purposes: (1) improve HIV education; and (2) determine the success of an HIV education program.

The five general guidelines are:

1. Focus on a manageable number of important program-related decisions. Hence, it is critically important to identify the decision makers' most important program-related decisions. Generally, there are two kinds of decisions. The first one, known as program improvement decisions, is the decision that deals with improving the program. The second decision, known as program-continuation decisions, is concerned with whether or not the program will be continued. It is the task of skillful evaluators to work with decision makers in identifying most important program-related decisions and to focus the evaluation process on those specific important decisions.
2. Select and administer suitable assessment instruments. It is generally recommended that four types of data be collected when evaluating HIV/STD educational programs:
   a. the extent to which individuals engage in HIV/STD risk behaviors;
   b. the extent to which individuals display key skills needed to reduce their chances of being infected;
   c. individual attitudes that are likely to influence their HIV/STD-related behavior; and
   d. extent of individuals' knowledge, or prevention, of HIV/STD. Appropriate instruments related to the above four types of data may be obtained from professional journals, health-related professional organizations, and governmental agencies such as the CDC. Utmost attention must be given to the notion that instruments produce reliable and valid results for the specific population.
3. Use a data-gathering design consistent with the orientation of the evaluation. Depending on the purpose and extent of support for evaluation, different evaluational designs can be utilized. Usually, a simple design like a "pre-test, HIV/STD education program and post-test" design using a representative sample of a target population is used for evaluations concerned with program improvement. For evaluation concerned with program-continuation decisions, a more sophisticated design, such as the pre-test/post-test, control-intervention groups, may be needed. Obviously, this design involves two groups, with only one group initially receiving HIV/STD education and the second group untreated serving as a control group.
4. Use data-analysis procedures that yield understandable results. The data analysis procedures should match the type of data gathered. The best method of data analysis is the simplest one that is understandable and appropriate.
5. Report your results using a multilevel reporting scheme featuring written and oral reports. In order to better serve various audiences interested in the community HIV/STD program, various evaluation reports, including oral, executive summary, and technical reports, should be prepared and disseminated. Recommendations should be specific and based on collected data.

In conclusion, it is essential for HIV/STD educators to document the effectiveness of their programs by using sound evaluation methodology.

Sources of Information

Division of Adolescent and School Health: Centers for Disease Control. Handbook for Evaluating HIV Education. Prepared by IOX Assessment Associates, May, 1992.

Colyle, Susan L., Boruch, Robert F., and Turner, Charles F., eds. Evaluating AIDS Prevention Programs. National Academy Press, Washington, DC, 1989, and the Expanded Edition, 1991.

The Joint Committee on Standards for Educational Evaluation. The Program Evaluation Standard. 2nd edition, Sage Publications, Thousand Oaks, CA, 1994.

Nero, David, "Evaluation: What Is It?" a section from a book entitled, Program Evaluation, by Brinkerhoff, R.O. et al, Kluwer-Nijhoff Publishing, Boston, 1983.

Torabi, Mohammad R. "General Standards for Educational Evaluation." Health Values, 17(4):57-59, July/August, 1993.

Back to Top
Back to FACT SHEETS Index