Education | Evaluation Methods and Models
Y535 | 5502 | Kim Metcalf


This course is intended to provide you with an understanding and
awareness of the basic philosophical, procedural, and technical
aspects of evaluation.  The primary goal is to help you achieve a
level of basic skill in the application of acceptable and efficient
models to the evaluation of programs, focusing on those associated
with evaluation of educational programs with applicability to related
fields.

Our approach will be pragmatic rather than ideological, with an
underlying assumption that effective evaluation requires multiple
methodologies.  To this end, we will emphasize the development of a
range of skills across "paradigms", discussing the utility and
efficacious use of several evaluation methods or models.

More specifically, it is my hope that you will find the course helps
you to:

Be aware of and understand the basic concept of evaluation and its
constituent elements, philosophical and methodological perspectives
within and outside the field that can inform evaluation practice, and
the standards that guide the profession.
Learn about, understand, and develop a level of acceptable skill in
using (i.e., selecting, implementing, interpreting) a number of basic
evaluation approaches and techniques.
Experience and become comfortable with the context of evaluation
through direct involvement in or exposure to the "day-to-day" work of
evaluators.

Required Materials

There are five resources from which we will draw substantially and
which you will need to purchase for use throughout the semester.  It
is my hope that, in addition to there contribution to your work in the
course, you will find these materials to be valuable future resources.
They are:

Patton, M.Q.  (1982).  Practical Evaluation.  Newbury Park, CA:  SAGE
Publications.

Pawson, R. and Tilley, N.  (1997).  Realistic Evaluation.  London:
SAGE Publications.

Wholey, J.S., Hatry, H.P., and Newcomer, K.E.  (1994).  Handbook of
Practical Program Evaluation.  San Francisco:  Jossey-Bass.

Worthen, B.R., Sanders, J.R., and Fitzpatrick, J.L.  (1997).  Program
Evaluation:  Alternative Approaches and Practical Guidelines (2nd
ed.).  New York:  Longman Publishers.

In addition, you will need to purchase the packet of supplemental
readings for the course available at T.I.S.

Grading and Expectations

We enter into the course with a mutual assumption that our time is
valuable, and that we must make the most efficient use possible of it,
both in class and outside.  I fully expect that each of you has
enrolled because you are genuinely interested in learning about the
field of evaluation and that working together we can help each of you
be successful in the course.  To help you focus your time and
energies, the following expectations, particularly as they apply to
your course grade, are provided:

Participation:Because we meet only once each week, it is imperative
that you attend each class meeting and that you come to class fully
prepared (i.e., having thoroughly read and considered assigned
readings, having completed assignments or out-of-class projects,
etc.).  This element of your performance will represent 25% of your
grade in the course.

Shadowing:Within the first few course meetings, you will have the
opportunity to meet several of our professional research-level staff
from the Indiana Center for Evaluation.  Each of these individuals is
involved directly in overseeing and managing two or more evaluations
through the Center.  Through mutual agreement, each of you is to
identify one of these individuals and develop a method by which you
can work with them to help you: (a) develop a sense of the work of an
evaluator; (b) have guided experience in one or more of the
fundamental roles or responsibilities of an evaluator (e.g.,
developing proposals, preparing contracts or budgets, negotiating data
collection with clients, developing or implementing data collection
approaches, guiding field-based evaluation staff, preparing evaluation
reports, etc.); and (c) preparing a concrete product that reflects
what you have gained from this experience (details below).  This
experiential element contributes 25% of your grade in the course.

Project #1:Working with one or more of your classmates, you are
expected to prepare a comprehensive literature review of one of the
evaluation orientations to be discussed during class meetings on
January 29 or February 5..  Two products are expected to result from
this process: (1) A clear and logically written paper reporting the
results of the literature review (5-7 double-spaced pages, fully
supported with complete citations, prepared in APA style); and (2) a
brief presentation and interactive discussion of the orientation
(30-45 minutes, with appropriate supportive materials).  The paper
will constitute 10% of your course grade; the presentation/discussion
will represent 5% of your course grade.

Project #2:The major course project can be completed individually,
with one or more of your classmates, or as a direct outgrowth of your
"shadowing" experience.  The expectation of this project is that you
will produce either a complete, reasonable, and effective proposal for
evaluating an existing program or preparing a complete evaluation
report of a project that has been completed.  In either case, the
product is to be no more than 15 pages and include all necessary
components.  More information will be provided about this project as
the semester continues.  The final project will represent 35% of your
course grade.  You will present this product to your classmates on one
of the final two class meetings, with the associated written product
due on that date.

Weekly Course Outline and Schedule
The course is organized into two fundamental areas: (1) theoretical
and conceptual undperpinnings of evaluation; and (2) technical or
practical tools or approaches to conducting evaluation.  Though these
areas are undoubtedly interrelated, for practical and logistical
reasons we focus on them as distinct topics.  The first few weeks of
the course will emphasize the theoretical and conceptual aspects of
our broader topic, approximately the last half of the course will
emphasize technical and practical aspects.

DATE  TOPICS FOR DISCUSSION              APPLICABLE READINGS/MATERIALS

8 Jan    Course Introductions and Overview   1. Syllabus
Course Overview and Purposes        2. Class Roster
Peer Introductions                  3. Information Cards
Course Expectations and Grading     4. From Worthen (Chpts.
Assignments for Orientations Research  4-6)

15 Jan  Martin Luther King, Jr. Day (No Class)

22 Jan  Evaluation Purposes & History
Defining Evaluation                   1. Worthen et al.
Research vs. Evaluation                  (Chpts. 1-3)
Development and History of Evaluation 2. Patton (Chpts.
Current Status of the Profession         1-2)Patton (1996)
3. Scriven (1999)
4. Shadish (1998)

29 Jan  Basic Orientations for Evaluation
Outcomes-Oriented Evaluation          1. Worthen et al.
Brian Horvitz                            (Chpts. 4-6)
Marlene Munn                          2. Additional resources
Joy Stephens                             as identified
Participant-Oriented Evaluation
Darbi Haynes
John Nally
Abby Nardo
Craig Whedon
Paper Due

5 Feb   Basic Orientations for Evaluation
Expertise-Oriented Evaluation          1. Worthen et al.
Paul Binford                              (Chpts. 8-11)
Nolutho Diko                           2. Additional resources
Brian Dodge                               as identified
Ozgul Yilmaz
Adversary-Oriented Evaluation
Mary Hancock
Lauryl Lefebvre
Ozgur Taskin
Win Yee Wang
Management-Oriented Evaluation
Ana Coreia
Charles GrahamEmily Hixon
Paper Due

12 Feb  Introduction to Researchers and Projects
Brief interviews with research staff     1. Handouts as needed
Overviews of current Center projects

19 Feb Alternative Views or Models of Evaluation
Evaluation for Advocacy                   1. Chelimsky (1987)
2. Fetterman (1994)
3. Greene (1997)
4. Scriven (1997)
5. Vanderplaat (1995)

26 Feb Alternative Views or Models of Evaluation
Theory-Based and Realistic Evaluation     1. Weiss (1997)
2. Pawson and Tilley
(entire text)
5 Mar  Tools for Conducting Evaluation
Identifying and Working with Stakeholders 1. Worthen et al
(Chpt. 13)
2. Patton (Chpt. 3)
3. Cousins & Earl
(1992)
4. Patton (1994)
12 Mar Spring Break (No Class)

19 Mar Tools for Conducting Evaluation
Defining Evaluation Purposes              1. Worthen et al.
(Chpt. 14)
2.  Patton (Chpt. 4)
3.  Weiss (1998)

26 Mar Tools for Conducting Evaluation
Developing an Evaluation Methodology      1. Worthen et al.
Managing Evaluation Projects                 (Chpt. 15)
2. Patton (Chpt. 7)
3. Worthen et al.
(Chpts. 16-18)
4. Bell in Wholey
(Chpt. 21)

2 Apr  Tools for Conducting Evaluation
Preparing and Presenting the Evaluation   1. Worthen et al.
Report                                       (Chpt. 19)
2. Hendricks in
Wholey (Chpt. 23)

9 Apr  AERA Conference (No Class)

16 Apr Presentation Preparation (No Class)

23 Apr Presentations
Proposal or Report Due

30 Apr Presentations
Proposal or Report Due