R561
Distance:
Evaluation and Change in the Instructional Development Process
Summer Semester, 2006
Logistical Information
| IST Core Class | Credits: 3 | Section: 10251 | Location: Online |
Click HERE to Jump to Our Schedule
Click HERE to Jump to A Summary of Assignment Due Dates
Instructor
|
Randy A. Knuth , PhD |
Chats are held every Thursday evening at 8 pm Bloomington time starting May 18th. We use MSN Instant Messenger so please get that loaded and ready to go. Please request to add yourself to my buddy list. My screen name is Dr. Knuth and my hotmail address is EvalIST@hotmail.com. You can begin adding to my buddy list on Monday, May 15th. |
| NOTE: Students may expect responses to e-mail inquiries from the instructor about course content and assignments within 24 hours on weekdays, and within 48 hours on weekends. For quick reference, please include “R561D Question” in the subject line of all email inquiries. | |
Overview
Evaluation and change are two integral elements of the instructional and performance technology processes. The course is divided into two segments to reflect these two elements. The first 4 units of the class deal with the evaluation segment, and the last 2 units deal with the change management segment.
Evaluation of education, training, and performance improvement programs deals with making judgments about worth. Education, training, and performance improvement strategists and managers need information about the worth of programs in terms of the degree to which program results achieve intended objectives, whether or not results are desirable, and are they achieved in cost-effective manners. Such information can be gathered and analyzed during education, training, and performance improvement events or after intended interventions are applied. Or, the information can be weighed against personal, organizational, or social indices. You will learn about these contexts of evaluation and develop the knowledge and skills necessary to carry out education, training, and performance improvement program evaluations from a number of perspectives.
Change Management deals with the study of how change occurs in people and organizations. As interventions takes place in education, training, and performance improvement programs, individuals, teams, and their organizations develop the potential to apply new knowledge and practices and improve performance. The actual application of new knowledge and performance improvement requires that change occurs. Often, change is resisted. Educators, trainers, and performance technologists can enhance knowledge application and performance if they devise strategies to reduce resistance to change. You will learn how change management is viewed as integral to the instructional and performance technology processes and develop the skills to apply tools that are used to identify and reduce resistance to change.
Course Objectives:
At the completion of the evaluation unit, course participants will be able to:
1. Understand basic concepts and terminology associated with instructional and performance improvement evaluation.
2. Explain the basic purposes and uses of evaluation within different instructional and performance improvement environments.
3. Explain the role of evaluation within the instructional design and performance technology processes.
4. Use qualitative and quantitative data gathering techniques in evaluation activities.
5. Analyze and interpret evaluation data and information.
6. Report the results of evaluation activities.
At the completion of the change management unit,
course participants will be able to:
7. Identify major steps required to implement change associated with an instructional or performance improvement process.
8. Identify different individuals and groups in an organization that can impact planned change.
9. Explain how different organizations might view change associated with instructional and performance improvement interventions and identify and use strategies for enhancing the successful implementation of the interventions.
Course Requirements:
Participation:
Students are expected to contribute to online discussions of unit practice
questions, as well as join periodically in synchronous chats.
• Answers to practice questions (embedded within each unit) will be
posted and discussed by students in the practice questions section of Oncourse. It is up to individuals to periodically post answers
and offer feedback/discussion to their peers. Instructors will monitor and
offer comments and clarifications as necessary.
• Each week, on Thursday evenings, students
are required to participate in chats at 8pm Bloomington,
Indiana time (Eastern Time). These one hour sessions are meant to provide time for questions and answers with instructors
and fellow students.
Instructors will assign participation grades by reviewing the quality, and—to a lesser extent—the quantity of postings, in the practice question discussions and chats.
Unit Quizzes:
Students will complete quizzes for 4 of the 6 course units. Each quiz will
be worth 25 points and will assess students’ knowledge of unit readings and website content.
The quizzes, which are individual, open book/open note assignments, will be
posted on Oncourse at scheduled times, and students’ answers must be posted
within a designated time frame in order to receive credit.
Unit Exercises:
Students will complete exercises for 4 of the 6 course units. Each exercise
will be worth 50 points, providing students with opportunities to apply skills
learned in each unit. Unit exercises are individual assignments.
Unit Deliverables:
Students will complete unit deliverables for 5 of the 6 course units. Each
deliverable will be worth 100 points, assessing students’ abilities
to synthesize and analyze what they have learned in each unit. Two of the
unit deliverables are individual assignments; three may be done either independently
or in a group of 2 to 3 students.
Percentage Values for Course Activities
| Participation | 20 % |
200 points |
| Unit Quizzes | 10 % |
100 points |
| Unit Exercises | 30 % |
200 points |
| Unit Deliverables | 40 % |
500 points |
| ______ |
________ |
|
| 100 % |
1000 points |
Grading Guidelines
The following definitions of letter grades are a guide to the evaluation
of student performance and an indication to students as to what level of performance
earns a given grade. These definitions have been endorsed by the School of
Education Policy Council and are published in the Bulletin of the School of
Education Graduate Program.
| A | Extraordinarily high achievement--complete command of the course content. |
| A- | Outstanding achievement--nearly complete command of the course content. |
| B+ | Very good work--above average in performance and comprehension. |
| B | Good work--solid and acceptable performance. |
| B- | Fair work--acceptable performance on most, but not all aspects of the course. |
| C+ | Not wholly satisfactory--marginal performance on several aspects of the course. |
| C | Marginal performance--minimal performance and comprehension on most aspects of course. |
| F | Unacceptable work--inadequate performance and comprehension on most aspects of course. |
Required Textbooks:
Shrock, Sharon and Coscarelli, William (2000). Criterion-Referenced Test Development, 2nd ed. Washington, DC: International Society for Performance Improvement. Available at www.ispi.org.
Phillips, J. (1997). Handbook of Training Evaluation and Measurement Methods (Improving Human Performance Series) 3 rd Edition. Butterworth-Heinemann. ISBN: 0884153878
You should already have:
Morrison, Gary R., Kemp, Jerrold E., & Ross, Steven M. (2001) Designing Effective Instruction (4th edition). New York: John Wiley & Sons. [required text – core]
Van TiemVan Tiem, Darlene M., Moseley, James L., Dessinger, Joan Conway (2004). Fundamentals of Performance Technology: A Guide to Improving People, Process, and Performance, 2nd Edition. Washington , DC : International Society for Performance Improvement. Available at www.ispi.org.
Readings Materials Available in Education Electronic Reserves or Linked from the Course Web site:
Available online at:
http://ereserves.indiana.edu/eres/coursepage.aspx?cid=3582
Password: evaluation
Andersen [in electronic reserves]. Luebke, J. F. & Bumpass, S. E. (2000). Managing the information of an executive development program. In J. J. Phillips (Ed.), Performance analysis and consulting (pp. 167-179). Alexandria, VA: ASTD.
Birnbrauer, H. (1996). Improving evaluation forms to produce better course design. Performance & Instruction 35:1, 201-204.
Bragg, D.D. (1995). Working together to evaluate training. Performance & Instruction 34:10, 195-199.
Brethower, D. & Smalley, K. (1998). Integrating Evaluation into Performance-based Instruction. In Performance-Based Instruction. San Francisco: Jossey-Bass. Chapter 12, pp. 156-181.
Brinkerhoff, R. O. & Dressler, D. (2002). Using evaluation to build organizational performance and learning capability: A strategy and a method. Performance Improvement 41:6, 14-21.
Burkman, E. (1987). Factors Affecting Utilization. In R.M. Gagne (ed.) Instructional
Technology: Foundations. Hillsdale, NJ: Lawrence Erlbaum Associates. pp..429-455.
Caudron, S. (1997, February). The human side of a technology launch. Training & Development 52:2, 21-24.
Dormant, D. (1986). The ABCDs of Managing Change. In Introduction to Performance Technology. Washington: National Society for Performance and Instruction. Chapter 17, pp. 238-256.
Duck, J. D. (1998). Managing Change: The Art of Balancing. In Harvard Business Review on Change. Boston: Harvard Business School Publishing, pp. 55-81. (originally published in November/December, 1983 issue of Harvard Business Review).
Garland, K.P. (1995). Diffusion and Adoption of Instructional Technology. In G.J. Anglin (Ed.) Instructional technology: Past, present, and future, 2nd ed. Englewood, CO: Libraries Unlimited, pp. 282-287.
Kaufman, R., Keller, J, and Watkins, R. (1995). What works and what doesn’t: Evaluation beyond Kirkpatrick. Performance & Instruction 35:2, 205-209.
Kifer, E. (1995). Evaluation: A General View. In G.J. Anglin (Ed.) Instructional technology: Past, present, and future, 2nd ed. Englewood, CO: Libraries Unlimited, pp. 384-392.
Kirkpatrick, D. L. (1998). The Four Levels: An Overview. Ch. 3 in Evaluating Training Programs: The Four Levels, 2nd ed. San Francisco: Berrett-Koehler.
Kirkpatrick, D. L. (1998). Evaluating Reaction. Ch. 4 in Evaluating Training Programs: The Four Levels, 2nd ed. San Francisco: Berrett-Koehler.
Korth, S. J. (2001). Consolidating needs assessment and evaluation. Performance Improvement 40:1, January, 38-43.
Kotter, J. P. (1998). Leading Change: Why Transformation Efforts Fail. In Harvard Business Review on Change. Boston: Harvard Business School Publishing, pp. 1-20. (originally published in March/April, 1995 issue of Harvard Business Review)
Lee, S.H. and Pershing, J. A. (1999). Effective reaction evaluation in evaluating training programs: Purposes and dimension classification. Performance Improvement 38:8, September, 32-39.
Leshin, C.B., Pollock, J., and Reigeluth, C.M. (1992). Develop Performance Measures for Each Task. In Instructional Design Strategies and Tactics, pp. 54-72.
Molenda, M. & Pershing, J. A., (in press). The strategic impact model: An integrative approach to performance improvement (PI) and instructional systems design (ISD). TechTrends.
Newby, Tony (1992) Training Evaluation Handbook. San Diego, CA: Pfeiffer. (Chapters 1 - 4). [This book is out of print and not available, unfortunately.]
Pershing, J. A. & Pershing, J. L. (2001). Ineffective reaction evaluation. Human Resource Development Quarterly 12: 1, 73-90.
Pershing, J.L. (2002). Using document analysis in analyzing and evaluating performance. Performance Improvement 41:1, January, 36-42.
Peterson, B. D. & Bothell, T. W. (1999). Measuring the Impact of Learning and Performance: The Franklin Covey Company. In J. J. Phillips & T. K. Hodges (Eds.), Measuring Learning and Performance (pp. 115-134). Alexandria, VA: ASTD.
Pershing, James A. & Lee, Sung Heum. Analyzing Needs for Performance Improvement: Processes and Core Competencies. Draft, 1999. Sections 9, 10, 11, 12, 13, 14, 16, 17, 18, 19, 21, & 22. Available at Collegiate Copies. (these are linked from each lesson as PDFs).
Rogers, E. M. (1995). Diffusion of Innovations, 4th Edition. New York: Free Press. Chapter 1 – pp. 1-37 and Chapter 5 – pp.161-203.
Strebel, P. (1998). Why Do Employees Resist Change? In Harvard Business Review
on Change. Boston: Harvard Business School Publishing, pp. 139-157. (originally
published in May/June, 1996 issue of Harvard Business Review)
Wang, C. (2000). How to grade essay examinations. Performance Improvement
39:1, January, 12-15.
Yeh, S. S. (2001). Tests worth teaching to: Constructing state-mandated tests that emphasize critical thinking. Educational Researcher 30:9, December, 12-17.
Recommended Textbooks:
Mager, Robert F. (1997) Preparing Instructional Objectives. Atlanta, GA: Center for Effective Performance Press. Available at www.ispi.org. Scanned portions of chapters 5, 6, & 7.
Pyrczak, Fred (2003). Making Sense of Statistics: A Conceptual Overview, 3rd ed. Los Angeles: Pyrczak Publishing. Available at www.pyrcazk.com.
Rossi, Peter H., Lipsey, Mark W. & Howard E. Freeman. (2004). Evaluation: A Systematic Approach, 7th Edition. Thousand Oaks: Sage.
Van TiemVan Tiem, Darlene M., Moseley, James L., Dessinger, Joan Conway (2004). Fundamentals of Performance Technology: A Guide to Improving People, Process, and Performance, 2nd Edition. Washington , DC : International Society for Performance Improvement. Available at www.ispi.org.
NOTE: Conducting evaluation in instructional and performance technology requires knowledge and application of statistical concepts and tools. Some students may enter R561 with such skills and knowledge; for those who do not, there is a section on the R561 course website with statistics self-tests and reference information to help students acquire the necessary statistics skills and knowledge.
R561 Distance Course Schedule:
Introduction
Unit 1: Basics of Evaluation
Unit 2: Reaction... Level 1 Evaluation
Unit 3: Learning ... Level 2 Evaluation
Unit 4: Evaluation Synthesis
Unit 5: Change Management I
Unit 6: Change Management II
| Unit | Week of: | Topic | Readings | Assignments |
|---|---|---|---|---|
| May 8 | Course Introduction & Orientation | Syllabus |
||
| 1 | May 15 |
Basics of Evaluation Evaluation: Concepts and definitions Evaluation perspectives Evaluation-driven ID and performance improvement |
Required: Optional: |
Quiz: May 22-24 (M-W) DELIVERABLE EXAMPLES:
|
| 2 |
May 29 |
Reaction ... Level 1 Evaluation Reactionnaires: Purposes and uses Reactionnaires: Developing questions and response categories Level 1 data analysis and data presentation |
Required: Optional: |
Quiz: May 29-31 (M-W) |
| 3 | Jun 12 Jun 19 Jun 26 |
Learning…Level 2 Evaluation Fundamentals of testing Instructional objectives Cognitive item development Practical tests Observations & interviews Level 2 data analysis and data presentation Preparing evaluation reports |
Required: Optional: |
Quiz: Jun 19-21 (M-W) |
| 4 | Jul 3 Jul 10 |
Evaluation Synthesis |
Required: Optional |
Exercise: Jul 12 (W) |
| 5 | Jul 17 Jul 24 |
Change Management I Introduction to Change Management Diffusion Simulation Game: Play & Debrief |
Required: Optional: |
Quiz: Jul 25, due Aug 4 (F)
|
| 6 | Jul 31 |
Change Management II Change Management Tools Change Management Synthesis |
Required: Pershing & Lee, Sec. 19, Sec. 21, & Sec. 22 Strebel Duck Kotter |
Deliverable: Aug 11 (W) |
Summary of Assignment Due Dates
Monday |
Tuesday |
Wednesday |
Thursday |
Friday |
May 8
|
9
|
10
|
11
|
12
|
15
|
16
|
17 Pre-course Survey |
18
|
19
|
22
|
23
|
24 |
25 | 26
|
29 Unit 1 Deliverable |
30
|
31 Unit 2 Quiz |
Jun 1
|
2
|
5
|
6
|
7
|
8
|
9 Unit 2 Del. (Draft) |
12 Unit 2 Exercise |
13
|
14
|
15
|
16 Unit 2 Del. (Peers) |
19
|
20
|
21 Unit 3 Quiz |
22
|
23 Unit 2 Del. (Final) |
26
|
27
|
28
|
29
|
30
|
Jul 3 Unit 3 Exercise |
4
|
5
|
6
|
7
|
10
|
11
|
12 Unit 4 Exercise |
13
|
14
|
17
|
18
|
19 Unit 3 Deliverable |
20
|
21
|
24
|
25
|
26
|
27 Unit 5 Exercise |
28
|
31 Unit 4 Deliverable |
Aug 1
|
2
|
3
|
4 Unit 5 Quiz |
7
|
8
|
9
|
10
|
11 Unit 6 Deliverable |