Multiple Option System (Multi-Op) of Course and Instructor Evaluation

Manual

Acrobat Reader Logo
Download .pdf

Contents


I. INTRODUCTION

Multi-Op is a computer-based system for constructing and analyzing rating instruments for student evaluation of courses and instructors. The Multi-Op system features a catalog of 198 items from which faculty members can select items for rating forms. The items permit instructors to collect student reactions in three major areas: students' intellectual development resulting from the course, instructors' teaching style and methods, and specific course elements such as exams, grading, lab activities, written assignments, etc. Most of the items use a Likert scale on which the student indicates extent of endorsement: Strongly agree, Agree, Undecided, Disagree, or Strongly disagree. There are also a number of open-ended questions asking for written comments from the students. The students write their comments on the reverse side of the evaluation form. Multi-Op users may also write several items of their own and have them included on the student evaluation form along with those chosen from the catalog. Instructors may choose to use either the traditional paper/pencil Multi-Op form or computer-administered Multi-Op.

[Back to top]

II. PRINTED MULTI-OP FORMS

Faculty who wish to use the paper/pencil Multi-Op forms have two options. They may select their own items from the Multi-Op catalog for an individually-designed form, or they may use a preprinted standard Multi-Op form.

Instructor-Designed Form
Instructors who wish to design an evaluation form uniquely suited to their courses and teaching methods may select their statements from the Multi-Op catalog of items. The numbers of the selected items are coded on a MULTI-OP ITEM SELECTION FORM B that is sent to the IU Bloomington Evaluation Services and Testing (BEST). BEST then prints the individually designed form containing the items selected by the instructor. Faculty complete a Form B for each different instrument to be prepared. There are two alternative procedures available for selecting items and preparing the Form B: paper / pencil and online.

Paper / Pencil Form B:

Obtain a MULTI-OP SELECTION FORM B from the BEST office in Franklin Hall M005 or call 855-3357 for a form(s) to be sent by campus mail. Your departmental office may also have a supply of Form B's.

Select up to 35 one-line statements from the Multi-Op catalog of items listed in Appendix A of this booklet. Using a number two pencil, code the catalog numbers of the selected items on the Form B. Faculty who use instructor-designed forms may write up to 15 items of their own and have these appear on the student rating form along with items from the Multi-Op catalog. Instructors who want to write items should code on the Form B the catalog numbers from 171-185 corresponding to the number of items they want to write. The instructor-written statements should be put on a 3.5" DOS-formatted floppy disk and sent to BEST along with the Form B. Instructions on preparing the disk containing the instructor-written items are included elsewhere in this document. Keep in mind that the scannable form permits up to 35 lines of text. If some of the instructor-written items are more than one line in length, the instructor may have to reduce the overall number of items on the evaluation form. The instructor-written items do not have to use the Likert scale (Strongly agree-Strongly disagree) response format, but may use any other five- option response format, such as Excellent-Very Poor, Always-Never, etc.). One is restricted only by the 35-line maximum on the printed evaluation form.

Provide the other information requested on the Form B including name and campus address, course and section number, semester and year, and the number of forms needed. If any of this identifying information is missing, the order cannot be processed. Instructors are also asked to indicate the course level (lower division, upper division, or grad) and the reference group with which they wish to be compared.

Send the completed Form B to the BEST office at least two weeks before the date of intended use. A deadline of two weeks prior to the end of classes is routinely observed each semester. The requested number of printed forms will be delivered to your office.

Online Form B:

The Multi-Op Paper-based Course Evaluation Forms Request System allows faculty to select items from the online Multi-Op catalog. The system eliminates the need for darkening bubbles on the paper/pencil Form B selection sheet as described above.

Standard Multi-Op Form

A standard preprinted form consisting of 30 frequently used Multi-Op items is available to those faculty who do not want to select their own items. For orders under fifty sheets or branch campus orders, faculty can obtain the standard form by calling the BEST office (855-3357) and providing name, address, and the quantity needed. The forms will be sent by campus mail within two days. For IUB orders exceeding fifty sheets, please make arrangements to have the forms picked up in Franklin Hall M005. A Form B is not required at the time of ordering, but one must be submitted as a cover sheet with the completed evaluations when they are submitted to BEST for scanning. The Form B must have name and campus address, the course and section number, and the reference group.

Please note that printed Multi-Op forms cannot be photocopied. The scanner cannot read Xeroxed forms.

[Back to top]

III. HOW TO ADMINISTER MULTI-OP EVALUATION FORMS

Paper Multi-Op Forms

  1. Distribute the printed evaluation forms (Form C's) and read the following directions to the students.

During the next few minutes I would like to collect your anonymous opinions about some important aspects of this course. Your candid responses will be appreciated. Please read the instructions printed on the form and use a #2 pencil to blacken the circle that shows how much you agree with each statement. Marks must be dark and completely fill the circle. The machine used to read these forms does not read ballpoint or felt tip ink. For items requiring written responses, use the box provided on the reverse side of the form. When you have finished, Mr./Ms._______________ will collect the forms. The forms will not be returned to me until all course grades have been submitted to the registrar's office.

After reading these instructions, the instructor should leave the room.

  1. You should have previously selected a student or a departmental staff member to collect the forms, place them in a return envelope addressed to Scanning Technician, BEST, Franklin Hall M005, and put them in a campus mail box; or you may ask the student or staff person to deliver the forms to the BEST office. Tell the person who collects the forms to align them (use the cut corner for aligning) and to place the Form B on top before putting them in the prepared envelope. Please do not staple or paper clip forms or use a rubber band around them. All forms to be processed together for a single class must be submitted in a single envelope.

Important note:

If a student is asked to collect the forms, the student should do so in such a way that he or she cannot identify which classmate completed a particular form. If possible, each individual completing a form should place it in the collection envelope.

  1. Please note that BEST has a policy of not processing groups of forms if they contain fewer than five. Beyond the cost consideration, class groups of this size yield statistics that are very unstable and are more subject to misinterpretation than those from larger groups.
  2. BEST will process evaluation forms immediately following the end of final examinations. The forms are processed in the order in which they are received. As they are completed, the forms and analyses are returned to the mailing address that has been filled in on the Form B.

[Back to top]

IV. INTERPRETING THE MULTI-OP ANALYSIS

The Multi-Op analysis that BEST provides summarizes the student responses to the evaluation items quickly and easily, but interpretation of the responses is likely to be more difficult. The value of student ratings to instructors depends on the care and accuracy of their interpretations.

Demographic Data

The first section of the printout contains course and instructor information, e.g. course, section, level, distribution by class, expected grade, and required vs. elective status.

One should first determine what PROPORTION of the class completed the evaluation. If 75% or more responded, the results probably represent an acceptably reliable sample of opinions. If between 50% and 74% of the class respond, one should be somewhat cautious in their interpretation of the responses. If fewer than 50% respond, the results may be biased and should not be considered a valid sample of student opinion.

Other demographic characteristics of the class may affect the ratings:

Item Summaries

The next thing to examine is the percent of students choosing each response category to each item. That is, look at the number who strongly agree, agree, or disagree with each item. You may want to give special attention to those items on which the frequency of favorable responses falls below your expectations, especially those items representing important objectives of the course. Instructors expect to succeed in areas that are important aspects of the course. If a third of the students in the class give unfavorable ratings to some aspect of the course, you may want to take a hard look at that behavior.

You may also want to look at how the responses are distributed. Do almost all responses cluster around one or two options? Sometimes opinion may be strongly divided, with responses falling at the extreme response categories or dispersed across all categories. Such a pattern might suggest that the instructor was working better with some types of students than others.

Next look at the means for each of the statements. The mean is an average for each item calculated by using weightings of a four (4) for Strongly Agree to a zero (0) for Strongly Disagree. If all students marked SA to an item, the mean would be 4; if all students marked SD, the mean would = 0. Statements with high means indicate relative strengths in teaching; lower means indicate relative weaknesses.

Norms

Information on how other instructors are rated on the Multi-Op items can help an instructor interpret relative strengths and weaknesses. Thus, the next thing to examine on the printout is the normative data for each item. First, the instructor's rating on an item is compared to ratings for all other instructors in the specified reference group, and the result is expressed as a percentile rank (the % GROUP column). The higher the percentile rank, the more favorably the instructor is rated with respect to others in the reference group. For instance, a percentile rank of 80 on an item indicates that 80 percent of the ratings of other instructors in the reference group on that item were equal to or lower than this particular rating.

The next column (% UNIV) in the printout compares an instructor's ratings on each item with the ratings received by all other Indiana University professors who selected the item. The percentile rank is interpreted in the same way as above, except that the reference group is the entire University rather than one of the specific reference groups. Of course, there are no norms reported for the instructor-written items since they are unique to each instructor. The norms are updated with each Multi-Op run.

  1. Category Averages

Items are grouped in the analysis according to the categories used in the Multi-Op catalog. Category averages are given for each of these groupings. For example, if an instructor selects three items from the "Enthusiasm and Intellectual Stimulation" category, the percent of students responding to each option, means, and N's are averaged across those three items. Percentile ranks are not available for all the different item combinations possible; thus, no reference group percentiles are reported for category averages.

  1. Personal Norms

A good standard for instructors to use in interpreting the ratings is their own performance in similar courses in previous years. It is therefore a good idea for instructors to save their Multi-Op evaluations to see what changes may occur from year to year. They may, over time, be able to determine what factors influence their ratings positively or negatively.

Written Responses

Much valuable information is contained in the students' responses to the open-ended questions on the Multi-Op evaluations, but BEST does not analyze these written comments. We suggest that someone within the department take responsibility for typing these responses for each faculty member.

More information is available in Student Ratings of College Teaching: What Research Has to Say by Dr. Lucy C. Jacobs.

[Back to top]

MULTI-OP CATALOG OF ITEMS

  1. GLOBAL ITEMS
    1. Overall, I would rate the quality of this course as outstanding.
    2. Overall, I would rate this instructor as outstanding.
    3. Overall, I would recommend this instructor to others.
  2. INSTRUCTOR CHARACTERISTICS
    1. Organization and Clarity of Presentation
      1. My instructor organized this course well.
      2. My instructor is well prepared for class meetings.
      3. My instructor explains the material clearly.
      4. My instructor answers questions carefully and completely.
      5. My instructor makes difficult material easily understandable.
      6. My instructor illustrates relationships among topics.
      7. My instructor uses teaching methods well suited to the course.
      8. My instructor uses good examples that help clarify the important concepts in the course.
      9. My instructor speaks clearly and is easily understood.
      10. My instructor covers material at a pace I can follow.
      11. My instructor clears up points of confusion for me and other class members.
      12. My instructor summarizes major points in lecture or discussion.
      13. My instructor held my attention.
    2. Enthusiasm and Intellectual Stimulation
      1. My instructor is enthusiastic about teaching this course.
      2. My instructor makes the subject interesting.
      3. My instructor inspires interest in the content of this course.
      4. My instructor emphasizes a conceptual grasp of the material.
      5. My instructor stimulates my thinking.
      6. My instructor motivates me to do my best work.
      7. My instructor asks questions which challenge me to think.
      8. The course stimulated me to seek other courses in this field.
    3. Knowledgeability
      1. My instructor is knowledgeable on course topics.
      2. My instructor suggests references for added reading/ research.
      3. My instructor incorporates current developments in the field.
    4. Rapport and Respect
      1. My instructor treats students with respect.
      2. My instructor shows genuine interest in students.
      3. My instructor developed a good rapport with the class.
      4. My instructor is regularly available for consultation.
      5. My instructor is fair and impartial when dealing with students.
      6. My instructor adapts to student abilities, needs, and interests.
      7. My instructor recognizes when students fail to comprehend.
      8. My instructor listens attentively to what students have to say.
      9. My instructor makes me feel free to ask questions in class.
      10. My instructor accepts opinions different from his/her own.
    5. Laboratory Instructors
      1. My lab instructor clearly explains the procedures to be used.
      2. My lab instructor provides sufficient help in the lab.
      3. My lab instructor relates lab exercises to lectures and readings.
      4. My lab instructor is prepared for lab lectures and discussions.
      5. My lab reports are graded fairly.
      6. My lab instructor promptly returns reports/assignments.
      7. My lab instructor is available during office hours.
      8. My lab instructor provides helpful feedback on lab reports.
    6. Discussion/Discussion Leaders
      1. My instructor manages discussions so that they are helpful to my learning.
      2. The discussion sessions are well organized.
      3. The discussion clarifies lecture material well.
      4. My instructor raises challenging questions for discussion.
      5. My instructor is skillful in developing classroom discussion.
      6. My instructor encourages me to participate in class discussions.
      7. The amount of class time allotted to discussion is adequate.
  3. COURSE ELEMENTS
    1. Objectives and Content Relevance
      1. The objectives of this course are clearly stated.
      2. Announced course objectives agree with what is taught.
      3. I know what is expected of me in this course.
      4. Progression of the course is logical from beginning to end.
      5. Topics covered in the course are well integrated.
    2. Reading and Assignments
      1. Overall, I would rate the textbook/readings as excellent.
      2. I am pleased with the text required for this course.
      3. Course assignments help in learning the subject matter.
      4. Course assignments are interesting and stimulating.
      5. Directions for course assignments are clear and specific.
      6. Complexity and length of course assignments are reasonable.
      7. The amount of reading/homework is reasonable.
      8. The total amount of material covered in the course is reasonable.
      9. Developing the term project was a good learning experience.
      10. Duplicated handouts are valuable supplements to this course.
    3. Instructional Technology
      1. The technology used in this course provides high quality instruction.
      2. Instructional technology is well coordinated with course materials.
      3. The instructor used technology in ways that helped my learning of concepts and principles.
      4. My instructor's use of new technology increased my overall learning in this course.
      5. More uses of instructional technology would enhance learning in this course.
      6. I acquired computer skills in this course that I can apply in other areas.
      7. As a result of this course, I feel more positive about using computers.
    4. Difficulty
      1. The level of difficulty of this course is appropriate for me.
      2. This course is one of the most difficult I've taken.
      3. The amount of work required for credits earned is appropriate.
      4. Standards for student achievement are reasonable.
    5. Evaluation and Feedback
      1. The grading procedures for the course are fair.
      2. The grading system for the course was clearly explained.
      3. Grades are assigned fairly and impartially.
      4. My grades are an accurate indicator of how much I have learned.
      5. My instructor collects enough evidence for valid grading.
      6. The exams cover the most important aspects of the course.
      7. Exams in this course are fair.
      8. Exams require me to do more than recall factual information.
      9. Exams accurately assess what I have learned in this course.
      10. Feedback on exams indicated clearly my standing in the course.
      11. My instructor returns exams promptly enough to benefit me.
      12. My instructor makes helpful comments on my exams or papers.
      13. My papers were graded fairly.
      14. Exams are adequately discussed upon return.
      15. Exams are reasonable in length and difficulty.
    6. Classroom Atmosphere
      1. The instructor exhibited no attitudes or behavior demeaning to minorities.
      2. The instructor exhibited no attitudes or behavior demeaning to women.
      3. The instructor promotes an atmosphere conducive to learning.
    7. Other Course Elements
      1. Guest speakers contribute significantly to this course.
      2. Student presentations contribute significantly to this course.
      3. Field trips offer insights that class materials do not.
  4. STUDENT DEVELOPMENT
    1. Participation and Effort
      1. I have been putting a good deal of effort into this course.
      2. The course requires more time and effort than others at this level.
      3. I was interested in the content of this course before taking it.
      4. I keep up with the studying and work for this course.
      5. I always prepare before coming to class.
      6. I actively participate in class activities and discussions.
      7. I feel that I performed up to my potential in this course.
    2. Knowledge and Skills
      1. I learned a lot in this course.
      2. The course improved my understanding of concepts in this field.
      3. I developed the ability to solve actual problems in this field.
      4. I developed skill in critical thinking in this course.
      5. I learned to communicate more effectively.
    3. Personal and Social Growth
      1. This course increased my interest in the subject matter.
      2. I enjoyed learning about this subject matter.
      3. I learned to value new viewpoints in this course.
      4. This class caused me to reconsider some of my former attitudes.
      5. I developed more confidence in myself in this course.
      6. I learned to work with others through this course.
      7. This course made me more aware of my interests and talents.
      8. I developed some leadership skills because of this course.
      9. I developed awareness of societal problems due to this course.
      10. I was motivated to do more than the minimum requirements in this course.
      11. I would like to take another course from this instructor.
    4. Vocational Development
      1. I developed skills needed by professionals in my field.
      2. The course was worthwhile in terms of my career objectives.
  5. SPECIAL CLASS ORGANIZATIONS
    1. Laboratory
      1. The labs are important to learning in this course.
      2. The amount of lab work required in the course is adequate.
      3. Lab assignments are interesting and stimulating.
      4. Lab assignments are reasonable in length and complexity.
      5. Lab sessions are well organized.
      6. Lab facilities are adequate.
      7. I had adequate time to complete lab exercises.
    2. Team Teaching
      1. Team teaching provided insights a single instructor could not.
      2. Instruction is well coordinated among the team teachers.
    3. Seminars
      1. The seminar approach is effectively used in this course.
      2. This seminar encourages students to learn from one another.
      3. Challenging questions are raised in this seminar.
      4. Discussion in this seminar is stimulating.
    4. Clinic and Clinic Instructor
      1. The amount of clinical experience offered is adequate.
      2. I was exposed to a variety of clinical problems.
      3. Prior course work prepared me to handle clinical tasks.
      4. The clinic time is adequate to perform the required procedures.
      5. I have responsibility for patients commensurate with my abilities.
      6. There is good rapport between my clinical instructor and me.
      7. My instructor offers constructive criticism away from patients.
      8. My instructor demonstrates the clinical techniques I was expected to develop.
      9. The amount of supervision in the clinic is adequate.
      10. My instructor identifies specific problems with my clinical technique.
    5. Field Experience (Practicum, Internship, Student Teaching)
      1. The amount of field experience I had was appropriate.
      2. My field experience was well coordinated with my course work.
      3. I received adequate supervision at the field site.
      4. University and field site personnel work well together.
    6. Performing and Studio Arts
      1. I was exposed to a variety of performance/art techniques.
      2. Performance/art projects are extremely valuable in understanding the course.
      3. Performance/art projects are appropriate to the level of the course.
      4. My instructor's demonstrations of techniques are clear and concise.
      5. My instructor values my creativity and/or originality.
      6. Evaluations of my performance/artistic products are constructive.
      7. The instructor is sensitive to students when giving critiques.
      8. My instructor is able to diagnose technical problems.
      9. I can apply the learning in this class to work in my future profession.
      10. Performances provided me the opportunity to show my learning.
  6. CURRICULUM SPECIFIC
    1. My instructor helped me to learn more about writing well.
    2. My instructor helped me to read more perceptively.
    3. My instructor challenged me to think and write clearly.
    4. My instructor commented usefully on my written work.
    5. My instructor returned written work in a reasonable time.
    6. My reading abilities are stronger as a result of this class.
    7. My writing abilities are stronger as a result of this class.
    8. The course helped me to read with greater appreciation and understanding.
    9. I increased my skill in expressing myself orally.
  7. INSTRUCTOR WRITTEN AND OPEN-ENDED ITEMS
    1. Instructor Written Items
      1. (Instructor provided item 1)
      2. (Instructor provided item 2)
      3. (Instructor provided item 3)
      4. (Instructor provided item 4)
      5. (Instructor provided item 5)
      6. (Instructor provided item 6)
      7. (Instructor provided item 7)
      8. (Instructor provided item 8)
      9. (Instructor provided item 9)
      10. (Instructor provided item 10)
      11. (Instructor provided item 11)
      12. (Instructor provided item 12)
      13. (Instructor provided item 13)
      14. (Instructor provided item 14)
      15. (Instructor provided item 15)
    2. Open-ended Items
      1. What did you like most about the course and/or the instructor?
      2. What did you like least about the course and/or the instructor?
      3. What could the instructor do to improve the course or his/her teaching effectiveness?
      4. What is the most valuable thing you learned in this class?
      5. What changes would you make in the lectures?
      6. What changes would you make in the exams?
      7. What changes would you make in the discussion sections?
      8. What aspects of the course were most valuable?
      9. What aspects of the course were least valuable?
      10. Describe the strongest aspect of the instructor's teaching.
      11. Describe the weakest aspect of the instructor's teaching.
      12. What would you most like to tell the department in order to help improve this course?
      13. Does this evaluation form allow you an accurate critique of the course?

[Back to top]

INSTRUCTOR-WRITTEN ITEMS

Faculty who wish to use instructor-written items (catalogue numbers 171 to 185) can have these items printed on their Multi-Op Evaluation Sheets (Form C's).

To include instructor-written items on printed Form C's, faculty will need to submit a typed copy of their instructor-written items along with the marked Form B to the scanning room. It should be remembered that there is limited space on the Form C, so ideally questions should be no more than 90 characters long (including spaces and punctuation marks). Longer questions may be submitted, but will take up two lines. This will mean that 1) the questions may no longer line up with the response positions; 2) since the Form C has room for only 35 lines of text, the total number of questions may need to be adjusted.

For more help: If you have difficulties getting started or while running the system, please e-mail, call 855-1595, or visit BEST's office: Franklin Hall 014.

[Back to top]