| Adrian German, Hung-Luen Chen |
Myrtle Beach, South Carolina November 1997 |
College teaching and learning are more effective when information regarding performance is provided both to the student who learns and to the instructor who teaches. Tests, quizzes, minute papers, course evaluations, and other methods of acquiring performance-related information are firmly established elements of the academic curriculum. However, technological advances which can enhance the acquisition of such information have typically not been incorporated into college instruction, a field where tradition is strong. This paper reports on the implementation and use of an integrated electronic delivery system used at IU-Bloomington to support the measurement and evaluation of teaching and learning. We describe in detail the features of this system, including its bi-directional feedback character, and illustrate how a small unit can serve the larger campus using client-server and web technology. After summarizing our experience with QuizSite over the past 18 months, we discuss several implications for distance education and identify future paths of work.
Motivation As Light points out in [1] undergraduates are very clear about which of their courses they respect most, where they learn most, and why. Three are the crucial features listed for such a course: immediate and detailed feedback on both written and oral work; high demands and standards placed upon them but with plentiful opportunities to revise and improve their work before it receives a grade, thereby learning from their mistakes in the process; and frequent checkpoints such as quizzes, tests, brief papers or oral exams. The key idea is that most students feel they learn best when they receive frequent evaluation, combined with the opportunity to revise their work and improve it over time.
Feedback about performance is most effective when it is timely and frequent. More than half the students in Light's study felt that they learned the most when given more opportunities to see how they are doing, although 45% admitted they sometimes find such quizzes to be irritating. Nevertheless, two thirds of the students asserted that these exercises have a positive effect on their attitudes toward a course as well as their actual learning.
Although immediate feedback on frequent quizzes is highly desirable, Light points out that "students understand that for faculty members the process of feedback and revision is expensive and time consuming" ([1], p. 32). Clearly, students' and faculty's desire for frequent and timely performance feedback must be better balanced with the demands on classroom, as well as personal, time. The delivery system described in this paper addresses this critical need.
Most of the functionality of QuizSite is summarized online at the IU Bloomington Evaluation Services and Testing web page [5] Any instructor at IU that has a valid network ID can open an account with QuizSite on his own by following the steps outlined in the documentation. Guest accounts, however, require the explicit intervention of the system administrator. In QuizSite instructors maintain a database of items, group them in exams, quizzes, or surveys, and schedule those events for online administration. There are three main kinds of activities that can be delivered through our system: quizzes or practice exams as well as tests; evaluations and surveys; and minute papers.
QuizSite offers a very flexible and completely automated scheduler for exams and quizzes. The selection of items sustains the creation of multiple forms (either randomly or by chosen design) which makes QuizSite a suitable tool for the offering of personalized assignments or exams to students ([7], [9]). The items supported by this version of QuizSite are mostly objective (multiple-choice, matching, numeric, and fill-in the blanks, which can be graded automatically by the system) but essays and short answers can also be collected by QuizSite and graded by instructors on-line. Items are kept in item banks and they can be indexed by topic and by any other criteria on which a suitable taxonomy is based ([3], [4]). Instructors have the freedom to base their item banks on any taxonomy they like.
Instructors can also consult the electronic logs that record student activity to identify and focus on the specific help that one or more students may need with respect to a problem. From a global perspective an analysis of all problems of a certain type over the entire class of students (a feature that QuizSite provides) can also offer valuable insight and numeric data about the relative difficulty of the items (which should be directly proportional to the effectiveness of the course from the point of view that each item is trying to illustrate).
Although it places greater emphasis on individual work, QuizSite does allow the implementation of cooperative learning strategies of the sort described by Bartlett [6].
Instructors value feedback about their course, especially when the course is still in progress. Formative evaluations permit mid-course modifications that can improve course quality. From the students' perspective, feedback opportunities encourage a sense of ownership in the course. As students provide course feedback, they not only influence the course with their opinions, but improve communication between themselves and their instructors. In QuizSite, feedback can be either anonymous or nonanoynmous, according to the faculty's preference. With access set accordingly, QuizSite acts as a third party between students and their instructors to guard the accuracy and the validity of the access policy that has been agreed upon.
The minute paper [10] is another frequently used method that forces students in action and provides continuous feedback throughout the course. Becker, in [2], points out that the idea of monitoring the process of education from the start of the course to its termination, and monitoring students from the day they enter to the day they graduate, fits well with the industrial model of total quality management. Although answers to minute papers may or may not be graded the responses can be used to monitor comprehension and (indirectly) attendance, and focus the discussion in the following class period.
QuizSite has been used by more than 70 instructors in about the same number of courses, with a total enrollment of about 5,700 students. Very little administrative overhead costs are associated with its usage. Most student- and faculty-users are associated with Indiana University Bloomington, although students from other institutions (e.g., Kent State University) have also used the system. Its usage by other universities appears to confirm its utility as a valid distance learning tool. Three out of eight faculty seminars on novel approaches to using technology in the classroom that were presented at the 1997 Teaching and Learning Technologies Summer Conference in Blomington have described instructional projects in which QuizSite had been extensively used.
Internally QuizSite partitions the class workspace into a set of virtual mailboxes in which instructors can place groups of items that have a certain access policy attached. The set of mailboxes is defined by the instructor that initially submits a class roster to the system. Students' mailboxes in a certain course can be easily dropped or added, and the structure of the roster supports the definition of groups for student group projects.
The current version of QuizSite is a CGI/Perl implementation with a custom database backend. Scheduled changes for future releases include the addition of a Sybase DBMS backend; re-coding of student and instructor interfaces in Java; adding an adaptive testing component or multiple evaluation module (as described in [8]).
The development of QuizSite was prompted by the need for a simple, quick, and non-intrusive means of communicating performance feedback between faculty and students, in which faculty can provide rapid feedback and positive reinforcement to the students, who in turn, can provide to their instructors insightful comments that may improve course quality. Through timely provision of feedback information, students are better able to monitor their own learning and correct deficits before it is too late. As one student said, after using QuizSite: "I love these quizzes. There is no better way to study for the exams. If it wasn't for these quizzes I'd be failing the class right now because I am horrible at taking multiple-choice exams. Thank you for having them."
The project was also prompted by the observation that in large classes students can lose a significant amount of their original motivation over time. Reducing the absolute size of classes may be virtually impossible, given institutional constraints. However, the subjective sense of class size can be shrunk by using systems such as QuizSite. Perhaps the most attractive feature of QuizSite is that it enables the instructor to reach more students than would be possible through direct human interaction in a large class. By reducing the perception of class size, students are likely to become more actively engaged in the learning process, an effect that has been observed in classes that have used QuizSite. To quote another student: "Having quizzes due for each chapter prior to the test has pushed me to read nearly all of the text (which is unusual for me..) and I have really enjoyed the reading".
2. William E. Becker, Teaching Economics to Undergraduates, Journal of Economic Literature, 1997
3. Robert J. Marzano, Debra Pickering, and Jay McTighe, Assessing student outcomes: performanse assessment using the dimensions of learning model, Alexandria, Va, Association for Supervision and Curriculum Development, 1993
4. Benjamin S. Bloom, editor, Taxonomy of educational objectives; the classification of educational goals, by a committee of college and university examiners, New York, Longmans, Green, 1956-64
5. http://www.indiana.edu/~best
6. Robin Bartlett, A Flip of the Coin - a Roll of the Die: an answer to the free-rider problem in economic instruction, Journal of Economic Education, Spring 1995, 26(2), p. 131-139
7. http://www.pa.msu.edu/educ/CAPA/
8. Arie Dirkzwager, A Computer Environment to Develop Valid and Realistic Predictions and Self-Assessment with Personal Probabilities, p. 146-166, in Leclercq and Brunoa, editors, Item Banking: Interactive Testing and Self-Assessment, Berlin 1993, Springer-Verlag
9. http://www.uiowa.edu/~itsisdg (outdated now)
10. Thomas Angelo, Patricia Cross: Classroom Asessment Techniques, A Handbook for College Teachers, The Jossey-Bass Higher and Adult Education Series, Jossey-Bass Publishers, 1993