In evaluating the success of the course, the instructors tried to assess both what the students learned (in terms of both skills and content) and the learning environment of the classroom. The latter is important because
Learning applied ethics requires students to take intellectual risks; it requires students to give close examination to their beliefs, values and methods of thinking about adequate and inadequate professional behavior. Clearly, some environments encourage this kind of risk taking and other environments discourage it. [352-353]The team's attempts to measure the learning environment included development of a fairly elaborate instrument, which was ultimately deemed unsuccessful; they concluded that the use of a standard student evaluation is adequate for this purpose.
As Elliott and Stern point out, a prerequisite for evaluating the success of a course is a clear understanding of the instructional objectives of the course, and their goals are particularly well-stated:
1) be able to clearly describe relevant scientific conventions including: laboratory practice, institutional responsibility, etc.;To measure the success of the course, the team of instructors used a pre- and post-test method. Extensive efforts were taken to assure the validity and reliability of the evaluation, including the use of three outside scorers and a fourth outsider who compiled and analyzed the results. The test required students to read an edited article on an actual case of misconduct of science taken from the journal Science (included as an appendix to the article) and answer the following question:2) be able to describe what leads to ethical problems including causes inherent in the social context of the practice of science;
3) be able to identify ideal scientific practice and consider how to bring scientific conventions more in line with the ideal;
4) be able to separate behaviors into four categories: morally prohibited, required, permitted, and encouraged, thus illustrating an understanding of the role of the scientist in society. [349]
Identify the ethics problems in this case. Discuss what the individuals involved did right. Discuss what the individuals involved could have or should have done differently. [350]The results were disappointing. There was no inter-rater reliability, and "there was no significant difference between how students approached the vignette at the beginning of the term and how they approached it at the end." The authors identify several reasons for this failure.
We realized that even if students had learned the material they had not been encouraged to express what they had learned [because] the post-tests were not graded. . . . Few students made any attempt to integrate the three questions and to evaluate the responsibilities of individual moral agents as complex people. . . . We did not provide the students with any explicit instruction in conducting systematic moral analysis. We instead expected students to intuit the process by examining a series of cases. [350-351]Given these deficiencies, it should not have been a surprise that the pre-test/post-test model did not work. To my eye, the last reason given is the most telling. The students might have been able to do good moral analysis by the end of the course, but if they were not given any practice or instruction in how to do an explicit, written moral analysis, it is not surprising that they failed to rise to the occasion when they were asked to do so at the end of the course. I can imagine that the students may have been able to do the analysis, but were simply unprepared to express it in the format required. I think it is likely that the fault was not with the course, but with the design of the evaluation.
After this initial failure, the instructors tried a different method for assessing student learning. Instead of replicating the pre-test exactly, they asked their students at the end of the course to do a meta-analysis of their pre-test.
(1) We provided [three] short vignettes (Appendix C). Two contained issues of ethical importance (from the faculty's perspective) and one did not.Unfortunately, the authors do not include item 2 (the more explicit instructions for the pre-test) with this article.(2) More explicit instructions were given for the pre-test and students were told that a Ôhigh quality' response to the pre-test/post-test was necessary to receive a ÔPass' in the class.
(3) At the time of the post-test, students received back their pre-tests, with instructions to analyze how well they had responded to the pre-test (Appendix D). [351-352]
The authors describe the new pre/post-test method as a success, but say, "As we did not involve external evaluators, the report of the results are anecdotal but compelling."
This article, the only one of its kind that I know of, will be very valuable to anyone designing a course of study in the responsible conduct of science. I also recommend the two books published as a result of the same project:
Last updated: 06 May 1997
URL: http://www.indiana.edu/~poynter/tre4-2b.html
Comments: pimple@indiana.edu
Copyright 1997, The Trustees of Indiana University