Trudy Banta- Discussion
Nancy White: I have a question about student evaluations, and I'm sure we all have our horror stories about them and the things that, the different vehicles, or whatever the term is, instruments or questionnaires that universities use. The one are asked to do by our administration has a, I don't know how many questions, and I forgot whether you get up to five or seven. But, one of the questions is, "Exams and assignments are known from the beginning of the class." And every syllabus that I've written in my life has every assignment, the due date, the date of the final exam–all this stuff. And never, never once have I gotten, you know, the highest number on that question, which leads me to . . . [laughter]
TB: I can't answer this question . . .
NW: No, no, well here's the question that I have for you, which, I don't know, maybe it's naïve, but is there anyway to evaluate the evaluations of students. In other words, their reliability of them. I mean we all know the basics that if you're the least bit challenging you get less favorable evaluations and stuff like that, but is there any way to . . . and I know there is now some way to ask what grade is expected in here , which would be a step forward from where I come from . . . But, what other . . . have you–you guys are professionals, how have you managed to deal with that issue?
TB: Well is there any way, do you have any opportunity to ask the students after that has happened are there any opportunities for you to. . .
NW: No, it's usually at the end of the class . . . but I see what you're saying, if it were done earlier maybe they wouldn't . . .
TB: Well, if these students are in your school, still in your program, is there an opportunity for them to get together for a club meeting, or in the next class, or something. Someway to get a little group of them together, or ask them as individuals, as you see them, "Ah, tell me about this item?!" There's no better way to get information than to ask students. So I would just take a random sample and say, "Tell me how you saw this." Don't ask them how they rated you but just say, "What were you thinking about when you were answered this question." [an uproar of commentary, laughter]
Rosemary Joyce: Basically along the same lines, this may me a terrible thing, but before I give the things out, I stand up in front of the class and go over the things and explain to them how I'm going to read the results that if they really want to influence my making up of the course, that the kinds of things that I want to hear from them are what will be useful and what won't be useful. That may be biasing the process.
Trudy Banta: And are you in the room when they do . . .
RJ: No, then you hand them out and you get out of the room, but the thing is that I decided that a long time ago these measure, because of these certain things, I get ratings on things I don't do. Everybody does, they answer every question! So I started this originally just to go over the things and say, "This really doesn't apply in this class. You don't have to do something." But then I found that it helped me to go over it and say, "This question says this, now it's not helpful to me if you just say this." In fact, you need to go to the back of the page and say, "Here, these are the answers where you get to write your little answers. It doesn't help me if you say, ‘Great course!" And I use the positive scenario, not the negative scenario. [laughter] Because, I don't want to admit that any of them would say anything negative. [I say] "Just don't tell me it's a great course, tell me what worked for you." And then actually, I did originally because I'm terrified. Now, I have to add that I don't read my reviews. I no longer look at my evaluations because they are just too–as data quality–that was the question. What's the quality of that data? First of all, do students really know, at that point, what they've learned. Secondly, do these forms do this–I've had students actually to write me and tell me, after the fact, what they liked about the course, what they felt was useful. And I get a lot of responses that way, I know I don't get the negative responses, but I get the people, who I'd listen to anyway. [laughter]
TB: I think that what you're trying to do is motivate the students to take this seriously, your using also a point, a built-in point of contact. But, quite honestly, all that I have said is really not on that final evaluation. I'm talking about taking the stuff students turn in every day and taking a second look at that across students–forget the names and seeing how that fulfills your objectives. That's one thing, the indirect measures I'm talking about are going to happen during the course. They're going to be a two or minute thing that you give at the end . . . the muddiest coins, you know that one.
RJ: Just the 3-minute question? [TB: Yes] Maybe people don't know about this sort of thing.
TB: Well, at the end–just before the class ends–give them maybe 5 minutes, And just say, "Okay, what did you learn today. What was the most important thing, what really stands out for you, and what is still not clear. . . . " And, you could also says, "How was the assignment. Did this assignment give you the background you needed for today?" You pick up, even a hundred of those, you can read them almost as fast as you can put them down. And, then you know–what they got, what stood out, and what they didn't get. And you may even know why, because they may have told you. And so that is just one way of a thousand different ways that you can ask students questions about how they are learning, and how they've experienced a particular assignment. Just think of what is it that you want them to know, and just ask them. And it need not even be written, you can just say, "Well, you know, what's still not clear."
RJ: Although the written part helps [TB: yes] two ways. First of all, the students who don't want to talk. Secondly, the students who really have an issue, especially if they write and don't have to sign it. They will tell you.
TB: Yes, it is much better to do it that way.
Anne Pyburn: But I also like to do it where they sign it because when I have a big class, I use it for attendance. And, you don't even have read them all, just file them behind their name. My AI's keep a file, and there are punitive results for not attending class so at the end of the semester you just count how many you got for each person, divide how much they get off their final grade by how many are missing. We also keep it on the Web so they can see whether or not they got credit for attending so if somebody's cards are lost, they can get in touch [with us] and say, "I was there, where's my card!" And I don't fight with them about that, and I'm not a policeman. If they've got their roommate going to class to fill out their cards well, okay. But I've got 140, and I find that they–even if you don't say, "Did you like my lecture, or did you get the assignment." If I ask them something else, they will take the opportunity to write on the back, "What were you talking about?" Or "that wasn't clear." And,you know, they're incredibly blunt. I just love reading them, and it's a lot of people who would never say anything in class, who will say something really funny or wonderful on the cards.
NW: And they do sign it?
AP: Oh yeah, they put their name and date on the card because we use them for attendance, but somehow the cards–I guess they're like e-mail, they feel . . . they'll tell you stuff.
Liz not IUPUI: Do you do that every period? [Ann: I do it every class] Or do you vary the question.
AP: I vary them. What they do is at the beginning of class they buy 3 x 5 cards as soon as possible. They have the 3 x 5 cards every day, and every day when get–and this is the other thing I find is that I, you know I'm an old hippie. I don't like policing people, if you hate to come to class and don't want to come, fine. But, what I found out is that they want credit for coming to class, it makes them feel good. This way they get credit! So if I forget to ask the question, they remind me.
NW: But that's so much for you to have to keep all this–
AP: No, it takes. I give them 3 minutes to write the question, so it doesn't take any lecture time, and then–I actually, for this huge of a class I have teaching assistants. I make them file them. I mean I have 140 students in a course; they [the teaching assistants] look them over and say, "You know, 20 out of 30 of the ones I got were worried about that concept with law. You should probably talk about that some more."
RJ: I did the classic 3-note question at Harvard with my 270-student class, and that one, there, I actually went through all 270 cards, could do it in, between one of the lectures of the next one. Because you don't have to read all of them, because, in fact, what happens is they're repetitive, especially if you ask the classic 3-note question: what was the one point that really interested you, what point is still unclear. Ideally, what you get is repetition and then you actually are factoring out the noise level of the person who was asleep because of last night, and you actually get a diagnostic tool, and I could actually then change my next lecture. And sometimes dramatically and come in and say, "Okay, big issue. You've all completely misunderstood this, so we're going back over it."
TB: Yeah, and they're a couple of other ways to do that. One is to have a web-site where people can give you the questions that they have later after class or before the next one. And another way is to ask three or four students to form a team, depending on how large the class is. And then they can gather information and learn social science skills at the same time by interviewing their classmates, and then letting you know how things are going. I just want to call your attention to this really wonderful reference, this whole thing of asking questions in order to get formative information in a quick way, it's called ‘Classroom Assessment' and Tom Angelo and Pat Cross, Pat was at Harvard and then at Berkeley, Tom got with her at Harvard. So, it's a book, there's a matter of fact 2 or 3 out, and so they're just full of these little things. It takes a few minutes to read about them and a few minutes to give them.
Glen Doran: I have a comment, something that both Nancy and Ruth said made me want to speak, and I'm very shy most of the time. [laughter] Nancy implied there was some option in terms of these students and class assessments. . . They are mandated; we're required. And, for twenty years I have heard faculty complain about them, and only for about the last three years have I been the one responsible for actually see them as chair of the department. And, by in large, there's a pretty tight match between student anecdotal remarks about faculty members, and their pseudo-science scores. I've had people coming in crying, "Can I drop this class? This is the worst class I've ever been in. We're not learning anything!" And I'm representing five people, or ten people, and if you add those up . . . and match them to the course–real high correlation.
NW: Well there's got to be at least some course correlation, but that's not the issue. The issue is that . . .
GD: But you can also add your own questions, and most of us don't bother to do that. You can have a handout, you can put it in there and say these are the five questions or however many you want to ask at the end of the day.
NW: And of course, what they write on the back, for the written ones instead of what the machine grades, but when the evaluators evaluate what do they zero in on? The numbers, "Oh, this ones a 3.6 this ones a 2–you know, because they don't have time to reeaad hundreds of written ‘this is the best class I've ever taken.'"
GD: Well, I think we do. We should take the time to read them; we should look at them whether we want to or not.
AP: Here's another thing, where I'm probably also terrible, that I cope with for meaningless questions. I learned early on that students do write down whatever you tell them. I once made the mistake of telling a class that–I said of myself, "I'm just loopy." And they wrote that I was loopy on . . . [laughter]. But what if I tell them that I care about them . . . [laughter and discussion on whether they actually write this down] Oh they do though! Oh yes they do! If you have a 140 students, how do they know that you care about them? I mean you try to show them with all things that you do, but they don't perceive all the work that goes into a . . . so if you mention from time to time, "Now I'm giving you an extra 10 points because I really care about you." They will write that on their evaluation! [laughter]
TB: Well, let me just remind you at this point that I really have not been talking about the end of the course evaluations. I think that's almost too late [agreement from others] for the kind of corrective action that we're talking about. So, yes, we all have issues with those in the course evaluations, and I certainly agree that we ought to be asking our own questions, especially as you design new courses, and new segments in courses. You want to know right away how it worked, and at the end you want to know how it worked. So we've got to help you figure how to ask the questions that will give the formative feedback you need to take those little corrective steps along the way–to keep you going toward the goal that you have. And, writing out that goal to begin with and heading for it with the questions, that's what we want to do.
GD: The other comment was that–I teach the human osteology class, and it usually sends people into the hall screaming by the mid-semester . . . it's usually considered one of the real killers. And there are a couple of other faculty members who have real killer classes that are notorious, and they do quite well on the [evaluations], those are always okay. So challenging them–this is what Lion was saying yesterday–you know, if you challenge them, they seem to both rise to it, and they don't resent it. If its' truly challenging for learning, not truly challenging for busy work or something. I got a 35% drop-out rate after the first osteology lecture, I consider it a success. I tell them, "You cannot be here and take 15 hours, or 17 hours. If you don't have somewhere between 15 and 20 hours a week to devote to this class, it will be extremely unpleasant because there is no way that you can cram for this information. There's no way that you can sit at home in your room, you're going to have to physically be here." It's like the practical exams for the invertebrates, you know, you can't over [?] to your head to learn it.
Dean Snow: I was just thinking that this kind of rational, well-thought through, planned-out program of assessment is a heck of a lot of work to put together, and least as compared with, you know, recycling multiple choice questions. How do you get faculty to do this? You know at a research university at least, I think probably most institutions of higher education, faculty are little entrepreneurs, you know, they're little independent empires, with a great deal of time devoted to raising money and doing research and publishing, not to mention committee-work and professional obligations and so-forth. So instituting something like this across a faculty at least in the fields I'm familiar with is that top-down approaches really don't work very well, your department chair basically is the go-between between the dean and the faculty . . .[laughter].
TB: Well, this little example, it seems to me is, could be convincing. How much time is it going to take you to do a checkmark, to grade the paper and just make a checkmark---
DS: The time is in putting that thing together for, you know, how many things you're going to do during the course.
TB: Well, but then, you weigh that against deciding on your own that they really need to develop outline skills and spending time doing that, instead of doing what they really need, which is the oral defense.
DS: Or the option is giving them A, B, C, or D on the final paper, that's the alternative . . .
TB: Well, are faculty interested in improving what . . . or in helping students learn?
GD: Don't ask that [laughter]
Vin Steponaitis: You know, it seems to me, if I understood Trudy correctly, this can be done at two levels, and I think this is what, if I understand what your asking. You can do this, an individual faculty member can do this for his or her own class, or a faculty can do this as a kind of programmatic assessment. And, what I heard you ask is, how do you get your faculty to do the latter . . . . to get the faculty to evaluate their own courses.
DS: I could see motivating individual faculty to do this, but it's negotiating this as a departmental policy---
TB: Well, it seems to me that, first of all, that rich conversation informed by data can only take place if we have the data, and so, I think that if enough people found that this was helpful to them and in the long run saved them time, and then brought that information to the table when the faculty meet to talk about the curriculum--it's just like the mathematicians at Virginia Tech, somebody told them that it would be a good idea to get some information from seniors about the program, it was actually their state, they were very top-down about telling faculty they need to evaluate. So, they said, well what can we do? Well maybe an exit interview, and so they did. And lo and behold, it just hit them between the eyes! Well they're believers in it now because they learned something from these exit interviews that they didn't know, and it made them take another look at their curriculum.
NW: I would love to hear from the students here, who've been listening to all this and ways to make classes better. This may not be the right time, but that's one of the reasons for being here. . . . Just think about, maybe think about. [Ann clarifies that the undergraduate present was asked to observe, but not asked to make a presentation]
----
Joe Watkins: Yeah, I just wanted to share an experience that I had in teaching a class for archaeology majors who were seniors, mostly seniors and some juniors. And we required a research paper, and when it came back, I edited them very carefully and saw many deficiencies in a lot of the basic kinds of things about style, organization, formatting, bibliographical referencing. And I marked off heavily and gave them back to them. I had several seniors come in and say, after the initial shock, "Thank you so much for this, you know, we never had this kind of feedback." And I said, "Your seniors, haven't you written papers in all you're other classes?" And they said, "Yeah, but the teachers made comments like this." [I said] "So what did they do?"–they wrote a sentence or two at the bottom and gave a grade and that was it, and it gets to the point of following these outcomes through all the way. I mean, what a shame that these are seniors about to graduate, and nobody had told them about basic issues of how to write a paper–
GD: And they're expressing their appreciation for something that you thought was . . .
JW: Yeah, and that taught me something, and it also comes back to the point that Bill made that about, this requires time. To get 20 papers in, you can do two things: you can go through it that carefully, which will take a lot of time, which will be helpful, or you can do the quick summary at the end and let them come to you if they have a question. And a lot of people make that 2nd choice, and this is something we have to deal with in all the things we're talking about, everything here takes time, and then–
DS: The problem is that guys like me aren't evaluating you on that, we're evaluating you on research and other things that have nothing to do with your teaching methods.
[More discussion, too many voices]
Another Response: There has to be a shift in the value-system at the university at the level of the chair and the college, to value and reward good teaching, not just---
TB: . . . In your hand-out the one where I took the terms (SEE) acceptable, satisfactory, unacceptable and defined them, now defining them, I admit, takes time, but then if you put an O next to a paragraph, or an A, or your scoring the bibliography and the students know that, and you give them an O, an A or a U. They know what it means, they know what it means, you don't have to write. If you take the time to define what an A and a B and a C is, what the outstanding–however you want to do it. If you can define your evaluations, then you can use shorthand, and you have given them feedback but it is taking you much less time than if you had written something out on every page.
AP: There are actually four of my students in the room. I would like for any of them if they have something to say, Erin.
Erin Kuns: Well, I–no offense, but if I received a paper that had an O, an A and a U by all my paragraphs, I would–that paper would feel to me more like the paper that had the 2 sentences at the bottom than the paper that had extensive feedback. Because it requires, even though you may have defined what's acceptable and unacceptable and okay or whatever the O was–outstanding, there we go. It leaves a lot of room for questions from the student, you know, why is this wrong, why is this good, where as the more hand-written feedback sort of answers those questions and gives you the sense of whether or not it's okay or awful. And so, I think I would be really frustrated with at least that shorthand, although other shorthand systems that, I know I've seen professors that pretty extensive shorthand system for everything and so they sort of combine the two options of saving time, but also giving more specific feedback.
TB: Well, if one looked at the A for bibliography: references mostly current, few citation errors, coverage adequate, most relevant. And if the student can read that and if they have a question or if you could have an A and a R for it's the references that are a problem.
Another Response: Yeah but it's still not clear enough, it's not saying which references were relevant and which were properly cited. They need the details.
TB: Well, I think your exactly right, but we're talking about on the one hand wanting to save time and not even wanting to get into this, or giving the full feedback. And so, is there a happy medium that will coax some of those who don't want to spend the time into least giving some feedback.
A: Sarah?
Sarah: I was just going to say I'm a grader this semester. I don't know if other colleges have this phenomenon, but Indiana University, for certain disciplines like Gender Studies they have graders as the AIs, so I've been grading along with the professor, very much discussing every assignment and pretty extensively. And how the professor set up doing extensive feedback is having an assignment that is repeated throughout the semester. The first couple assignments you give as much feedback as you possibly can, and she has the luxury of having a grader to help her do this. But, you're giving as extensive feedback as you possibly can, and then after they've gotten that, then you hold back, and if there's still people missing it--you might have a couple papers out of 20, but you don't have to keep repeating this extensive feedback. And you have this sort of set up standard format, that they may, you know, mix-up of the first couple times, but it's obvious at least by the third time around, and it gets easier. In one of the assignments, they're learning to find the argument in the articles and as a little of a problem that seems to graduates, well I think to graduate students it's not a problem [chuckling]. As easy as that seems, it's really difficult for students who are reading for information, they're not reading critically so it's a great assignment, just getting them to start reading critically . . .
Liz from IUPUI: I have to acknowledge that I'm on the campus of Vice Chancellor Banta and these edicts that she gives to try to campus-wide evaluations trickle down to me, a faculty member. I was going to say that two responses to Bill's comment: one is, and I noticed that you had that on there, the NCA guidelines quote, which is the [TB: North Central Association] this looming accreditation factor is a key motivator because we all know everybody's going to have to sort of step up and anteing up at some point. And if these instruments are in place that you something to work with as opposed to, "The assessor's coming, the assessor's coming!" So there's that, something that brings us all together as faculty members that gets, that cuts across the fiefdoms affect. And then, the other thing that I wanted to respond to is as a course designer in the context of this project, I think for me when I am hearing all this, it's both exciting and also a little frightening because I realize that what I teach from intuitively, and because I'm passionate about my subject and I've always been drawn to teaching and I just do it. It's not in contradiction to these goals, but I haven't always been as systematic and now that this is going to be on the web and shared and potentially replicated. I've got to be so conscious of everything I do in a way that's going to take an incredible amount of work.
TB: Well, don't give up your spontaneity and your enthusiasm!
IUPUI: I know but as I'm looking at this it's going to be a huge task for every assignment to come up with things, you know, week-by-week objectives. Some of those, it comes to me as I'm racing up the stairs–‘Oh yeah, that's what we do this week.'
TB: Of course.
Other evaluator dude: If what we produce looks too complicated, people just ignore them–
TB: Exactly, you can't do something for absolutely every lesson, but you do have outcomes for the course, and you can do a checkmark the day that you're going to work on that particular thing, and also don't think that you have to measure everything, that's ridiculous. How can we possibly your enthusiasm and your rapport with your students and that sort of thing, it's not that. We want to take 5 or 6 or 8 things and give students multiple opportunities to practice those, and then see if they learn those. Along the way, let's hope, they learn a lot of other things too, like enthusiasm for Museum Studies.
AP: I think some of this stuff . . . try some of it and remember that most of the people present are here to help you do that, so you're not on your own . . .
Dean Snow: I think, we have to–it'd be a good idea at this point to recognize that there's institutional variability and one particular dichotomy that I think is a serious one. I show up every year at the AAA meetings, sit in on the heads and chair's contact that they have, and I realize sitting in that group that people who are trying to run departments at places like the University of Southern North Dakota at Hoople have a very different set of problems on their hands than I have, and the whole business of accreditation weighs very heavily on institutions like that, and they are forever being meddled with by state legislators and state bureaucrats of various kinds, who feel that they can just waltz on campus and jerk them around, and they do it all the time. Now, when it comes accreditation when those folks show up at the University of Michigan or Penn State or at Berkeley, it causes not even a ripple. [laughter] You know, they stay in a hotel for a few days, and then they go away. [more laughter] They politely send in all kinds of positive remarks, and yet I'm the guy that has probably more problems with my faculty because I've got senior faculty who think this is all crap, and they don't want anything to do with any of it. So, the only access I have to change is with junior faculty [more laughter]. So here we are! I really think that there's tremendous institutional variability here in what you can do and how you can do it, and we're under very, very different sets of pressures, and we haven't really recognized that in the last couple of days.
TB: Well if I might say, I was on your teaching ?assessment? [lots of laughter] In fact . .
DS: Were your accommodations adequate?
TB: We actually made a few recommendations that . . .
DS: Did we pay any attention?
TB: Yes, people now have better jobs . . .
GD: I mean they're real. Our provost reads those s-s-and serves, I guarantee that when it comes up in regards tenure, he pays attention. And, the younger faculty, in terms of student evaluations and teaching excellence, he does pay attention to the highest level and because of that, and people can see the consequences, there is a ripple affect, people do, in fact, have motivation. It's funny, I was talking to another chair outside of Arts and Science, and he made the same comment that the dean said. The older faculty, you know they think this is all crap, they line up and march in, they believe that that's the way they're going to be evaluated. They understand that.
Maria Franklin: Does anybody else have those posted on-line? Because at U-T, all the course evaluations are on-line, that's how students . . . it's mandatory.
Multiple people: You mean the final numbers?
MF: the summary, the course evaluations that student's turn in. [question] Oh, just the numbers, not the narratives.
Ruth Tringham: I think that what Trudy's saying is that those final evaluations are not the most valuable.
TB: Not for the work we're going to . . .
Pam Cressey: Well, I just want to follow up on what all of you are saying on that side of the room. The issue here for us, I think, is what you were saying Bill is: how do you get even us, much less them, who are going to be reading these are we're trying to influence, to put together goals, objectives and measures, to do these things. Now it's one thing to come up with a measure for certain kinds of content knowledge, a date of a projective point, or a period of something. But, when we go back to what we did at Juaculla Springs, which was hard work, and we were really getting down to basics and people were expressing themselves emotionally, not only objectively, and that's where we got to a lot of the stuff. Here, look at what we've come up with, I think at some point here, we have to deal with, how will we write goals, have objectives, have case studies, activities, . . . measured in the course. And then, if we can get also at a curriculum level within each department to look at what these are. How are we going to measure, and look at what we're saying: fundamental skills–communication, problem-solving, and this is one of the problems that all of us, that wherever I am, whatever sector you're in–when we talk about students coming out, when we even talk about ourselves. I wasn't trained in anything I do, zero.
Okay? I'm sorry, but that is the truth. I mean I analyzed 20,000 obsidian stone tools, using a book that somebody else wrote. I still don't know whether I did it right or not, but since I did 20,000 I tried to do it as consistent as possible. When I started doing historical archaeology in this country, nobody trained me in ?-ware and cream-ware and that kind of stuff, I had to go other places and try and figure it out and beg and borrow type-collections to figure it out. How do we measure that so that when Jeff or somebody gets a CR–a student to come in that you know, they know how to write. Are faculty going to teach how to write, Ricardo? Is that why people become PhD's and teach content, is because they want to go back and do these things, but say we do that checklist, which I completely agree with, and let's say that they're all unacceptable in speaking, which you need to be acceptable at least, as an archaeologists in speaking, at least acceptable, if not outstanding. How do we go about doing that? And many of us are not even, you know, we may not be great speakers, and we're not trained to speak. So that means partnerships with other people on campus, and that means really changing to whole course, I'm done.
TB: And it may mean that you find someone in the Speech Department who is willing to give your students particular opportunities in a particular course, and may mean changing to that, changing the elective or the requirement to that–
PC: It does, so this is a big structural change, where . . . I was reading this over this morning, and do you realize what that this thing says, what we said, or what George and Susan wrote: is that we don't need major changes in the curriculum. What we need to do is infuse these values into the current curriculum in the current classes, but when I look at these things, and I think about accountability. I don't see those really [connecting].
Rosemary Joyce: I just have to dissent because I think that this is a premature judgment of what the enterprise is, and the fact is that those seven values. The only point at which we're going to be able to see whether we're infusing them in these curricular materials is when we have the curricular materials of the course designers go back after this workshop and design, and that's why this is a three-year process with recursion to–I think that the kind of issue that you brought up, that the kind of issue that you brought up, that you weren't specifically trained to deal with obsidian, or specifically trained to deal with the historic ceramic sequences, is actually kind of a red-herring in that it would seem to be a content issue, where what I think you're trying to say is that the things that we train students in, we cannot predict . . . we will never be able to predict all of the things students will have to be able to do.
So what we have to do is actually do is work on those critically-thinking skills and the things that, in fact, are somewhat portable, and infuse these things in the exercises, the evaluation, the kinds of questions that we actually ask people, which isn't just for evaluation and assessment, but it's also the way that you reinforce what was the important material, that's part of the reason, and as far as getting faculty motivated to do it, the top-down approach seems to me to be, you know, we're thinking about the stick, but if the carrot is that if you do this, teaching is more enjoyable, and that's the thing that will ultimately convince people to do it. That's maybe why the younger faculty are willing to do it, and somebody who's been teaching the old model, which is very unrewarding, eventually gets so burned out that they just don't want to do it. And one final thing, as long as I'm ranting, following the grading thing. The twenty papers, and I absolutely agree, you can do this kind of detailed thing on the first two papers and on the third one, but 20 papers is an unusual circumstance, and I think we have to really think about that in creating curriculum for large numbers of people teaching out there. My small class at Berkeley, my Meso-American Intro class, the small as it's been is 85 students. It's routinely 135 and 150, and I get a grader who's paid for 2 hours of grading per student across the semester, and that happened to me the first time I taught that course at Berkeley, and that happened to me the first time I taught that course at Berkeley after having taught it at Harvard with luxurious amounts of support at which I said, "Okay, all [Interjection?] these writing assignments I did, I'm not going to be able to do a writing assignment a week or else there won't be the rapid feedback, which is the thing that makes a ?. . . writing assignment successful." So the O, A, U thing is better than no rapid feedback, and we're again talking about an extraordinarily diverse body of people that we're trying to create resources for, and I would continue to stress that we have to be very flexible about the way we understand it, thus this will seem too much of a straightjacket for anybody who's at all adopting any ?...
TB: I just want to point out that those of us who are evaluating here are thinking that these 7 things need to be listed as outcomes because that's what you're after, and then you need to figure out how they're going to be taught and be taught in a lot of different ways, and then you've got to assess the student learning to see if, in fact, these things are learned, and if the students are failing then, you need to go back and look again at how it's being taught. So this alignment of these goals is what you're after with teaching, with assessment, and the getting that feedback through yourselves out of the assessment to say, are we helping students learn this stuff or not?
Vin Steponaitis: You answered my question before I asked it. Pam's comments raised the question that I think all of us have been thinking about, and just speaking for myself, I've learned so much from the presentations I've heard over the last few days that I know that it will inform my own teaching. But, I've been thinking about how do we take this and present it as part of the results of this project in a way that will have an impact, or that might have an impact on the discipline, and one way obviously is that the course designers will incorporate this in their own various ways as they see fit. But then, I was thinking is there anything we can do beyond these model courses? And, I certainly haven't thought this through, but I'm raising the question, and I guess at this point, I'm thinking on the fly and my answer is ‘yes and no.' I mean, yes, I think minimally one thing we should do is make a small, briefly annotated bibliography of some of the resources that you all have brought to our attention, and that doesn't preclude also having a bigger bibliography, but I think if we just give, put on the website, a reading with no annotation, people look at this and say, "forget about it." But if we put 3 key things you should read, and a brief paragraph or a sentence under each one saying what this does, maybe some people will actually read it . . . .
So that's one idea, but the other thought is that to what extent do the 2nd kind of evaluation that Trudy was talking about, which is to sort of look at the curriculum as a whole and how it's working. And that presupposes that we're doing a curriculum as whole, which I don't, you know, we've been discussing that over the last couple of days and I think there's been some difference of opinion on that. I personally don't think it's practical to do a whole curriculum just because I think if we would present a whole curriculum, it won't be adopted by anyone because of the differences among universities and colleges in the way archaeology's taught, and the fact that archaeology's always taught in the context of a larger department, most who do things other than archaeology. . . . So if we were to come up with a curriculum, then perhaps we could think about ways of doing a sort of overall [approach?], but if we don't come up with a curriculum, and we just come with pieces, we're probably best to focus on doing the kind of things Trudy's talking about at the individual course level, rather than trying to do it overall.
TB: Exactly.
Ann Pyburn: I have something to say to this that I need to bring up sometime today, so I'll just take a real quick minute to go ahead. I think that we don't have any sticks, we just have carrots, so all we can do is make this look as delicious to colleagues as we possibly can, so any thing that you can do to candy-coat what it is that you're offering to make the feeling to other archaeologists, to your colleagues, please do it, shamelessly. I think, ultimately, whether or not, we have, what kind of complete or discreet curriculum, we come up with; it's going to be in the court of the course designers, because the people who are going to ask you questions about how complete is going to be, so far have all been people who are consultants to the course designers. It's in the court of the designers how they want their designs to fit together, and I would like to say that I think that course designers should make course that respond to the MATRIX project, not just to the departmental needs of the particular departments. I don't see why you can't do a happy combination of that, I think that some of the combinations that people have turned out to be quite wonderful, but I think that, please think first about designing something for the MATRIX project that will go beyond your department.
I think we should all want to combine the interest of the individual designers and their expertise with all this available support, and that's, the meeting is about showing what support you can get, showing you what kind of information, what kind of knowledge, what kind teaching, what of content-support is available. So now, you've got the smorgasbord in front of you, tell us what you want. It seems to me that flexibility is really important for this, but also some kind completeness. The courses that we've proposed so far, that have been proposed so far are reasonably diverse, I think a little replication might be okay because no-one, as Bill pointed out, is ever going to take all 12 of these courses, and it might be okay to have Intro courses designed by two different people so that there's some choice within the discipline, "Gee, well I like the one designed by Nancy rather than the one designed by Francis, so I think I'm going to use Nancy's intro course." Or, "I'd like to use part of Nancy's and part of Francis's." The other thing is that Skip and Susan are talking about designing one course that's actually two semesters that fit together, but that could stand alone, an intro course that has _- emphasis on physical and _-emphasis on archaeology, but it's still intro to North American, native North Americans so it could be, and there's a kind of a thing, that could be worked really well for lots of contexts–for the campus where there isn't a physical anthropologist to teach the one side, and the campus where there isn't an archaeologist to teach the other side.
At any rate, . . . I hope you will put up course resources, including readings and lectures in some cases, and sources, video-clips, we can do video-interviews of specialists, stick ‘em up there on the web, slides, the manuals instead of syllabi, projects, comments, and hopefully comments in the introduction about which of the seven concepts are going to be emphasized in a particular course. And I like the idea of making them as kind of decomposable models, you know like the sedentary thing. [others chuckle] We have these modules, or sets of modules, the sort of course plans, we can then provide, an overview of the final 12 that proposes several possible curricula that could come from the set we've got. If you use course A, you probably need to choose B, C, or E. If you choose E, then you'll be missing ethics if you don't include F. You see what I mean? So we can make . . . the recipes for the courses, now here's the meal plan, here are a series of meal plans, or what would be a complete curriculum, according to our committee. And, you know, you wouldn't even have to [use everything], it could say. "or substitute your own culture area class." We could also do the kind of if/then thing. If you just use this module from this class, be sure to include module from class 12. See what I mean, it's kind of a leggos.
RJ: And the kind of, I think the checkbox thing that Trudy showed us would work very well to put the principles as overarching goals for everybody, not necessarily going to be equally obvious and evident in all course, but at least then it's a checklist and you can look at it and then you may find that at first you think, there's no impact, I personally think that these all would be in all of them, but let's say you don't think that ethics and values will be in your course, and then you're going along and you suddenly realize, "Oh my god, of course this in an ethical issue!" And then, you can check the checkbox off as a way to go about doing this.