Trudy Banta: Using Evaluation to Design Effective Courses
Introduction of Trudy Banta by Ed Neal, she's an expert on program assessment
TB: Wow . I'm sorry. [laughter] First of all I appreciate the opportunity to come to another campus every now and then. I'm at a very interesting hybrid of Indiana and Purdue University in Indianapolis. And, I'm a member of a faculty in higher education here at Bloomington, but I take part in most of the faculty meetings at the other end of the TV, watching the meeting take place on the screen. So, it's much fun to come a place filled with limestone the way this place is. You heard from my colleagues yesterday, and as we thought about it–even though we didn't confer very much before we came, I think the work they did with you yesterday really builds very well to talk about taking one more step. And that is, I call it a second look at student learning.
First of all, a rich conversation about student learning informed by data–what does this mean? Well, I was thinking about that, and thinking that you might do, like an archaeologist might walk into a faculty lounge, if there are those any more and encounter a philosopher, a faculty member of philosophy. And, as the conversation proceeds, the philosopher says, "You know we prepare our undergraduates to go to graduate school." And so, as an archaeologist looking for the data, you say, "Well then, how many of your undergraduates go on to graduate school?" And . . . he . . . says, "Well, Connie came back to see me at Thanksgiving so at least two must go on to graduate school." And then the archaeologist said, "Out of how many?". "Well, I just teach sophomores and I'm not sure how many graduated last year, maybe 12, 15, maybe 20." And so the archaeologist is not impressed by these data and turns the subject to the weather or baseball. And so, three or four years later, the archaeologist comes back to the same place and encounters another philosopher. This one says, "I can't wait to tell you, I'm so excited, we're changing our curriculum!" And the archaeologist says, "Well, tell me all about it--Why?" "Well, about 3 or four years ago we decided that we would extend for all of our graduates the opportunity to use e-mail, the university account. In their junior or senior year, we told them that we wanted to keep in touch with them, and in order to continue to build our program, we would be asking them some questions from time to time, some things about their experience, about what they learned while they were with us and what they didn't learn. And we told them that it would be very important for them to get back in touch with us and let us know and respond to us, and promise not to take very much of their time. So for the last three years, we've gotten over 90% of the graduates responding, and we find that fewer than 20% are going to graduate school. So we've decided to change our curriculum to a more applied, action-oriented curriculum that will help them learn to work in teams and make presentations and do other things that will help them in work that they may do immediately after college." THIS is the conservation informed by data that most of us would prefer. So, that's what I will be talking to you about a little bit this morning.
There are all kinds of ways to assess student learning, and you know what they are because you use them all the time. There are assignments and then quizzes and exams that you give in class; projects and papers, journals, maybe some portfolios, where you observe students in laboratory settings or on site–all ways of assessing individual student development. In addition to those things, some give placement tests so that we know how to counsel students about the level of coursework to take. We may even give an exit program exam, in some fields there's actually a licensing or certification exam, and we want to even have a capstone experience in some areas. So, we assess individual students all the time, and we can assign grades to students on the basis of this individual assessment. And, students learn to assess their own strengths and weaknesses throughout life. But, there are other questions that we may want to ask. Just in general, are students strong in all of the areas that we have in mind, or are there some places where there may need to be new teaching or teaching in some other way to help students grasp a particularly difficult concept? And then, which approaches produce the most learning for which student? This is a question that if we could answer . . . we could really move ahead in terms of producing learning.
So, what I think we need is a second look at what students are giving us, student work. We take the materials that we have for individual students, we give the grade, and then we put in a pile–it doesn't matter who the student is; and then we look across the students for each of the questions or problems, and we get a sense of how many students are getting it and how many are not. If we have a course with multiple sections, we could look at students across sections, or students across courses if we were looking at the entire curriculum. This we might call group assessment, and we use the very same things as raw material that we use for individual assessment. But now in addition for group assessment, we might ask, do students benefit in a group, in an individual interview session, in a survey instrument. We might have a look at transcripts to see how students are progressing through the curriculum, what kinds of courses they are taking–that would give us information that might be useful. Certainly where they are placed–are they going on to graduate school or are they going on to academic positions after graduate school? Have they received any particular recognition for their work? What about how they do in graduate school? And where do they go after that and success on the job. All of these are pieces of data that we can use to improve our programs. Incidentally, the data then also could be used if we were approached by a regional accreditor like the North Central association, which is our regional accreditor here. As they come and make visits to campus they ask us to provide evidence that students are learning, they're achieving what we want them to achieve. And by taking that second look at the data, they say, we can encourage excellence teaching and work to improve student learning. North Central is particularly interested in the changes in curriculum, or in the changes in pedagogy, whatever may be done on the basis of the data were collected in the student improvement part of this.
So, to summarize what I've just said, assessment has many purposes. The first two of these are the individual student assessment, where students learn content by virtue of the feedback we give them and with this can assess their own strengths. 3,4, and 5 are what we get when take a 2nd look across students, in order to see where there might be some concerns and take steps to improve instruction, grab some fruit to help curriculum or a service like advising, and then we can also demonstrate accountability.
Yesterday Ed and Lion talked to you about writing clear objectives, using a variety of methods and materials in teaching, helping to make students active learners through a variety of active learning strategies, and perhaps also about developing some assessment tools for testing higher order intellectual skills such as analysis, synthesis and evaluation. Here is a real life example (I think Ed gave you some materials that talked about the particular steps, or pieces, or traits, or elements of an assignment or an exam or whatever). Here we have a broad goal to develop a research paper, but that can be split up into several steps–finding a topic, developing a bibliography an outline, writing drafts, and finally an oral presentation. So, in taking the second look across student work, we've got 5 students here and we have just simply recorded after we get a new grade, whether they were outstanding, acceptable or unacceptable. This is a 3-point scale but it could 5-point or a 7-point, however many points you want to use. But just glancing at it–what would you do next with this group of students? [See visual 1] Where would you start?
[Audience mumbles oral presentation]
TB: A couple places look like some focused work might help. Without this information, without this 2nd look, you wouldn't necessarily know, you wouldn't have the data to tell you what your next step needs to be with this particular group of students. So I think that's saving you time, a 2nd look may take a little more time, but I think it will save you from going down the road of–"Well, how about let's work on outline." [chuckling] So, I indicated that these are the pieces of the assignment, and here's outstanding, acceptable and unacceptable. Let's take just the bibliography and define what outstanding, acceptable and unacceptable really mean. When we can do these for ourselves, it's easier for us to feel like we've graded objectively, no matter when we do it–early morning, late evening–whether we're feeling good or hungry or whatever. Perhaps, best of all, we can almost use a shorthand–O, A, U--then the students know because we given this information to them, and they also know exactly how they are going to be assessed in terms of their bibliographies. It's that alignment–testing with teaching and evaluation. So giving meaning to each of those ratings or areas would be a good thing to do. Here's one more example. Ruth [Tringham] was talking yesterday about the experience of a student trying to imagine what it was like in a particular setting a long, long time ago, trying to immerse herself in the setting. I think journals can help students express that, and we are finding ourselves teaching reflective writing. This journal, especially if it is explained and assessed in this way, I think can help students step-by-step get into reflective writing. If, for instance, we ask that the entries just describe, no emotion involved, just describe the site, the people, the site. And then, describe the responses that they have, just putting what strikes you. And then, try to analyze and evaluate your experience in that site to find meaning–causes, effects, patterns. Then, when we look at the journal we can grade it: well-done, satisfactory, unsatisfactory. Again, with descriptions for each of those, and the students will understand why they received the grade that they did.
For a curriculum, there are going to be a number of outcomes, and if you want to see whether students are having opportunities to learn these things, we can look at the syllabi, we can talk with instructors and find out if certain outcomes are being covered in that course. Now, here is one (see )–number four–that only got covered in the senior year. It could be that bibliography! They never really had an opportunity to do the bibliography until they were seniors, or at least juniors. And here's one (see), and maybe they only got to make an oral presentation when they were sophomores; they didn't make any others. Now obviously there are going to be more than three courses; this might just be a three course sequence that we're looking at, but you could spread this out and put all the courses in the curriculum and have lots of checkmarks. But the point here is how many opportunities, over what period of time are students going to have the opportunity to learn the things that we stand for, and this is one way of doing it.
Another way of doing it would be to look at–and this is right from Ed's institution in Pharmacy. The course instructors, and this is sections of the same course, so the course coordinator for that course is asking the instructors to, on a daily basis, to just say, "what are you going to try to accomplish for the students today." Post that on the Web, and then at the end of the semester, they can look at all those daily outcomes that were anticipated and see whether, in fact, the objectives were all covered. They can do this at the end. That's sort of a real practice-oriented way to do it. And, they look also at syllabi, and the students' evaluations of the courses, so they have multiple ways of finding out whether students actually had these opportunities.
Here's another way of looking at this whole process. If we have our goals written in with action verbs (See), and then we see which courses they are being taught in. In those courses we can ask the students what they've learned–through portfolios or tests or projects or whatever–and that verb almost dictates the kind of assessment activity that is needed. And so, for anyone who sort wrings their hands and says, "what is the best assessment we can use?" Just look at the goal and what is stated, and this will almost leap out. I have another version of this in your hand-out so I'll tell you about it in a second.
Q?: What did you mean by finding the uses there?
TB: Oh, well let's just look at it right now. The very last page in the hand-out is a matrix. (See) And I believe that yesterday, Ed and Lion asked that they develop some goals and objectives for student learning. Well you can take those and put the goal in the first column and the objectives, what students know and can do, in the second column. And then, how will you have the students learn it–that's where are the courses in the curriculum that will help them learn these things. Or, do it for an individual course, what are the experiences in the course that will help the student learn? Will it be an out-of-class assignment, will it be a paper, will it be a project, will it be an on-site kind of thing? How will they learn this? That's what goes in this column. And then, how will you measure it, that's the assessment part, through a test or asking students whether they actually mastered certain concepts. So what have you learned, what are the findings. That would be the findings here. (See again) We found that [the students were] pretty strong in being able to define the topic, but not so strong in doing the bibliography, very strong in doing the outline, not so strong in doing the oral presentation. So that's what we learn. Next column, what are we going to do about it. That's up to you.
I've been talking primarily thus far about direct measures of learning. The assignments, the tests, the projects, the papers are direct ways of finding out what students know and can do. But there are also some indirect measures that are also very helpful to us in this process because if a student makes a 95 on an exam and another one of them makes a 45, what did the one who made 95 know and what could they do for the one who made 45? And why [did the student make a 45]? Well these indirect measures can begin to tell us why.
The typical course evaluation that most faculty will give to students toward the end of class that will give you some information, but you can add items to that to ask questions like, Did the course cover the objectives that were listed in the syllabus? And for each of those objectives, how much did your knowledge increase? Did the teaching methods that were used help you learn? Did the assignments help you learn? Questions like that will give you feedback about what you might do differently to help students learn–how to do the bibliography or how to make an oral presentation.
Here is an indirect measure, but I think it's a great way to . . . . When you say, how much did your knowledge increase that's a pretty subjective thing for students to respond to. But here is a way. Cross and Angelo talk about the background knowledge probe, which could be used as a pretest. And you could just put down a set of terms, and the during the first day of class find out how much they already know by asking them, Have you ever heard of this? (SEE) Do you know a little bit about it? Or do you have a clear idea and can explain it? And, I think this is a way of getting real information in an objective way about how much they know. And then at the end of this course, you have them prove that they're up to D by actually explaining right there on the spot. And you can compare the A, B, C's and D's that you got at the beginning with those at the end and have another measure of what things were really learned well by lots of the students and what things were not-so-well learned.
I'm going to finish here with just a few generalities. We're all faced with a choice of "Shall we select something that's already available to do our assessment or shall we design something ourselves?" And, a very important thing to help you make this decision is that match or alignment with your goals and objectives. What produces the best match? If it's the standardized exam, great, but most standardized exams are not going to really match what you're trying to teach. So that leaves us with the alternative of designing things ourselves, and another thing is that no one of these assessment methods is going to tell you everything you're going to want to know, they're not perfectly reliable or valid. So we need to use a variety of measures and look at the ways in which they come together and tell us the same thing. And that strengthens our feeling that we've gotten the best information that we could.
I've already talked about direct and indirect measures–qualitative and quantitative–we want both. We want perhaps to give some multiple choice questions where they're appropriate, but also give lots of opportunities for students to write their own opinions. And the pre and post background knowledge probe would be a way to do that, another way to do that would be just to give the final exam right at the very beginning. I think that might be discouraging, [chuckling] but that's another way to look at gains and then always using built in points of contact instead of calling people in on a Saturday morning to take a test as is done on some campuses. And, if you would like some indirect measures of the entire student experience, seniors need to apply for graduation and that might be a time to give them a little questionnaire or call them in for an exit interview with the department chair or . . . .
Here is a pretty good example I think of some of the things that faculty say that they get out of this ‘Second Look.' They can clarify both for the students and for themselves the goals and objectives that they have for student learning. When students are involved in this process, giving you feedback and knowing that you're listening and making changes based on what they're telling you, they are more motivated to do the work. You want to certainly determine their retention in course learning and perhaps over time. One of my stories that I've gotten over the years from case studies . . . at Virginia Tech they did an exit interview with their seniors in math and found that to their horror and chagrin there were many seemingly basic concepts that the seniors had just forgotten. If they looked then back through their curriculum, using their little checkmark system for where did the students learn it, they found that the students learned it when they were freshmen. The faculty had assumed that these things were so important they were just carrying that knowledge right along with them, but they never had another opportunity to practice it. So that led the math faculty to get together and say, "What are those things that every math major's got to have. And then, let's make sure that they get practice at various points in the curriculum. Let's put a check mark where there's an opportunity . . ."
So consistency of instruction in required courses, as you know, a section of 101 by one professor and a section of 101 by another may be very different. And, should there be a few things that we all can agree ought to be taught in 101, and if so, then we would use our assessment activity to determine whether there is consistency. Factors that inhibit student, the checkmarks again showing the bibliography . . .and identifying course material reinforced in a capstone because we think it's so important that they shouldn't walk out the door without that knowledge.
So these are some of the things that are involved in taking that ‘second look' to assess the learning that is taking place and to always be in a position to correct all along the way so that more is learned by more students. For my heart, this threesome works with you over the next years to evaluate this project I hope that we'll be doing some ‘second looks' at direct learning and also doing some of those indirect measures to get the formative feedback about what you're trying to accomplish to make sure that the students are getting it, and they're constantly getting you information back about how well. Questions?