I had meant to get back to this subject sooner than this but I have been ill. In my last blog -- the one right before this one -- I tried to explain why I think much of the current discussion, at least the discussion that we are having at Mount Allison, with regard to Student Evaluations of Teaching (SETs) is both misguided and, in fact, moving us in the wrong direction. Rather than promoting good teaching, it seems to be driven by an abstract series of propositions that use a certain discourse -- students want a voice -- even while they will deny students the very voice that they want. In other words, if we can agree on first principles (that students should have a voice in the way the university is run and that we want to improve post-secondary teaching), we can almost certainly find good and productive ways to meet both objectives.
This is important to say because when I raise concerns about Student Evaluations of Teaching, one of the first things people say to me is "so you don't want students to have a voice." To the contrary, the problem is that SETs won't really do that. Why? There are two questions to answer in this regard: how do they work now? how should they work?
Right now, most places that use SETs, have them work like this. At the end of the semester, students fill out a form that evaluates faculty on the basis of various criteria on some scale. We use a 1 (strongly disagree) to 5 (strongly agree) scale but I've seen models that have used other scales over the years. The precise scale, of course, does not really matter too much. The issues are that it is (a) accurate and (b) used consistently over time. After students complete these evaluations, they are sent to the dean who then goes over them and, on this basis and that of other criteria, makes an assessment of whether or not the faculty member is satisfactorily doing their job. There is a big asterisk here for Mount A: submission of SETs to the dean at the end of the academic year is voluntary. My best guess is that over 80% -- I've heard one person in the know say 90% -- submit them. (I'll give you what I do in another blog).
What has happened? Well, no one would argue that SETs alone -- the supposed way student voice is integrated into the teaching process -- is *the* evaluation of teaching. All the guides (including that of our own Purdy Crawford Teaching Centre) say that SETs should not be the sole criteria of the evaluation of teaching, that interpretation of SETs is needed (a point I made in my last blog) and that we need to assess change over time (is a teacher getting better, worse, staying the same; are they doing some things well and not others, etc.). So, using this range of criteria, the dean factors in SET numbers and offers his (our deans are all male) assessment.
There are a couple of things here to note:
1. We are not clear, here at Mount A at least, what other criteria are actually being used. I am going to suggest that there are some criteria that should be used but these are not specified and, in my experience, confuse even faculty members.
2. The student voice has been lost. Students make both qualitative and quantitative comments. In my experience, the quantitative ones are the ones that people want to read because they are the easiest to seemingly understand. I don't think they are for reasons I explained in my last blog but I do believe that this idea -- that numbers are easier to understand than written comments -- is a common fallacy. By looking at numbers only (which is what deans do -- not claiming this is a fault, merely trying to be empirically accurate), not a single student voice is actually heard. Instead, they are merged together into an aggregate and that aggregate stands for student views.
I'll leave off on the first point I made for now and address it in another blog in order to concentrate on the second. Let me give an example to illustrate my point (hypothetical).
Example: one question on our SETs at Mount A (one to which I pay a great deal of attention) asks students whether or not the method of evaluation was fair and appropriate. Imagine my responses were as follows:
Number of students Ranking
3 1
2 2
2 2
18 4
8 5
Admittedly these are not bad numbers I just made up and that might allow you to instantly see the logic of what I am doing. My average score on this question would have been 3.94, fairly close to a 4 (agree) on our scale. So close, in fact, that no dean would ever raise a concern about it. But 7 students did not agree. Two were neutral (3) and five (1 + 2) disagreed or strongly disagreed. Hence, slightly more than 20% of the class had problems -- of one sort or another -- with the mode of evaluation. But, in this example, this 20% (one out of five) is not heard. Now, think about this. If one out of five of anything had problems with what you were doing ... that would be cause for some concern would it not? If you owned a restaurant and one out of five customers thought your food sucked ... you'd be concerned, right? But, that ain't gonna happen the way our system (or, as far as I can tell any system of SETs evaluation) is set up. Thus, the very mechanism that is intended to provide student voice is actually depriving one of five students of their voice.
But, I think the problems go deeper than this. Not only have 20% of students been stripped of their voices but the other students have been as well. They have to trust that the dean will reflect their views. In other words, they don't get their own voice. Their voice is little more than the equivalent of a quality control telephone survey. Ever done one of those? Were you satisfied that you, as a person, were actually listened to? This is my point: SETs don't provide student voices but mediated student voices that rely on another person to articulate their views for them. What is more, that person is (a) not a student, (b) a single individual, (c) has no student oversight himself, (d) does not have to justify his assessment to anyone or explain the criteria he used. In short, this is a mess not because the people involved are inherently bad people (they are not) but because no one would ever set up a method to hear student voices in this fashion if they actually wanted to hear student voices and ensure that the system was operating in a fair, responsive, and reasonable way. For instance, and just to make my point, if you believed that students should have their "say" would you design a system that gave their say to someone else? If you were interested in ensuring that a large number of people were heard, would you intentionally reduce their voices to a single voice who was not one of their number?
This is also a problem because if leaves a series of important questions unanswered which students, in my experience, use their SETs to answer. For example: what does a rating of four (or, any other number) mean? What does a three (neutral) mean? (Think about that ... an interesting question. Its neither an endorsement nor a criticism ... so how do we read it?) Do we have any criteria that differentiate one number ranking from another; that is, what is the difference between a 4 and a 5 or between strongly agreeing and agreement? What does it mean to ask -- in the same question -- that a mode of evaluation is "fair" and "appropriate"? Could it be fair but not appropriate? In other words, what, for students, constitutes "fair"? What constitutes "appropriate"?
These are not idle questions, as I have said, for two reasons. First, they are not idle because they are what we are trying to accomplish. We want education to be fair. We want assessment to be appropriate. Moreover, I suspect that we could, relatively easily, agree on a set of criteria that met the standard of fairness and appropriateness if we would just talk about it. In other words, rather than trying to find ways to implement rules of evaluation, why not hear student voices by asking them to comment on questions? Why not enter into a conversation on the subject?
Second, because my students often use their qualitative comments to augment and explain their quantitative assessments. Some will focus on specific lectures; some will have concerns about a specific assignment. Others will question the merits of specific assigned readings or suggest other readings that might improve. In other words -- and without putting words into my students' mouths -- they treat their qualitative comments as something that the way the assessment of the quantitative comments does not: as part of an on-going conversation about a specific course and the way it was taught and developed. They are offering to have the very conversation that they want to have. The problem, then, lies not with them but with the way SETs are used. It refuses engagement in the very conversation that we can and should have.
Let me sum up. I don't have problems with SETs. I use them and use them all the time. They are useful and most faculty devote a great deal of attention to them. The problem I have with them is proposals to implement them as an evaluative component of faculty work under the guise of hearing student voices. What I've tried to explain -- and you can let me know whether or not I've succeeded -- is that the process of using them as a mode of faculty evaluation necessarily fails to meet the goals that are being set for them. Rather than facilitating the voice of students, the use of SETs as tool of evaluation deprives students of their voices and transfers it to (at Mount Allison) an unaccountable administrator (who never has to report to any student or student body about any decision or evaluation they have made and never do) while short circuiting the very conversations that students seem to want to have about post-secondary education. I said that no one would intentionally design a system that worked in this way or, if they did, most of the rest of us would wonder about it. For example, if you wanted your car repaired, you would not knowingly do things that prevented it from being fixed. This is what SETs do in terms of hearing student voices. And, that is too bad because those voices are important and should be heard.
No comments:
Post a Comment