Student Evaluations of Teaching ... Having a Real Say Part I

In my last couple of blogs I've tried to address so of the issues surrounding Student Evaluations of Teaching (SETs) at the post-secondary level. I've tried to argue several points:

1. SETs are useful tools that can help instructors improve the quality of their teaching
2. Mandatory SETs assessed by a dean with punitive repercussions don't allow student voices to be heard but simply, in fact, transfer author away from students to administrators (who are not responsible to students)
3. There are other problems with SETs that serve to erase student voices, particularly those who hold minority views
4. Most faculty do already listen to students and they do so in a variety of ways. It is, in fact, one of the more time-consuming (and, I might say, rightly so) things we do in terms of teaching.

Let's take this discussion a step further and consider other ways of listening to students. One of the things that attracts people to SETs, I've said, is their seeming simplicity and neutrality. They provide numbers that seem simple (even though I've tried to indicate that they aren't) to interpret. There is some evidence that people are starting to understand that SETs do not meet their goals. That is: they are not having the effect that was desired when they were implemented. That effect, I believe, was to improve the quality of post-secondary teaching and ensure a student voice in the operation of universities. Both of these aims are more than "fair enough." They are legitimate. Yet, if the tool through which one tries to meet legitimate objectives fails ... should you continue to use that tool?

Let me try to address this question by looking at where it comes from (and, why it is important) and how we might better proceed. I'll argue for a multiple pronged process that involves a reconsideration of first principles as well as a willingness to actually communicate as a mechanism of ensuring  that students have a say. Before I do this, however, let me draw what is an important distinction, particularly for the argument I am about to make below.

The distinction I want to draw is between processes that are intended to ensure communication and those that are coercive and punitive. Communicative processes are those I discussed in my last blog. People's voices are heard through a process of talking to each other. Coercive punitive approaches are those that use a threat or an implied threat in an effort to achieve certain results. For instance, if my boss wants to improve how I do some aspect of my work he -- and, my boss is a he -- has choices in terms of how he goes about it. He could (a) talk to me and work through whatever the problem is (perhaps along the way, he discovers that the problem is, for instance, idiosyncratic and has occurred because of an illness on my part or some other temporary problem) or (b) threaten me. "Nurse, if you do not meet standard X on SETs, I will rate you as 'unsatisfactory' for teaching and this could have implications for your pay." We don't go quite that far at Mount A, but there are people who want to. There are people who believe, in fact, that my dean should be able to dock my pay, in effect, then and there, if I don't meet whatever standard he feels I should.

In the discussion below, I'll call (b) coercive/punitive or words to that effect. I have colleagues who will protest that characterization or use a different language to explain it but if you think about it for a minute, you'll see my characterization is accurate. What is going on here is that coercion (or, the threat of punishment) is being used as a motivational tool. If you find the name disturbing ... that might say more about your views on coercion than my characterization of it.  My view, to tip my hat up front, is that people are way, way, way too fast to use coercion as a tool. There are other, better, and more mature ways of approaching problems and issues and goals, as I will sketch out below.

The Issue

This is an important question because of the process through which we have gone  at Mount Allison, which is, I think, pretty normal. When I first started at Mount Allison SETs were voluntary: some faculty had students complete them; others did not. There was a form one could use provided, I think, by the student union. We progressed from that to mandatory SETs and from that to a debate about two further points (a) should mandatory SETs be evaluated and pay docked if a dean determines a faculty member did not meet a standards -- that is should we adopt a punitive approach as an institutional policy --  and (b) should we have more than one set of SETs during the semester? In other words, instead of having students complete SETs at the end of the semester, should they also complete  them at the halfway mark of a course? Some faculty do this voluntarily, but should they also be made mandatory? I have heard some people suggesting that these might, in fact, be completed several times during a semester. In other words, students would be filling out surveys in the manner of a Nanos Research rolling poll.

You can see what has gone on. We have moved through a series of stages -- voluntary, mandatory, punitive assessment and multiple evaluations. The reason for this progression, I think, is not just that someone somewhere loves assessment and imposing penalties. I think this transition has occurred because people were not happy with the results. It is logical to change what we are doing if we are not getting desired results. The problem, in this case, is not that we are not getting the results that we desire (indeed, I'd argue that we actually are). The problem is that we are vague as to what the results actually are. What does it mean, for instance, to have "a say?" How can we measure improvements in education? Surveys and numbers seem like a self-evident way, but are they?

I am not dissing SETs. I think they have a role and I've said this a number of times, but they are not the only tool we should use nor inherently the most effective. I said in my last blog that I thought direct communication was the most efficient way to address problems of teaching and learning. "Prof Nurse, I did not understand X. Could you explain it?" This is efficient because it helps the student learn as opposed to punishing me after the student has *not* learnt. Think about that. Punishing me after the fact serves no effect. I get my "comeuppance" but the student still has not learnt. From an educational perspective, what good is that?

The issue, then, in my view, is not to take a tool that is not producing desired results and make more use of that tool. This is what people who want to ramp up SETs and increase their coercive and punitive character are actually saying. In a perhaps more articulate language, they are arguing this: the tools we have are not giving us our desired results so let's make greater use of those tools. Does this make sense to you? I think the issue is something different. The issue is to re-assess our beginning point and have a discussion about what our goals are and what they mean in practical terms.

For example, students want to heard. What does that mean? Do you want to be heard for the sake of being heard? That might be fair enough but I actually doubt it is the case. I suspect that students want to be heard because they are interested parties in education, have a contribution to make to pedagogy as well as specific class content, and want to reward -- not just punish -- those faculty who they feel have really helped them out on a variety of levels.  In addition, I think some simply want to be part of a conversation about the future and development of post-secondary education. That is: they want to play a role in improving it.  The point I've made before holds true again: none of these goals can be met through more punitive uses of SETs evaluations by deans or other supervisors.

Student Perspectives

I can't articulate a student perspective, because I am not a student but I make the above statement on the basis of extensive conversations with students over the years. Very, very, very few students I have encountered actually want punitive and coercive measures against faculty. I have talked to one or two who believe that there should be coercion and threats in pedagogy. But, the vast, vast majority are well-meaning, well-intentioned people who are sympathetic to faculty and the different aspects of their jobs. I've directly asked classes I've taught these types of questions and always walked away with favourable impressions of the discussion that ensued.  Indeed, what impressed me was the difference between what students say and what people who speak in students' names say.

For instance, I've had people (say, faculty or administrators) speaking for students say "students want a voice and that means that your dean should be able to dock your pay." They have not used exactly those words but that is, in effect, what they said. As I said, I've had in 15 or so years teaching only one or two students who have made that case. Most don't want to see my pay docked but want to see education improved. While I've heard faculty and administrators argue that threats are a good way of motivating faculty, I have actually never had a student say that they think threats and coercion are good motivators. Even the very small number who like them stop short of saying that threats will, ipso facto, improve teaching.

This is, then, an odd disjuncture and one that causes me a certain amount of pause. Students seem to be saying one thing, while those who speak in their names seem to be saying something else and the two do not match up.

Models

Because of this, we need to re-evaluate the model by which student voices are heard. By returning to first principles, by grappling with out objectives and with student aims, we can, I think, build much more accurately designed mechanisms for student voices to be heard and for us to make the most out of post-secondary education. Moreover, I think this can be done without coercion and punitive evaluations. I think, in fact, the overall results will be better.

This blog is already getting too long so let's leave off the issue of teaching and learning to keep focused on student voice. How can we hear student voices?  In addition to the significant amount of talking that already goes on between faculty and students, we, in Canadian Studies at Mount Allison, have done other things over the years.

First, we try to directly connect with the student club. Most department, programmes, units have a student club of one type or another. These might be made up of particularly motivated students, but ... good. There is no reason why we should ignore someone's voice simply because they are motivated, is there?

Second, we've had students complete end of degree surveys which call for extended qualitative comment. The idea here is that students finishing their degrees will have a certain perspective on what was good, bad, what helped and what hurt the learning process.

Third, we have student focus groups. We don't do these all the time, but since I've been at Mount A we've run them on academic integrity, the requirements of the honours programme, and to solicit feedback on curricular innovations.

Fourth, in class "academic moments". Usually in 3000 level courses (but these could be done at any level) I schedule a series of academic discussions (no more than five minutes unless there is a good point being made or someone who really wants to speak to an issue) about teaching and learning. Topics could include such things as the role of SETs in teaching, different modes of evaluation, responses to integrity problems, course formats, etc. These are, I suppose obviously, done on an opt-in basis. No one is forced to talk if they do not want to.

Some other ideas that could be used include:

• holding open fora on key matters of curriculum or teaching and learning
• regular (ever week, every month, once per semester, etc., whatever people want) faculty discussions with students about key issues, say the role of research in education, methods of evaluation, etc. 

And, I am sure there are still others. Thus, student views can be heard and integrated into courses and teaching practices. And, they can be heard directly. What is more, there is a lot of room for faculty and students to work on such things together. For instance, there is a constructive role for student unions in organizing regular faculty discussions (which would include give and take discussion) of academic issues (say, teaching and tenure; responding to learning issues; the merits of the lecture, etc.). Indeed, I'd suggest is a worthwhile collaboration.

Conclusion

Such mechanisms are not what people think about when they say "students want a say" but they provide that say in a direct fashion. In other words: the student speaks directly to me as opposed to filling out a form which someone else interprets (or ignores if you are not part of the "average"). This approach also allows for an opt-in mode of engagement. No one has to participate. It is an option if you want it and, therefore, respects your right to choose how, why, and when you want to engage teaching and learning issues. Moreover, such an approach illustrates confidence and trust in each other. Rather than threatening Faculty Member X, I do the opposite: I show I respect him or her and trust them to do their job by responding constructively to dialogue. They also establish dialogue -- direct communication or conversation -- as a normal operating procedure. I think this might take a short time to "catch on" but I also think that its effect are good in even the shorter run. Punitive approaches, by contrast, infantilize faculty and treat them as lazy individuals who don't care about their jobs and must be forced into line. In other words: they send exactly the wrong message for a community of scholars and this is ... after all, what we are suppose to be, no?

Having a Say: Student Evaluations of Teaching and Student Voice

So, if Student Evaluations of Teaching (SETs) do not accomplish their goals, what should we do? This is an important question and one that requires a sophisticated discussion and response because, in my view, too often the response is part of the problem. Many people to whom I have spoken, for instance, will say "Yeah, I know SETs aren't perfect but it is the only tool we have so we have to use it." This is the equivalent of trying to turn a screw with a toothpick. The toothpick may be the only tool you have but using it to turn a screw will just waste your time and will never accomplish your goal. In this case, using the only tool you have is counterproductive, particularly when there is a better and straightforward answer: get the right tool.

The other thing some people do when I make statements like the above is to think that I am suggesting that we ditch SETs completely. What I am suggesting is that we need to engage in a complicated procedure that recognizes that listening to student voices cannot be reduced to a fill-in-this survey, as if that could convey student voices. I am also suggesting that if we want to listen to students -- and I am arguing we should -- then we should actually listen to students; not someone who is not a student and who is not answerable to students who is speaking in their name.

So, how do we do this? Despite the complexity of the final result, I don't think that hearing student voices is "rocket science". Let me divide my comments into two sections: practical tips and broader engagements. Today I'll address what I'll call practical tips, although, as I want to explain they are not really practical. Instead, there is a philosophy here to which it is important to pay attention because if we don't, we are missing what is, in fact, a key opportunity to acknowledge and foster communication between faculty and students.

Practical Tips

The first thing that we need to recognize is that faculty, in fact, listen to students all the time.  The vast majority of the faculty I know (which is now a large number at a broad range of universities) spend a large part of their time listening to students. They meet students for consultation and extra help, raise issues in class, talk after class and before. The vast majority of faculty to whom I speak go through their SETs (both the quantitative and qualitative comments) in intricate detail to tease out information. Said differently, faculty are already listening. The idea, then, that we need some sort of mechanism -- perhaps even a coercive mechanism -- to get faculty to listen to students is, in fact, wrong, plain and simple. Everyone has, I am sure, their story of the arrogant prof who thought they were the cat's meow. I have at least one. But, we should no more let this anecdotal evidence stand in for the majority than we would in any other case. I did not teach last year because I was on sabbatical but the year before I spent about 8 hours a week meeting with students. Think about that. If I worked a regular 40 hour week ... that would be 20% of my work time (before I'd given a single lecture, ran a single seminar or tutorial, organized a single extra help session, marked a single paper, etc.) that was devoted to just talking to students on subjects that they pick.

The second thing we need to recognize is this: faculty are already listening in another way. The fact that SETs are not the be-all-and-end-all of student voice does not mean that they cannot play a useful role; the vast majority of faculty to whom I speak believe this is so and devote considerable attention to reviewing SETs results. We need to consider that role and encourage faculty to make use of it. For instance, with regard to the quantitative evaluations ... are there any red flags? Are a given minority group of students raising consistent concerns about any particular issues, say fairness in evaluation? These are, then, often merged with qualitative comments to see if there are issues that need to be addressed. Many faculty now ask their students to complete voluntary mid-term evaluations as well that are designed to highlight and respond to any emerging problems in a course.

The upshot of points one and two -- to repeat myself -- is this: the idea that faculty are not listening to student voices and that we need to find some mechanism and potentially force faculty to adopt it -- or dock people's pay if they don't meet a certain standard -- is wrong. As a faculty member, it is in, in fact, in my interest to have an on-going conversation with my students.  While any one student may or may not see the results of that conversation (more on this  in a later blog), it does not mean that I did not listen, think about what you had to say, weigh it against other views articulated by students, consider the fairness of your ideas, etc. I do this -- as do other faculty -- not because I am forced but because I want to be the best instructor I can be. If you tell me you did not understand lecture X ... well, heck, it is in my interest to help you understand the idea and modify the way I explained X or I don't meet my own objectives (which is having people learn things).

The Philosophy of Dialogue

In one of my previous blogs, I noted that people like numbers. They like the quantitative assessment of faculty because it is easy to understand and relatively easy to draw conclusions. You can also make comparisons. Nurse got 4.0; Other Faculty member got 4.2; ergo Other Faculty Member must be better.  You can set a mark ... and look down to an average and see if that person has met that mark. If they have ... good, they are good. If not, they need improvement. But, as I said before, that is not really listening to students. It carries with it no inherent response to concerns, for example, ignores minority groups as if their voices did not count, and relies on a mediated articulation in which someone else -- someone who is not a student and has no connection to students (in addition, someone who has not been in the course) -- determines what students mean by the numbers they filled in. No student is asked what they meant, nor are interpretations confirmed with students, nor will the person making the determination ever have to explain the character and nature of their determination to students. Hence, numbers are not a good way to solicit student voices.

Direct conversation between faculty member and student is better. In fact, even having Faculty Members interpret and act on SETs is better then relying on the interpretation of a third party because one mediating step in the chain of communication is removed. Think about that for a second. If you want to communicate with me  -- you want to tell me something I am doing right or something I am doing wrong -- what is the most effective way for you to do that. I suspect, at this point, most of you said "tell you directly" and that would be right. A second way would be to fill out a survey.  It would be one step removed from direct communication but as long as I look over the  results and treat them seriously ... we are not in bad shape. We start to lose efficiency -- and voice -- however, if we start to say "I'll let someone else speak for me." We lose accuracy if we start to say "I'll let someone else whom I don't know and who will not check with me and who was not in the class with me -- that is, does not share the experiences about which I wish to communicate speak for me."

Thus, the idea that faculty listen to students in class, in office hours, in extra help sessions, by talking before and after class, by writing emails (I answer a minimum of two emails from students each day during the school year and often five or six, sometimes up to twenty), is not some sort of way of deflecting the problem. It represents a real and concerted mode of engagement that is intended to provide the most direct form of communication possible and, I might add, the most responsive.

What do I mean by responsive? This: you come and talk to me, saying something like "Professor Nurse, I did not understand thing X." I then say "OK, what about X did you not understand?" You say "I got the first part about 1 and 2, but after that I lost track of what was going on and missed 3 and 4." I say, "fair enough, it was a tough subject. Let's go over this, let me very quickly recap 1 and 2 and then explain 3 and 4 and see if that clears things up." You see what I mean? Direct communication addressed the problem that this student was having and addressed it then and there. Now, imagine the alternative: mediated communication based on SETs. The student fills out their SET at the end of the semester remember that they did not understand X from earlier in the term and scores a low number on comprehensibility (say, a 2). This then goes into the pile (see my previous blog for who aggregates and averages distort voices) and is averaged with the rest of the class, the vast majority of whom did understand my discussion of X (although why they understood -- because of my explanation or some other cause, like they just happened to know it -- is never addressed or even considered through this type of quantitative SETs) and so my final ranking on this measures is OK (say, 3.94); not wonderful but not low enough for my dean to raise any real concerns.

What has happened to the student's voice? It is gone ... lost in the mist of the average. Has the student's problem been addressed; that is: do they understand X any better than before. Well, no. Not in this example. Thus, we have failed on two counts. We have not heard the student and we have not addressed their problem. Now, I ask you ... if you were interested in hearing student voices and addressing problems ... which method of communication would you choose?

Conclusions

I get it. People like numbers. I watched Moneyball. Numbers can and are useful. No one is debating that. What we are talking about here -- what I am writing about -- is student voice. What is the most effective way to ensure that student have, as a friend of mine recently put it, "their say." The answer I am giving here might not be popular because it is running against a received wisdom that says only though surveys can students be heard. I am trying to say that I don't see that as the case. I am trying to say that I think most faculty are already listening to students and trying to address student concerns. Moreover, direct communication has merits. Its not just hippy, feel-good, peace and love stuff. It has appreciable merits in that it provides a way for each voice to be heard, it operates on an opt-in basis (those who want to use it, can; those who don't, can ignore it ... the choice remains with the student as to whether or not they want their voice heard), and it is responsive in a way that SETs are not. Moreover, this approach -- direct communication -- has the further merit of not relying on my good graces. I might like to pay attention to students. I might be a nice guy and like to listen to people but that is not the point. If I don't listen to students, I compromise my ability to do my job well. Students, in other words, don't learn what they are supposed to learn and hence my own performance as the guy at the front of the room is not what it could be.

I am not saying this argument is perfect. There is more that I will add to it in a future blog. But, I am saying that we have the basis, already in place of a very good, very effective way of ensuring that student voices are heard.

Student Evaluations of Teaching ... or, not again ... seriously?

I had meant to get back to this subject sooner than this but I have been ill. In my last blog -- the one right before this one -- I tried to explain why I think much of the current discussion, at least the discussion that we are having at Mount Allison, with regard to Student Evaluations of Teaching (SETs) is both misguided and, in fact, moving us in the wrong direction. Rather than promoting good teaching, it seems to be driven by an abstract series of propositions that use a certain discourse -- students want a voice -- even while they will deny students the very voice that they want. In other words, if we can agree on first principles (that students should have a voice in the way the university is run and that we want to improve post-secondary teaching), we can almost certainly find good and productive ways to meet both objectives.

This is important to say because when I raise concerns about Student Evaluations of Teaching, one of the first things people say to me is "so you don't want students to have a voice." To the contrary, the problem is that SETs won't really do that. Why? There are two questions to answer in this regard: how do they work now? how should they work?

Right now, most places that use SETs, have them work like this. At the end of the semester, students fill out a form that evaluates faculty on the basis of various criteria on some scale. We use a 1 (strongly disagree) to 5 (strongly agree) scale but I've seen models that have used other scales over the years. The precise scale, of course, does not really matter too much. The issues are that it is (a) accurate and (b) used consistently over time. After students complete these evaluations, they are sent to the dean who then goes over them and, on this basis and that of other criteria, makes an assessment of whether or not the faculty member is satisfactorily doing their job. There is a big asterisk here for Mount A: submission of SETs to the dean at the end of the academic year is voluntary. My best guess is that over 80% -- I've heard one person in the know say 90% -- submit them. (I'll give you what I do in another blog).

What has happened? Well, no one would argue that SETs alone -- the supposed way student voice is integrated into the teaching process -- is *the* evaluation of teaching. All the guides (including that of our own Purdy Crawford Teaching Centre) say that SETs should not be the sole criteria of the evaluation of teaching, that interpretation of SETs is needed (a point I made in my last blog) and that we need to assess change over time (is a teacher getting better, worse, staying the same; are they doing some things well and not others, etc.). So, using this range of criteria, the dean factors in SET numbers and offers his (our deans are all male) assessment.

There are a couple of things here to note:

1. We are not clear, here at Mount A at least, what other criteria are actually being used. I am going to suggest that there are some criteria that should be used but these are not specified and, in my experience, confuse even faculty members.

2. The student voice has been lost. Students make both qualitative and quantitative comments. In my experience, the quantitative ones are the ones that people want to read because they are the easiest to seemingly understand. I don't think they are for reasons I explained in my last blog but I do believe that this idea -- that numbers are easier to understand than written comments -- is a common fallacy. By looking at numbers only (which is what deans do -- not claiming this is a fault, merely trying to be empirically accurate), not a single student voice is actually heard. Instead, they are merged together into an aggregate and that aggregate stands for student views.

I'll leave off on the first point I made for now and address it in another blog in order to concentrate on the second. Let me give an example to illustrate my point (hypothetical).

Example: one question on our SETs at Mount A (one to which I pay a great deal of attention) asks students whether or not the method of evaluation was fair and appropriate. Imagine my responses were as follows:

Number of students                             Ranking

3                                                             1
2                                                             2
2                                                             2
18                                                           4
8                                                             5

Admittedly these are not bad numbers I just made up and that might allow you to instantly see the logic of what I am doing. My average score on this question would have been 3.94, fairly  close to a 4 (agree) on our scale. So close, in fact, that no dean would ever raise a concern about it. But 7 students did not agree. Two were neutral (3) and five (1 + 2) disagreed or strongly disagreed. Hence, slightly more than 20% of the class had problems -- of one sort or another -- with the mode of evaluation. But, in this example, this 20% (one out of five) is not heard. Now, think about this. If one out of five of anything had problems with what you were doing ... that would be cause for some concern would it not? If you owned a restaurant and one out of five customers thought your food sucked ... you'd be concerned, right? But, that ain't gonna happen the way our system (or, as far as I can tell any system of SETs evaluation) is set up.  Thus, the very mechanism that is intended to provide student voice is actually depriving one of five students of their voice.

But, I think the problems go deeper than this. Not only have 20% of students been stripped of their voices but the other students have been as well. They have to trust that the dean will reflect their views. In other words, they don't get their own voice. Their voice is little more than the equivalent of a quality control telephone survey. Ever done one of those? Were you satisfied that you, as a person, were actually listened to? This is my point: SETs don't provide student voices but mediated student voices that rely on another person to articulate their views for them. What is more, that person is (a) not a student, (b) a single individual, (c) has no student oversight himself, (d) does not have to justify his assessment to anyone or explain the criteria he used. In short, this is a mess not because the people involved are inherently bad people (they are not) but because no one would ever set up a method to hear student voices in this fashion if they actually wanted to hear student voices and ensure that the system was operating in a fair, responsive, and reasonable way. For instance, and just to make my point, if you believed that students should have their "say" would you design a system that gave their say to someone else? If you were interested in ensuring that a large number of people were heard, would you intentionally reduce their voices to a single voice who was not one of their number?

This is also a problem because if leaves a series of important questions unanswered which students, in my experience, use their SETs to answer. For example: what does a rating of four (or, any other number) mean? What does a three (neutral) mean? (Think about that ... an interesting question. Its neither an endorsement nor a criticism ... so how do we read it?)  Do we have any criteria that differentiate one number ranking from another; that is, what is the difference between a 4 and a 5 or between strongly agreeing and agreement? What does it mean to ask -- in the same question -- that a mode of evaluation is "fair" and "appropriate"? Could it be fair but not appropriate? In other words, what, for students, constitutes "fair"? What constitutes "appropriate"?

These are not idle questions, as I have said, for two reasons. First, they are not idle because they are what we are trying to accomplish. We want education to be fair. We want assessment to be appropriate. Moreover, I suspect that we could, relatively easily, agree on a set of criteria that met the standard of fairness and appropriateness if we would just talk about it.  In other words, rather than trying to find ways to implement rules of evaluation, why not hear student voices by asking them to comment on questions? Why not enter into a conversation on the subject?

Second, because my students often use their qualitative comments to augment and explain their quantitative assessments. Some will focus on specific lectures; some will have concerns about a specific assignment. Others will question the merits of specific assigned readings or suggest other readings that might improve. In other words -- and without putting words into my students' mouths -- they treat their qualitative comments as something that the way the assessment of the  quantitative comments does not: as part of an on-going conversation about a specific course and the way it was taught and developed. They are offering to have the very conversation that they want to have. The problem, then, lies not with them but with the way SETs are used. It refuses engagement in the very conversation that we can and should have.

Let me sum up. I don't have problems with SETs. I use them and use them all the time. They are useful and most faculty devote a great deal of attention to them. The problem I have with them is proposals to implement them as an evaluative component of faculty work under the guise of hearing student voices. What I've tried to explain -- and you can let me know whether or not I've succeeded -- is that the process of using them as a mode of faculty evaluation necessarily fails to meet the goals that are being set for them. Rather than facilitating the voice of students, the use of SETs as tool of evaluation deprives students of their voices and transfers it to (at Mount Allison) an unaccountable administrator (who never has to report to any student or student body about any decision or evaluation they have made and never do) while short circuiting the very conversations that students seem to want to have about post-secondary education. I said that no one would intentionally design a system that worked in this way or, if they did, most of the rest of us would wonder about it. For example, if you wanted your car repaired, you would not knowingly do things that prevented it from being fixed.  This is what SETs do in terms of hearing student voices. And, that is too bad because those voices are important and should be heard.

Student Evaluations of Teaching .... Why?

I had a long chat with a co-worker the other night and it was a good chat. I'll leave her name out of this since it is not fair to tar her with my thoughts, particularly since she may not share some or even all of them. We were talking about Student Evaluations of Teaching (SETs), those course-end evaluations that students at universities complete. Many universities require them; some requirement that they be included in packages submitted for tenure or promotion. Mount Allison does not, although we are required to complete a self-report and be assessed by respective deans every two years and most people include SETs in their reports. There has been remarkable controversy lately over SETs that, in my view, is frankly misplaced. Why? That is what I want to explain in this email, but I'll tip my hand at the outset: mandatory SETs that are assessed by deans will not accomplish their goals. Hence, they become a hoop through which people jump rather than a meaningful part of work in the post-secondary environment. What is more, we have alternatives. People who are looking to meet certain goals can use these alternatives to a better end.  SETs are not a waste of time but if you are looking to (a) improve teaching, (b) provide higher accountability, (c) ensure a student voice in the evaluation/assessment of profs ... SETs will meet none of these objectives. Hence, promoting them is not a good idea when we can more profitably direct our attention to mechanisms that will work.

One of the things about SETs that surprises me is how the debate -- at Mount A at least -- has become frozen in time. It has, in short, not kept up with the scholarship of teaching and learning as it pertains to SETs. Once, when I first started out in this gig (let's say 15-20 years ago), there was a vigorous debate about the accuracy of SETs. Do the scores students accord to profs accurately reflect teaching ability; that is a competence and so can be used as a tool of evaluation. This debate is old and settled. Anyone who keeps debating it is, frankly, missing out on a lot of scholarship. The scholarship on SETs falls into three camps. There are ardent defenders of their accuracy. There are ardent rejectors of their accuracy. Both are small groups. In other words, just about no one takes one side or the other in an extreme form. Instead, most people fall into a third camp that we could call "yes ... but ...." The basic premise of this perspective -- which represents the mainstream of SETs assessment the following:

• SETs are valuable tools that cannot be taken in isolation. They must be combined with other forms of assessment. In other words, to use them by themselves as your sole or even dominant teaching assessment tool would be wrong and would create inaccuracies. It would be like using a year-end poll instead of having an on-going democracy. 
• SETs cannot be assessed outside of a chronological framework; that is: one needs a time span, ideally several years, for SETs to make sense. In other words, the trend is more important than any single individual number.  This point should not surprise us because it is a standard point of statistical analysis. Ask Michael Adams if you don't believe me. It is the pattern into which the numbers fit that is important.
• SETs require interpretation. You cannot just look down a range of numbers, see that one prof has 4.1/5 on average in their evaluation (we use a standard 5 point scale at Mount A)  and determine that that person is better than someone who had a 3.9/5. Why? We have no longitudinal data, we don't know how many students each of these profs taught, we don't know their stage of careers, we don't know the courses, we don't know how they provided extra help, we don't know whether or not the students in their courses has prerequisites. In other words, all we have is a number outside of context. What if the first person had six students and taught only one class of advanced, motivated students who happened to collectively be a really nice and deferential bunch who assumed that the prof knew what they were doing? What if the other prof laboured for hours with hundreds of students in core courses that students hated taking but were required for their degrees, providing endless hours of extra help, marking, say, hundreds of papers (I'm not exaggerating. I mark hundreds -- seriously -- of papers each year.) Would we think that a situation where the first prof's numbers could have changed dramatically if one student had changed their SETs as fair and accurate? Don't believe me ... take some time and do the math yourself.  What is more, however, we believe that everything requires interpretation. Geographers interpret space; biologists interpret living things, historians interpret the past, sociologists ... society. After we teach this to our students ... why would we suddenly believe that it would not apply to our jobs? After we spend hours teaching students statistical analysis, political inquiry, quantitative methods .... why would we ditch it as if everything that we taught were unimportant? 

Put together, no one who has serious studied this issue says "yeah, go ahead, look at one number and that will tell you whether person X is good or bad at their job." My point is this: in having a debate about whether or not we should have SETs and whether or not deans should look at numbers to assess faculty, we are missing an opportunity to leave an old debate in the past and have a new conversation that serves all of us (students, faculty, administrators, the institution) better. To get to this conversation we need to ask this question: why have SETs? 

I've encountered two answers to this question, one of which I accept and one I reject. The one I reject is this: we assess everything so why not use this tool to assess teaching. I reject that argument for the following reasons:

•  We don't actually assess everything. Most professions are certified, go through probation and then are approved to work in that profession. We do this with faculty; it is called tenure. After certification, though, we don't assess most professions regularly. We don't assess the ice cream we eat; the plumber who  comes to our door; the mechanic to which we take our car. The idea that we assess everything is, therefore, wrong.
• Why would we use a method of assessment that might be inaccurate? Even if we agree that assessment is good and useful, we want that assessment to be accurate. Simply implementing a SETs-based assessment policy will, therefore, do nothing but say we have a policy implemented, the accuracy of which we cannot guarantee. Does that inspire your confidence? Imagine using that line for a different profession. We have imposed a method of assessment on our fire department but we cannot guarantee that it is actually accurate and so we actually tell you whether or not they'll put out the fire if you call. Does that instil faith in the fire department? What about a medical professional? Yes, we've assessed your doctor and they passed but we can't actually say whether or not this assessment is accurate so if you are sick you might or might not get the right advise. Hmmm ...

Sometimes I hear people say "we  have to do something." Yes, we do, but let's do something that meets our goals; not something that we do just to do because that is a waste of time and resources. 

The correct answer to that question is that we need to assess teaching to improve teaching. There is absolutely no other reason to do so. (Pause and think about that if you disagree with me and you'll actually see that that statement is accurate. If we are not assessing teaching to improve it, what other possible end could we have in mind?) Now I am on the same wavelength. And, now you can see why accuracy, change over time, and context are so important. Without these things, we run the risk of drawing wrong conclusions and making changes that would actually hamper good teaching as opposed to enhance it. For instance, a person who is getting better at teaching every year might be someone who is worthy of commendation even if their numbers are not quite as good as someone else's who has basically flatlined. A person who innovates -- say, with regard to technology in the classroom, experiential learning, mentoring -- might run the risk of a horrible failure but we might all be glad that she ran that risk because it helps us. Should we condemn her for that? 

What I would suggest is that as part of this new conversation that we consider our objectives and think about the range of ways we can meet them. We approach our own developments on an opt-in basis, knowing that there will be people who will not opt-in right away, potentially never. I never mind it if someone says to me "you know, Andrew, I'm going to see if this works before I sign on." So, if someone is skeptical and wants to wait until we have evidence of success ... well, heck, that strikes me as just plain good sense. 

We could begin by looking at what SETs are used for in the scholarship of teaching and learning as one first step. We might discover that there is a whole field of research and we don't need to reinvent the wheel. We might encourage team assessments, student open fora on teaching, open fora on conceptions of what constitutes good teaching, or even what constitutes a class, experiential learning (done right), different forms of extra help, encouraging faculty involvement in the scholarship of teaching and learning, encouraging faculty involvement in teaching conferences, developing new and flexible forms of teaching that meet student needs. In other words, there is a whole bunch of stuff that we can and we should do to improve teaching ... why not start doing it. SETs and some mode of thinking about those might be useful but there is so much more that we can do to encourage student voices, provide fora for them so they can be heard, and get encourage improvements in the quality of education. Yet, if we spend all our time discussing SETs and their accuracy and whether or not a dean should look at them ... well, none of the rest of this stuff -- from student voices to something else -- gets done. 

I might blog more on this in the future but one last thing ... there is a role for deans in this and I honestly don't understand why it is not being done right now. Why not encourage good teaching? Why not go to workshops and teaching conferences? Why not check in with faculty as the term goes along? One of the big problems with using SETs to evaluate teaching is this. Imagine, for instance, and for the sake of argument, that it works and it does help us separate a good teacher from a bad one. If that were true ... it does that only after the fact. In other words, and assuming the accuracy of SETs, we discover that Professor X (who got a bad evaluation) was not educating his students only after the course was done. By this time, the prof -- allowing a bad prof -- might have cost students their scholarship, or forced them to have to repeat courses, or even driven them from the subject because he was so bad. SETs, in other words, have a built in flaw: they are re-active. We need to be pro-active in developing good teaching. And, I might say, I am on board for anyone who wants to find ways of doing that.