Student Evaluations of Teaching ... Having a Real Say Part I

In my last couple of blogs I've tried to address so of the issues surrounding Student Evaluations of Teaching (SETs) at the post-secondary level. I've tried to argue several points:

1. SETs are useful tools that can help instructors improve the quality of their teaching
2. Mandatory SETs assessed by a dean with punitive repercussions don't allow student voices to be heard but simply, in fact, transfer author away from students to administrators (who are not responsible to students)
3. There are other problems with SETs that serve to erase student voices, particularly those who hold minority views
4. Most faculty do already listen to students and they do so in a variety of ways. It is, in fact, one of the more time-consuming (and, I might say, rightly so) things we do in terms of teaching.

Let's take this discussion a step further and consider other ways of listening to students. One of the things that attracts people to SETs, I've said, is their seeming simplicity and neutrality. They provide numbers that seem simple (even though I've tried to indicate that they aren't) to interpret. There is some evidence that people are starting to understand that SETs do not meet their goals. That is: they are not having the effect that was desired when they were implemented. That effect, I believe, was to improve the quality of post-secondary teaching and ensure a student voice in the operation of universities. Both of these aims are more than "fair enough." They are legitimate. Yet, if the tool through which one tries to meet legitimate objectives fails ... should you continue to use that tool?

Let me try to address this question by looking at where it comes from (and, why it is important) and how we might better proceed. I'll argue for a multiple pronged process that involves a reconsideration of first principles as well as a willingness to actually communicate as a mechanism of ensuring  that students have a say. Before I do this, however, let me draw what is an important distinction, particularly for the argument I am about to make below.

The distinction I want to draw is between processes that are intended to ensure communication and those that are coercive and punitive. Communicative processes are those I discussed in my last blog. People's voices are heard through a process of talking to each other. Coercive punitive approaches are those that use a threat or an implied threat in an effort to achieve certain results. For instance, if my boss wants to improve how I do some aspect of my work he -- and, my boss is a he -- has choices in terms of how he goes about it. He could (a) talk to me and work through whatever the problem is (perhaps along the way, he discovers that the problem is, for instance, idiosyncratic and has occurred because of an illness on my part or some other temporary problem) or (b) threaten me. "Nurse, if you do not meet standard X on SETs, I will rate you as 'unsatisfactory' for teaching and this could have implications for your pay." We don't go quite that far at Mount A, but there are people who want to. There are people who believe, in fact, that my dean should be able to dock my pay, in effect, then and there, if I don't meet whatever standard he feels I should.

In the discussion below, I'll call (b) coercive/punitive or words to that effect. I have colleagues who will protest that characterization or use a different language to explain it but if you think about it for a minute, you'll see my characterization is accurate. What is going on here is that coercion (or, the threat of punishment) is being used as a motivational tool. If you find the name disturbing ... that might say more about your views on coercion than my characterization of it.  My view, to tip my hat up front, is that people are way, way, way too fast to use coercion as a tool. There are other, better, and more mature ways of approaching problems and issues and goals, as I will sketch out below.

The Issue

This is an important question because of the process through which we have gone  at Mount Allison, which is, I think, pretty normal. When I first started at Mount Allison SETs were voluntary: some faculty had students complete them; others did not. There was a form one could use provided, I think, by the student union. We progressed from that to mandatory SETs and from that to a debate about two further points (a) should mandatory SETs be evaluated and pay docked if a dean determines a faculty member did not meet a standards -- that is should we adopt a punitive approach as an institutional policy --  and (b) should we have more than one set of SETs during the semester? In other words, instead of having students complete SETs at the end of the semester, should they also complete  them at the halfway mark of a course? Some faculty do this voluntarily, but should they also be made mandatory? I have heard some people suggesting that these might, in fact, be completed several times during a semester. In other words, students would be filling out surveys in the manner of a Nanos Research rolling poll.

You can see what has gone on. We have moved through a series of stages -- voluntary, mandatory, punitive assessment and multiple evaluations. The reason for this progression, I think, is not just that someone somewhere loves assessment and imposing penalties. I think this transition has occurred because people were not happy with the results. It is logical to change what we are doing if we are not getting desired results. The problem, in this case, is not that we are not getting the results that we desire (indeed, I'd argue that we actually are). The problem is that we are vague as to what the results actually are. What does it mean, for instance, to have "a say?" How can we measure improvements in education? Surveys and numbers seem like a self-evident way, but are they?

I am not dissing SETs. I think they have a role and I've said this a number of times, but they are not the only tool we should use nor inherently the most effective. I said in my last blog that I thought direct communication was the most efficient way to address problems of teaching and learning. "Prof Nurse, I did not understand X. Could you explain it?" This is efficient because it helps the student learn as opposed to punishing me after the student has *not* learnt. Think about that. Punishing me after the fact serves no effect. I get my "comeuppance" but the student still has not learnt. From an educational perspective, what good is that?

The issue, then, in my view, is not to take a tool that is not producing desired results and make more use of that tool. This is what people who want to ramp up SETs and increase their coercive and punitive character are actually saying. In a perhaps more articulate language, they are arguing this: the tools we have are not giving us our desired results so let's make greater use of those tools. Does this make sense to you? I think the issue is something different. The issue is to re-assess our beginning point and have a discussion about what our goals are and what they mean in practical terms.

For example, students want to heard. What does that mean? Do you want to be heard for the sake of being heard? That might be fair enough but I actually doubt it is the case. I suspect that students want to be heard because they are interested parties in education, have a contribution to make to pedagogy as well as specific class content, and want to reward -- not just punish -- those faculty who they feel have really helped them out on a variety of levels.  In addition, I think some simply want to be part of a conversation about the future and development of post-secondary education. That is: they want to play a role in improving it.  The point I've made before holds true again: none of these goals can be met through more punitive uses of SETs evaluations by deans or other supervisors.

Student Perspectives

I can't articulate a student perspective, because I am not a student but I make the above statement on the basis of extensive conversations with students over the years. Very, very, very few students I have encountered actually want punitive and coercive measures against faculty. I have talked to one or two who believe that there should be coercion and threats in pedagogy. But, the vast, vast majority are well-meaning, well-intentioned people who are sympathetic to faculty and the different aspects of their jobs. I've directly asked classes I've taught these types of questions and always walked away with favourable impressions of the discussion that ensued.  Indeed, what impressed me was the difference between what students say and what people who speak in students' names say.

For instance, I've had people (say, faculty or administrators) speaking for students say "students want a voice and that means that your dean should be able to dock your pay." They have not used exactly those words but that is, in effect, what they said. As I said, I've had in 15 or so years teaching only one or two students who have made that case. Most don't want to see my pay docked but want to see education improved. While I've heard faculty and administrators argue that threats are a good way of motivating faculty, I have actually never had a student say that they think threats and coercion are good motivators. Even the very small number who like them stop short of saying that threats will, ipso facto, improve teaching.

This is, then, an odd disjuncture and one that causes me a certain amount of pause. Students seem to be saying one thing, while those who speak in their names seem to be saying something else and the two do not match up.

Models

Because of this, we need to re-evaluate the model by which student voices are heard. By returning to first principles, by grappling with out objectives and with student aims, we can, I think, build much more accurately designed mechanisms for student voices to be heard and for us to make the most out of post-secondary education. Moreover, I think this can be done without coercion and punitive evaluations. I think, in fact, the overall results will be better.

This blog is already getting too long so let's leave off the issue of teaching and learning to keep focused on student voice. How can we hear student voices?  In addition to the significant amount of talking that already goes on between faculty and students, we, in Canadian Studies at Mount Allison, have done other things over the years.

First, we try to directly connect with the student club. Most department, programmes, units have a student club of one type or another. These might be made up of particularly motivated students, but ... good. There is no reason why we should ignore someone's voice simply because they are motivated, is there?

Second, we've had students complete end of degree surveys which call for extended qualitative comment. The idea here is that students finishing their degrees will have a certain perspective on what was good, bad, what helped and what hurt the learning process.

Third, we have student focus groups. We don't do these all the time, but since I've been at Mount A we've run them on academic integrity, the requirements of the honours programme, and to solicit feedback on curricular innovations.

Fourth, in class "academic moments". Usually in 3000 level courses (but these could be done at any level) I schedule a series of academic discussions (no more than five minutes unless there is a good point being made or someone who really wants to speak to an issue) about teaching and learning. Topics could include such things as the role of SETs in teaching, different modes of evaluation, responses to integrity problems, course formats, etc. These are, I suppose obviously, done on an opt-in basis. No one is forced to talk if they do not want to.

Some other ideas that could be used include:

• holding open fora on key matters of curriculum or teaching and learning
• regular (ever week, every month, once per semester, etc., whatever people want) faculty discussions with students about key issues, say the role of research in education, methods of evaluation, etc. 

And, I am sure there are still others. Thus, student views can be heard and integrated into courses and teaching practices. And, they can be heard directly. What is more, there is a lot of room for faculty and students to work on such things together. For instance, there is a constructive role for student unions in organizing regular faculty discussions (which would include give and take discussion) of academic issues (say, teaching and tenure; responding to learning issues; the merits of the lecture, etc.). Indeed, I'd suggest is a worthwhile collaboration.

Conclusion

Such mechanisms are not what people think about when they say "students want a say" but they provide that say in a direct fashion. In other words: the student speaks directly to me as opposed to filling out a form which someone else interprets (or ignores if you are not part of the "average"). This approach also allows for an opt-in mode of engagement. No one has to participate. It is an option if you want it and, therefore, respects your right to choose how, why, and when you want to engage teaching and learning issues. Moreover, such an approach illustrates confidence and trust in each other. Rather than threatening Faculty Member X, I do the opposite: I show I respect him or her and trust them to do their job by responding constructively to dialogue. They also establish dialogue -- direct communication or conversation -- as a normal operating procedure. I think this might take a short time to "catch on" but I also think that its effect are good in even the shorter run. Punitive approaches, by contrast, infantilize faculty and treat them as lazy individuals who don't care about their jobs and must be forced into line. In other words: they send exactly the wrong message for a community of scholars and this is ... after all, what we are suppose to be, no?

Having a Say: Student Evaluations of Teaching and Student Voice

So, if Student Evaluations of Teaching (SETs) do not accomplish their goals, what should we do? This is an important question and one that requires a sophisticated discussion and response because, in my view, too often the response is part of the problem. Many people to whom I have spoken, for instance, will say "Yeah, I know SETs aren't perfect but it is the only tool we have so we have to use it." This is the equivalent of trying to turn a screw with a toothpick. The toothpick may be the only tool you have but using it to turn a screw will just waste your time and will never accomplish your goal. In this case, using the only tool you have is counterproductive, particularly when there is a better and straightforward answer: get the right tool.

The other thing some people do when I make statements like the above is to think that I am suggesting that we ditch SETs completely. What I am suggesting is that we need to engage in a complicated procedure that recognizes that listening to student voices cannot be reduced to a fill-in-this survey, as if that could convey student voices. I am also suggesting that if we want to listen to students -- and I am arguing we should -- then we should actually listen to students; not someone who is not a student and who is not answerable to students who is speaking in their name.

So, how do we do this? Despite the complexity of the final result, I don't think that hearing student voices is "rocket science". Let me divide my comments into two sections: practical tips and broader engagements. Today I'll address what I'll call practical tips, although, as I want to explain they are not really practical. Instead, there is a philosophy here to which it is important to pay attention because if we don't, we are missing what is, in fact, a key opportunity to acknowledge and foster communication between faculty and students.

Practical Tips

The first thing that we need to recognize is that faculty, in fact, listen to students all the time.  The vast majority of the faculty I know (which is now a large number at a broad range of universities) spend a large part of their time listening to students. They meet students for consultation and extra help, raise issues in class, talk after class and before. The vast majority of faculty to whom I speak go through their SETs (both the quantitative and qualitative comments) in intricate detail to tease out information. Said differently, faculty are already listening. The idea, then, that we need some sort of mechanism -- perhaps even a coercive mechanism -- to get faculty to listen to students is, in fact, wrong, plain and simple. Everyone has, I am sure, their story of the arrogant prof who thought they were the cat's meow. I have at least one. But, we should no more let this anecdotal evidence stand in for the majority than we would in any other case. I did not teach last year because I was on sabbatical but the year before I spent about 8 hours a week meeting with students. Think about that. If I worked a regular 40 hour week ... that would be 20% of my work time (before I'd given a single lecture, ran a single seminar or tutorial, organized a single extra help session, marked a single paper, etc.) that was devoted to just talking to students on subjects that they pick.

The second thing we need to recognize is this: faculty are already listening in another way. The fact that SETs are not the be-all-and-end-all of student voice does not mean that they cannot play a useful role; the vast majority of faculty to whom I speak believe this is so and devote considerable attention to reviewing SETs results. We need to consider that role and encourage faculty to make use of it. For instance, with regard to the quantitative evaluations ... are there any red flags? Are a given minority group of students raising consistent concerns about any particular issues, say fairness in evaluation? These are, then, often merged with qualitative comments to see if there are issues that need to be addressed. Many faculty now ask their students to complete voluntary mid-term evaluations as well that are designed to highlight and respond to any emerging problems in a course.

The upshot of points one and two -- to repeat myself -- is this: the idea that faculty are not listening to student voices and that we need to find some mechanism and potentially force faculty to adopt it -- or dock people's pay if they don't meet a certain standard -- is wrong. As a faculty member, it is in, in fact, in my interest to have an on-going conversation with my students.  While any one student may or may not see the results of that conversation (more on this  in a later blog), it does not mean that I did not listen, think about what you had to say, weigh it against other views articulated by students, consider the fairness of your ideas, etc. I do this -- as do other faculty -- not because I am forced but because I want to be the best instructor I can be. If you tell me you did not understand lecture X ... well, heck, it is in my interest to help you understand the idea and modify the way I explained X or I don't meet my own objectives (which is having people learn things).

The Philosophy of Dialogue

In one of my previous blogs, I noted that people like numbers. They like the quantitative assessment of faculty because it is easy to understand and relatively easy to draw conclusions. You can also make comparisons. Nurse got 4.0; Other Faculty member got 4.2; ergo Other Faculty Member must be better.  You can set a mark ... and look down to an average and see if that person has met that mark. If they have ... good, they are good. If not, they need improvement. But, as I said before, that is not really listening to students. It carries with it no inherent response to concerns, for example, ignores minority groups as if their voices did not count, and relies on a mediated articulation in which someone else -- someone who is not a student and has no connection to students (in addition, someone who has not been in the course) -- determines what students mean by the numbers they filled in. No student is asked what they meant, nor are interpretations confirmed with students, nor will the person making the determination ever have to explain the character and nature of their determination to students. Hence, numbers are not a good way to solicit student voices.

Direct conversation between faculty member and student is better. In fact, even having Faculty Members interpret and act on SETs is better then relying on the interpretation of a third party because one mediating step in the chain of communication is removed. Think about that for a second. If you want to communicate with me  -- you want to tell me something I am doing right or something I am doing wrong -- what is the most effective way for you to do that. I suspect, at this point, most of you said "tell you directly" and that would be right. A second way would be to fill out a survey.  It would be one step removed from direct communication but as long as I look over the  results and treat them seriously ... we are not in bad shape. We start to lose efficiency -- and voice -- however, if we start to say "I'll let someone else speak for me." We lose accuracy if we start to say "I'll let someone else whom I don't know and who will not check with me and who was not in the class with me -- that is, does not share the experiences about which I wish to communicate speak for me."

Thus, the idea that faculty listen to students in class, in office hours, in extra help sessions, by talking before and after class, by writing emails (I answer a minimum of two emails from students each day during the school year and often five or six, sometimes up to twenty), is not some sort of way of deflecting the problem. It represents a real and concerted mode of engagement that is intended to provide the most direct form of communication possible and, I might add, the most responsive.

What do I mean by responsive? This: you come and talk to me, saying something like "Professor Nurse, I did not understand thing X." I then say "OK, what about X did you not understand?" You say "I got the first part about 1 and 2, but after that I lost track of what was going on and missed 3 and 4." I say, "fair enough, it was a tough subject. Let's go over this, let me very quickly recap 1 and 2 and then explain 3 and 4 and see if that clears things up." You see what I mean? Direct communication addressed the problem that this student was having and addressed it then and there. Now, imagine the alternative: mediated communication based on SETs. The student fills out their SET at the end of the semester remember that they did not understand X from earlier in the term and scores a low number on comprehensibility (say, a 2). This then goes into the pile (see my previous blog for who aggregates and averages distort voices) and is averaged with the rest of the class, the vast majority of whom did understand my discussion of X (although why they understood -- because of my explanation or some other cause, like they just happened to know it -- is never addressed or even considered through this type of quantitative SETs) and so my final ranking on this measures is OK (say, 3.94); not wonderful but not low enough for my dean to raise any real concerns.

What has happened to the student's voice? It is gone ... lost in the mist of the average. Has the student's problem been addressed; that is: do they understand X any better than before. Well, no. Not in this example. Thus, we have failed on two counts. We have not heard the student and we have not addressed their problem. Now, I ask you ... if you were interested in hearing student voices and addressing problems ... which method of communication would you choose?

Conclusions

I get it. People like numbers. I watched Moneyball. Numbers can and are useful. No one is debating that. What we are talking about here -- what I am writing about -- is student voice. What is the most effective way to ensure that student have, as a friend of mine recently put it, "their say." The answer I am giving here might not be popular because it is running against a received wisdom that says only though surveys can students be heard. I am trying to say that I don't see that as the case. I am trying to say that I think most faculty are already listening to students and trying to address student concerns. Moreover, direct communication has merits. Its not just hippy, feel-good, peace and love stuff. It has appreciable merits in that it provides a way for each voice to be heard, it operates on an opt-in basis (those who want to use it, can; those who don't, can ignore it ... the choice remains with the student as to whether or not they want their voice heard), and it is responsive in a way that SETs are not. Moreover, this approach -- direct communication -- has the further merit of not relying on my good graces. I might like to pay attention to students. I might be a nice guy and like to listen to people but that is not the point. If I don't listen to students, I compromise my ability to do my job well. Students, in other words, don't learn what they are supposed to learn and hence my own performance as the guy at the front of the room is not what it could be.

I am not saying this argument is perfect. There is more that I will add to it in a future blog. But, I am saying that we have the basis, already in place of a very good, very effective way of ensuring that student voices are heard.