Student Evaluations of Teaching ... Having a Real Say Part I

In my last couple of blogs I've tried to address so of the issues surrounding Student Evaluations of Teaching (SETs) at the post-secondary level. I've tried to argue several points:

1. SETs are useful tools that can help instructors improve the quality of their teaching
2. Mandatory SETs assessed by a dean with punitive repercussions don't allow student voices to be heard but simply, in fact, transfer author away from students to administrators (who are not responsible to students)
3. There are other problems with SETs that serve to erase student voices, particularly those who hold minority views
4. Most faculty do already listen to students and they do so in a variety of ways. It is, in fact, one of the more time-consuming (and, I might say, rightly so) things we do in terms of teaching.

Let's take this discussion a step further and consider other ways of listening to students. One of the things that attracts people to SETs, I've said, is their seeming simplicity and neutrality. They provide numbers that seem simple (even though I've tried to indicate that they aren't) to interpret. There is some evidence that people are starting to understand that SETs do not meet their goals. That is: they are not having the effect that was desired when they were implemented. That effect, I believe, was to improve the quality of post-secondary teaching and ensure a student voice in the operation of universities. Both of these aims are more than "fair enough." They are legitimate. Yet, if the tool through which one tries to meet legitimate objectives fails ... should you continue to use that tool?

Let me try to address this question by looking at where it comes from (and, why it is important) and how we might better proceed. I'll argue for a multiple pronged process that involves a reconsideration of first principles as well as a willingness to actually communicate as a mechanism of ensuring  that students have a say. Before I do this, however, let me draw what is an important distinction, particularly for the argument I am about to make below.

The distinction I want to draw is between processes that are intended to ensure communication and those that are coercive and punitive. Communicative processes are those I discussed in my last blog. People's voices are heard through a process of talking to each other. Coercive punitive approaches are those that use a threat or an implied threat in an effort to achieve certain results. For instance, if my boss wants to improve how I do some aspect of my work he -- and, my boss is a he -- has choices in terms of how he goes about it. He could (a) talk to me and work through whatever the problem is (perhaps along the way, he discovers that the problem is, for instance, idiosyncratic and has occurred because of an illness on my part or some other temporary problem) or (b) threaten me. "Nurse, if you do not meet standard X on SETs, I will rate you as 'unsatisfactory' for teaching and this could have implications for your pay." We don't go quite that far at Mount A, but there are people who want to. There are people who believe, in fact, that my dean should be able to dock my pay, in effect, then and there, if I don't meet whatever standard he feels I should.

In the discussion below, I'll call (b) coercive/punitive or words to that effect. I have colleagues who will protest that characterization or use a different language to explain it but if you think about it for a minute, you'll see my characterization is accurate. What is going on here is that coercion (or, the threat of punishment) is being used as a motivational tool. If you find the name disturbing ... that might say more about your views on coercion than my characterization of it.  My view, to tip my hat up front, is that people are way, way, way too fast to use coercion as a tool. There are other, better, and more mature ways of approaching problems and issues and goals, as I will sketch out below.

The Issue

This is an important question because of the process through which we have gone  at Mount Allison, which is, I think, pretty normal. When I first started at Mount Allison SETs were voluntary: some faculty had students complete them; others did not. There was a form one could use provided, I think, by the student union. We progressed from that to mandatory SETs and from that to a debate about two further points (a) should mandatory SETs be evaluated and pay docked if a dean determines a faculty member did not meet a standards -- that is should we adopt a punitive approach as an institutional policy --  and (b) should we have more than one set of SETs during the semester? In other words, instead of having students complete SETs at the end of the semester, should they also complete  them at the halfway mark of a course? Some faculty do this voluntarily, but should they also be made mandatory? I have heard some people suggesting that these might, in fact, be completed several times during a semester. In other words, students would be filling out surveys in the manner of a Nanos Research rolling poll.

You can see what has gone on. We have moved through a series of stages -- voluntary, mandatory, punitive assessment and multiple evaluations. The reason for this progression, I think, is not just that someone somewhere loves assessment and imposing penalties. I think this transition has occurred because people were not happy with the results. It is logical to change what we are doing if we are not getting desired results. The problem, in this case, is not that we are not getting the results that we desire (indeed, I'd argue that we actually are). The problem is that we are vague as to what the results actually are. What does it mean, for instance, to have "a say?" How can we measure improvements in education? Surveys and numbers seem like a self-evident way, but are they?

I am not dissing SETs. I think they have a role and I've said this a number of times, but they are not the only tool we should use nor inherently the most effective. I said in my last blog that I thought direct communication was the most efficient way to address problems of teaching and learning. "Prof Nurse, I did not understand X. Could you explain it?" This is efficient because it helps the student learn as opposed to punishing me after the student has *not* learnt. Think about that. Punishing me after the fact serves no effect. I get my "comeuppance" but the student still has not learnt. From an educational perspective, what good is that?

The issue, then, in my view, is not to take a tool that is not producing desired results and make more use of that tool. This is what people who want to ramp up SETs and increase their coercive and punitive character are actually saying. In a perhaps more articulate language, they are arguing this: the tools we have are not giving us our desired results so let's make greater use of those tools. Does this make sense to you? I think the issue is something different. The issue is to re-assess our beginning point and have a discussion about what our goals are and what they mean in practical terms.

For example, students want to heard. What does that mean? Do you want to be heard for the sake of being heard? That might be fair enough but I actually doubt it is the case. I suspect that students want to be heard because they are interested parties in education, have a contribution to make to pedagogy as well as specific class content, and want to reward -- not just punish -- those faculty who they feel have really helped them out on a variety of levels.  In addition, I think some simply want to be part of a conversation about the future and development of post-secondary education. That is: they want to play a role in improving it.  The point I've made before holds true again: none of these goals can be met through more punitive uses of SETs evaluations by deans or other supervisors.

Student Perspectives

I can't articulate a student perspective, because I am not a student but I make the above statement on the basis of extensive conversations with students over the years. Very, very, very few students I have encountered actually want punitive and coercive measures against faculty. I have talked to one or two who believe that there should be coercion and threats in pedagogy. But, the vast, vast majority are well-meaning, well-intentioned people who are sympathetic to faculty and the different aspects of their jobs. I've directly asked classes I've taught these types of questions and always walked away with favourable impressions of the discussion that ensued.  Indeed, what impressed me was the difference between what students say and what people who speak in students' names say.

For instance, I've had people (say, faculty or administrators) speaking for students say "students want a voice and that means that your dean should be able to dock your pay." They have not used exactly those words but that is, in effect, what they said. As I said, I've had in 15 or so years teaching only one or two students who have made that case. Most don't want to see my pay docked but want to see education improved. While I've heard faculty and administrators argue that threats are a good way of motivating faculty, I have actually never had a student say that they think threats and coercion are good motivators. Even the very small number who like them stop short of saying that threats will, ipso facto, improve teaching.

This is, then, an odd disjuncture and one that causes me a certain amount of pause. Students seem to be saying one thing, while those who speak in their names seem to be saying something else and the two do not match up.

Models

Because of this, we need to re-evaluate the model by which student voices are heard. By returning to first principles, by grappling with out objectives and with student aims, we can, I think, build much more accurately designed mechanisms for student voices to be heard and for us to make the most out of post-secondary education. Moreover, I think this can be done without coercion and punitive evaluations. I think, in fact, the overall results will be better.

This blog is already getting too long so let's leave off the issue of teaching and learning to keep focused on student voice. How can we hear student voices?  In addition to the significant amount of talking that already goes on between faculty and students, we, in Canadian Studies at Mount Allison, have done other things over the years.

First, we try to directly connect with the student club. Most department, programmes, units have a student club of one type or another. These might be made up of particularly motivated students, but ... good. There is no reason why we should ignore someone's voice simply because they are motivated, is there?

Second, we've had students complete end of degree surveys which call for extended qualitative comment. The idea here is that students finishing their degrees will have a certain perspective on what was good, bad, what helped and what hurt the learning process.

Third, we have student focus groups. We don't do these all the time, but since I've been at Mount A we've run them on academic integrity, the requirements of the honours programme, and to solicit feedback on curricular innovations.

Fourth, in class "academic moments". Usually in 3000 level courses (but these could be done at any level) I schedule a series of academic discussions (no more than five minutes unless there is a good point being made or someone who really wants to speak to an issue) about teaching and learning. Topics could include such things as the role of SETs in teaching, different modes of evaluation, responses to integrity problems, course formats, etc. These are, I suppose obviously, done on an opt-in basis. No one is forced to talk if they do not want to.

Some other ideas that could be used include:

• holding open fora on key matters of curriculum or teaching and learning
• regular (ever week, every month, once per semester, etc., whatever people want) faculty discussions with students about key issues, say the role of research in education, methods of evaluation, etc. 

And, I am sure there are still others. Thus, student views can be heard and integrated into courses and teaching practices. And, they can be heard directly. What is more, there is a lot of room for faculty and students to work on such things together. For instance, there is a constructive role for student unions in organizing regular faculty discussions (which would include give and take discussion) of academic issues (say, teaching and tenure; responding to learning issues; the merits of the lecture, etc.). Indeed, I'd suggest is a worthwhile collaboration.

Conclusion

Such mechanisms are not what people think about when they say "students want a say" but they provide that say in a direct fashion. In other words: the student speaks directly to me as opposed to filling out a form which someone else interprets (or ignores if you are not part of the "average"). This approach also allows for an opt-in mode of engagement. No one has to participate. It is an option if you want it and, therefore, respects your right to choose how, why, and when you want to engage teaching and learning issues. Moreover, such an approach illustrates confidence and trust in each other. Rather than threatening Faculty Member X, I do the opposite: I show I respect him or her and trust them to do their job by responding constructively to dialogue. They also establish dialogue -- direct communication or conversation -- as a normal operating procedure. I think this might take a short time to "catch on" but I also think that its effect are good in even the shorter run. Punitive approaches, by contrast, infantilize faculty and treat them as lazy individuals who don't care about their jobs and must be forced into line. In other words: they send exactly the wrong message for a community of scholars and this is ... after all, what we are suppose to be, no?