One of the things about SETs that surprises me is how the debate -- at Mount A at least -- has become frozen in time. It has, in short, not kept up with the scholarship of teaching and learning as it pertains to SETs. Once, when I first started out in this gig (let's say 15-20 years ago), there was a vigorous debate about the accuracy of SETs. Do the scores students accord to profs accurately reflect teaching ability; that is a competence and so can be used as a tool of evaluation. This debate is old and settled. Anyone who keeps debating it is, frankly, missing out on a lot of scholarship. The scholarship on SETs falls into three camps. There are ardent defenders of their accuracy. There are ardent rejectors of their accuracy. Both are small groups. In other words, just about no one takes one side or the other in an extreme form. Instead, most people fall into a third camp that we could call "yes ... but ...." The basic premise of this perspective -- which represents the mainstream of SETs assessment the following:
- SETs are valuable tools that cannot be taken in isolation. They must be combined with other forms of assessment. In other words, to use them by themselves as your sole or even dominant teaching assessment tool would be wrong and would create inaccuracies. It would be like using a year-end poll instead of having an on-going democracy.
- SETs cannot be assessed outside of a chronological framework; that is: one needs a time span, ideally several years, for SETs to make sense. In other words, the trend is more important than any single individual number. This point should not surprise us because it is a standard point of statistical analysis. Ask Michael Adams if you don't believe me. It is the pattern into which the numbers fit that is important.
- SETs require interpretation. You cannot just look down a range of numbers, see that one prof has 4.1/5 on average in their evaluation (we use a standard 5 point scale at Mount A) and determine that that person is better than someone who had a 3.9/5. Why? We have no longitudinal data, we don't know how many students each of these profs taught, we don't know their stage of careers, we don't know the courses, we don't know how they provided extra help, we don't know whether or not the students in their courses has prerequisites. In other words, all we have is a number outside of context. What if the first person had six students and taught only one class of advanced, motivated students who happened to collectively be a really nice and deferential bunch who assumed that the prof knew what they were doing? What if the other prof laboured for hours with hundreds of students in core courses that students hated taking but were required for their degrees, providing endless hours of extra help, marking, say, hundreds of papers (I'm not exaggerating. I mark hundreds -- seriously -- of papers each year.) Would we think that a situation where the first prof's numbers could have changed dramatically if one student had changed their SETs as fair and accurate? Don't believe me ... take some time and do the math yourself. What is more, however, we believe that everything requires interpretation. Geographers interpret space; biologists interpret living things, historians interpret the past, sociologists ... society. After we teach this to our students ... why would we suddenly believe that it would not apply to our jobs? After we spend hours teaching students statistical analysis, political inquiry, quantitative methods .... why would we ditch it as if everything that we taught were unimportant?
- We don't actually assess everything. Most professions are certified, go through probation and then are approved to work in that profession. We do this with faculty; it is called tenure. After certification, though, we don't assess most professions regularly. We don't assess the ice cream we eat; the plumber who comes to our door; the mechanic to which we take our car. The idea that we assess everything is, therefore, wrong.
- Why would we use a method of assessment that might be inaccurate? Even if we agree that assessment is good and useful, we want that assessment to be accurate. Simply implementing a SETs-based assessment policy will, therefore, do nothing but say we have a policy implemented, the accuracy of which we cannot guarantee. Does that inspire your confidence? Imagine using that line for a different profession. We have imposed a method of assessment on our fire department but we cannot guarantee that it is actually accurate and so we actually tell you whether or not they'll put out the fire if you call. Does that instil faith in the fire department? What about a medical professional? Yes, we've assessed your doctor and they passed but we can't actually say whether or not this assessment is accurate so if you are sick you might or might not get the right advise. Hmmm ...