Why Polls Suck

This morning The Globe and Mail, "Canada's national newspaper" released polling results apropos the up-coming federal general election. The story reported to show how the Conservative Party is in trouble in key "battlegrounds" but really what is shows is how even as well-respected a newspaper like The Globe can completely mess up reporting on polls.

Let me be clear, I don't necessarily disagree with the conclusions raised by the piece. They are more in the way of common sense than effective reporting and so don't really provide much with which anyone might disagree. The fact that the Liberals are running fairly well in Ontario should surprise no one who follows Canadian politics. The Liberals have dominated Ontario since 1993 and, really, established a solid base of support in opposition to Mulroney during the free trade election in 1988. In fact, the Liberals never ran particularly poorly in Ontario even as far back as Trudeau and Pearson. So ... having a poll that tells us the Liberals are running well in Ontario is like telling us that the sky is blue. To me, it hardly merits a front page discussion.

Likewise, those who follow Canadian politics have noted the declining popularity of the Conservatives in BC for some time. Not everyone one the east coast of Canada -- where I am located -- follows BC politics closely. The last provincial election, however, produced a resurgence of support for the NDP; the Liberals polling numbers have been good in BC for a while (in fact, the Liberals are running ahead of both the NDP and Conservatives right now) and the Green Party has a strong base in that province as well. All of this indicates that BC people have been far from united in their support of Conservatism. Again, to anyone who follows politics: no surprise here.

So ... why write a blog about an inoffensive political puff piece? Aren't there better things, someone might be saying now, that you can do with your time? The reason is the way the reporting addressed polling. The authors viewed polling as if it could be tracked over night, telling us how the Liberals were down -- overnight -- in Quebec; the Tories down -- overnight -- in Ontario; the Tories up -- overnight -- on the prairies. If a student submitted a paper to one of my courses with this type of analysis, he or she would be failed. The fact that a journalist does it for one of Canada's leading newspapers is just plain discouraging. How do I convince my students to develop good analytic techniques when, all about them, they have counter examples of what is, frankly, very poor analysis in Canada's leading journalistic organs?

Here is the deal: polling is just not that precise; nor do patterns of voter preference change that dramatically overnight. The journalists failed, for instance, to discuss how the margin of error in the poll might have affected the results. Margins of error tell us the normal ratio of error for a particular poll. So, if a poll says that the margin of error is "+/- 3% 19 times out of 20", what this means is that there is a normal three percent error ratio that comes with the poll. This can seemingly wide swings in popular support that are not, wide swings at all, but simply normal error rates manifesting themselves in the polls. So for example, the poll suggest that the Liberal party had dropped significant -- by 4% -- over night. Well, no. The poll didn't say that an no serious student of political behaviour would say that it did. If you consider the margin of error, then it is possible that Liberal support dropped by on 1%. This is not much of a news story, but it is far more accurate.

When the polling people say 19 times of 20, what they mean is that 95% of the time they are accurate to within 3% one way or the other (potentially, in an unusual scenario this could mean a swing range of 6% on any given number if we consider both the plus and minus). And, 5% of the time, they are wrong by even more than that. Professional pollsters and political behaviouralists know this. They are careful in the presentation of their data; they explain error rates and what they might mean. They tell us, for instance, that a 35% support for Party X does not mean 35%, but somewhere about there, perhaps as high as 38%; perhaps as lows as 32% (all other things being equal). And, they also note that they have a 5% chance of being just plain wrong.

Finally, the reporting told us virtually nothing about the demographics of the poll's sample group. Where was it done? Among decided voters? What was the rural/urban split of respondants? The gender and age splits? The class split? IOW, the reporting gave the readers virtually none of the information readers needed to help make sense of the poll. This is not bias in reporting -- I'll likely write about that later -- this is just bad analysis. The report on this poll is so flawed as to make the story useless to anyone who is seriously interested in the election. And, that's a shame because good, front page story real estate could have been used for real reporting.