2008-05-12

Why were the polls wrong?

This is not the first time in Serbia, and is not likely to be the last, in which the election results differ fairly widely from the results of preelection surveys. What are some of the reasons this might be happening?
  1. There are too few survey agencies. Not that I am wishing for more survey agencies (however convenient they might be in providing short-term employment for graduate students). But as a rule, one survey does not necessarily tell you much. It becomes possible to figure things out when you are able to make comparisons over time, or when you can compare surveys done by different agencies on the basis of their samples or methods of analysis. Patterns mean more than individual results mean.
  2. The experience is not long enough. More elections means that pollsters have more familiarity with sources of error, in particular the patterns in the tendencies of people to misreport their own preferences or likelihood of voting. Although elections have been fairly frequent in the past few years, the experience of free elections only dates to December 2000. Patterns are not yet well established.
  3. The political environment is volatile. In established democracies, most votes can be accounted for by regular patterns. Regions and populations have fairly consistent tendencies, and the factor that accounts for most voting behaviour is family tradition. This does not happen in environments where parties appear and disappear with some regularity and where the population changes.
  4. Something is at stake. People do lie, and they lie more often about things that matter. Surveys on the brand of toothpaste people prefer will always get more accurate results than surveys on people's sexual or religious practices. In many established democracies where major parties converge toward the centre, political preference is more like toothpaste. In polarised societies it is more like religion and sex.
  5. Things really do change. Survey agencies had a guess about how much the stabilisation agreement with the EU, signed two weeks before the election, might help DS. But the deal between Zastava and Fiat, and the visa concessions made by European governments, came in the final week. There wasn't time to account for them.
  6. The media and communications landscape is not unified. Not everbody has a telephone. Younger people are likely to bypass the phone companies entirely and rely on mobiles. This has the effect of skewing samples. Also, not everybody has access to the same information media. Independent sources of information reach the urban entres more regularly than they reach the smaller towns and villages.
Having said all that, the surveys have not been so terribly far off, and in hindsight it may be possible to say that Strategic and Medium did catch a growth of support for DS in the final week. And anyone who has been following the primary contests in the US Democratic party knows that the pollsters have a fairly uneven record in those elections, too, so some lack of predictive power is not necessarily a unique characteristic of newer democracies. And after all, the fact that the world is not entirely predictable is probably on balance a good thing.

1 comment:

bganon said...

Hello Eric nice summary. I think they really got it wrong this time - not that I'm particularly blaming them, but it will be a delight not to have to listen to pollsters next time round claiming to have an excellent record.

BTW did you see / hear the response of the media in the media center to the rests. Their impartiality kind of 'went for a burton' as they applauded. I bet they felt absolutely sick in SRS headquarters - and when Nikolic eventually emerged he had a face that looked like thunder.