With the election in Georgia approaching fast, polls are beginning to appear every week. Unfortunately, many of these polls are taken at face value. The reality is that at this point there is not a single pre-election poll that has demonstrated credibility. This does not necessarily mean that polling firms and newspapers are simply fabricating their data — it simply means that if they were simply fabricating their data, it would be very difficult for anyone to know.
So can we be confident that a poll is credible? There are a number of basic stipulations:
1. Reveal the sampling methodology. How, in other words, do the pollsters ensure that interviewing a few thousand people is representative of the entire electorate? Choosing respondents requires a) knowing where most people live, and b) having a very strong theory about which people are likely to turn out to vote on election day. This is very difficult stuff, and even tiny errors here can have tremendous consequences.
2. Tell us about the field work. Were the interviews done face to face or by telephone? When and how? Did the survey enumerators explain who they were working for, and is it possible that the respondents knew that they were looking certain answers?
3. Publish the questionnaire. What exactly was asked, and how, and in what sequence?
4. Document the non-response rate. How many people refused to answer? There are plenty of people who don’t pick up the phone, or who don’t have 30 minutes to talk to pollsters…and in this country, many of those people will vote.
5. Allow peer-review. Power point presentations for nonspecialists are fine, but make the data set available to peers for professional scrutiny (and of course you can restrict usage). If you really are confident in what you’re doing, this is the way to go.
If polls do not meet the standards, they really do not deserve to be taken seriously.
Too many commentators forget that the burden of proof is on the polling firms, not on the public. We seem to be entering a dangerous cycle, where there is a lot of awful information floating around, and no one has the ability to sort the good from the bad. This is as much a problem with what the public is demanding as what the firms are supplying. The public should beware, and commentators should be very cautious about taking firms’ power point slides at face value, until some basic methodological questions are answered transparently.
