Feature

Misunderstood Margins Of Error

Opinion polls need careful interpretation.

Philip Gendall

In the Listener recently, a writer noted that an opinion poll had estimated support for the New Labour Party at 4%, and commented that "...the new figure puts New Labour just outside the Heylen margin of error -- plus or minus 3%." Comments like this perpetuate the same misunderstanding which saw Jim McLay dubbed "Mr Margin of Error" when his standing in political opinion polls slumped to around 3%.

Although the media report survey results as if something called "the margin of error" exists, in fact, there is no such thing as the margin of error for a survey.

The purpose of survey research is to predict or say something about a population by studying a small sample of it. The assumption is that the sample is a small-scale representation -- a microcosm -- of the population, and that the proportions, averages and other measures we compute from the sample are likely to correspond to those in the population.

We know that our sample may not be an exact replica of the population from which it was drawn, but, if it is a random sample, we can use sampling theory to estimate a confidence interval, within which we are fairly certain that the true population value we are interested in will be.

Suppose that an opinion poll estimates that 50% of the population would vote for Labour. A repetition of the survey would probably not give the same result. It might, for example, give an estimate of 47%.

Sampling Statistics

If we continued selecting fresh samples and obtaining fresh estimates, after a time we would observe a definite pattern emerging. Eventually we would end up with a frequency distribution -- or sampling distribution -- which approximates the normal distribution. The mean of that normal distribution will correspond to the true proportion in the population. The standard deviation of that normal distribution is called the standard error of the estimate.

For a normal distribution, 95% of the observations lie within about two standard deviations of its mean. So it follows that there are 95 chances out of 100 that a proportion estimated from any sample, chosen at random, will be within two standard errors of the true population proportion. It also follows that, if we have selected only one sample, the population value is likely to lie within two standard errors of our sample estimate.

This is what happens in practice. We select a sample and find the proportion who would vote for Labour is, say, 50%, with a standard error of 1.5%. We don't know the true level of support for Labour in the population, but we can be pretty sure (95%) that it is between 47% and 53% -- our sample value plus or minus twice the standard error. This 3% value -- twice the standard error of our sample estimate -- is the margin of error for the estimate, and the interval 47% to 53% is the confidence interval for the estimate.

Different Margins Of Error

Public opinion polls typically survey 1,000 respondents. With this sample size and a sample estimate of 50%, the margin of error at the 95% confidence level is 3.1%. However, there is a different margin of error for every sample value between 0% and 100%. These error margins get smaller as the sample values get closer to 0% or 100%, larger as they approach 50%. The maximum margin of error occurs when the sample value is 50%, and this is the value quoted as the error margin for a survey.

Thus, the 3.1% margins of error quoted for Heylen or NRB polls of 1,000 people are actually the maximum error margins for these surveys, and for sample estimates less than or greater than 50%, the associated margins of error will be less than 3.1%.

The other factor which influences the margin of error is sample size. As sample size increases, the error margin decreases, and vice versa. The relationship between sample size and error margin is not directly proportional -- double the sample size from 1,000 to 2,000, and you only reduce the maximum margin of error by a third, from 3% to 2%. In practice, the effect of sample size on error margins tends to be ignored when more detailed analyses of survey results are presented.

If we take a typical poll of 1,000 people, the maximum margin of error for proportions of the whole population is 3%. Start taking smaller samples, and the error increases. If we want to talk about the opinions of people under 30, assuming there are 400 of them in the sample, the maximum error margin for this subsample is 5%. For a subsample of, say, 100 young people living in the South Island, the maximum error margin is about 10%, and for a subsample of 50 women aged under 30 and living in the South Island, it is about 14%.

In the opening comment, the actual margin of error for the estimate of New Labour's support concerned is about 1.2%, not 3%. I doubt that this would be much consolation to the party, since it still means that the most optimistic estimate of their support is just over 5%, and the least optimistic, just under 3%.

Nevertheless, it is clearly ridiculous to suggest that there is some special significance in the relationship between the level of professed support for New Labour and the poll's maximum error margin.

You could argue that the whole business of polling, particularly political opinion polling, is a waste of time. But the fact is that political careers are made or broken, and the fate of the country is influenced, by the reaction to political polls and other opinion surveys. Important decisions based on the correct interpretation of polls may be bad enough, but those based on incorrect interpretations are likely to be even worse.

Philip Gendall is Professor of Marketing at Massey University.