Can polls be reliable?

Election polls in many countries have seemed unusually unreliable recently. Why? And can they be fixed?

The most basic observation is that if one has a random sample of a population in which x% has some attribute then it is reasonable to estimate that x% of the whole population has that attribute, and that this estimate will tend to be more accurate the larger the sample is. In some polls sample size can be an issue, but not in the main political polls.

A fundamental problem with most polls is that the ‘random’ sample may not be uniformly distributed, with some sub-groups over or under represented. Political polls have some additional issues, that are sometimes blamed:

  • People with certain opinions may be reluctant to express them, or may even mislead.
  • There may be a shift in opinions with time, due to campaigns or events.
  • Different groups may differ in whether they actually vote, for example depending on the weather.

I also think that in the UK the trend to postal voting may have confused things, as postal voters will have missed out on the later stages of campaigns, and on later events. (Which were significant in the UK 2017 general election.)

Pollsters have a lot of experience at compensating for these distortions, and are increasingly using ‘sophisticated mathematical tools’. How is this possible, and is there any residual uncertainty?

Back to mathematics, suppose that we have a science-like situation in which we know which factors (e.g. gender, age, social class ..) are relevant. With a large enough sample we can partition the results by combination of factors, measure the proportions for each combination, and then combine these proportions, weighting by the prevalence of the combinations in the whole population. (More sophisticated approaches are used for smaller samples, but they only reduce the statistical reliability.)

Systematic errors can creep in in two ways:

  1. Instead of using just the poll data, some ‘laws of politics’ (such as the effect of rain) or other heuristics (such as that the swing among postal votes will be similar to that for votes in person) may be wrong.
  2. An important factor is missed. (For example, people with teenage children or grandchildren may vote differently from their peers when student fees are an issue.)

These issues have analogues in the science lab. In the first place one is using the wrong theory to interpret the data, and so the results are corrupted. In the second case one has some unnoticed ‘uncontrolled variable’ that can really confuse things.

A polling method using fixed factors and laws will only be reliable when they reasonably accurately the attributes of interest, and not when ‘the nature of politics’ is changing, as it often does and as it seems to be right now in North America and Europe. (According to game theory one should expect such changes when coalitions change or are under threat, as they are.) To do better, the polling organisation would need to understand the factors that the parties were bringing into play at least as well as the parties themselves, and possibly better. This seems unlikely, at least in the UK.

What can be done?

It seems to me that polls used to be relatively easy to interpret, possibly because they were simpler. Our more sophisticated contemporary methods make more detailed assumptions. To interpret them we would need to know what these assumptions were. We could then ‘aim off’, based on our own judgment. But this would involve pollsters in publishing some details of their methods, which they are naturally loth to do. So what could be done? Maybe we could have some agreed simple methods and publish findings as ‘extrapolations’ to inform debate, rather than predictions. We could then factor in our own assumptions. (For example, our assumptions about students turnout.)

So, I don’t think that we can expect reliable poll findings that are predictions, but possibly we could have useful poll findings that would inform debate and allow us to take our own views. (A bit like any ‘big data’.)

Dave Marsay



AV: Yes or No? A comparison of the Campaigns’ ‘reasons’

At last we have some sensible claims to compare, at the Beeb. Here are some comments:

YES Campaign

Its reasons

  1. AV makes people work harder
  2. AV cuts safe seats
  3. AV is a simple upgrade
  4. AV makes votes count
  5. AV is our one chance for a change

An Assessment

These are essentially taken from the all-party Jenkins Commission. The NO Campaign rejoinders seem to be:

  1. Not significantly so.
  2. Not significantly so.
  3. AV will require computers and £250M to implement (see below).
  4. AV Makes votes count twice, or more (se below).
  5. Too right!

A Summary

Worthy, but dull.

An Addenda

I would add:

  • There would be a lot less need for tactical voting
  • The results would more reliably indicate people’s actual first preferences
  • It would be a lot easier to vote out an unpopular government – no ‘vote splitting’
  • It would make it possible for a new party to grow support across elections to challenge the status quo.
  • It may lead to greater turnout, especially in seats that are currently safe

NO Campaign Reasons

AV is unfair


“… some people would get their vote counted more times than others. For generations, elections in the UK have been based on the fundamental principle of ‘one person, one vote’. AV would undermine all that by allowing the supporters of fringe parties to have their second, third or fourth choices counted – while supporters of the mainstream candidates would only get their vote counted once.”


According to the Concise OED a vote is ‘a formal expression of will or opinion in regard to election of … signified by ballot …’ Thus the Irish, Scottish, Welsh, Australians, who cast similar ballots to AV, ‘have one vote’. The NO Campaign use of the term ‘counted’ is also confusing. The general meaning is a ‘reckoning’, and in this sense each polling station has one count per election, and this remains true under AV. A peculiarity of AV is that ballots are also counted in the sense of ‘find number of”. (See ‘maths of voting’ for more.)


There is no obvious principle that requires us to stick with FPTP: all ballots are counted according to the same rules.

Should ‘supporters of fringe parties’ have their second preferences counted? The ‘fringe’ includes:

  • Local candidates, such as a doctor trying to stop the closure of a hospital
  • The Greens
  • In some constituencies, Labour, LibDem, Conservative.

AV is blind to everything except how voters rank them. Consider an election in which the top three candidates get 30%, 28%, 26%, with some also-rans. According to the NO campaign the candidate with a narrow margin should be declared the winner. Thus they would disregard the preferences of anyone who votes for their hospital (say). Is this reasonable?

AV is not widely used

True-ish, but neither is FPTP (in terms of countries – one of them is large), and variants of AV (IRV, STV, …) together are the most widely used.

AV is expensive

Countries with AV don’t have election machinery. Australian elections may cost more than ours, but it is a much bigger country with a smaller population. 

AV hand more power to politicians

See the Jenkins Commission.

AV supporters are sceptical

Opposition to FPTP is split between variants of AV, with single-member constituencies and forms of PR. The Jenkins Commission recommended AV+, seeking to provide the best of both. The referendum is FPTP and hence can only cope with two alternatives: YES or NO.

I don’t know that AV supporters are sceptical against a move away from FPTP – just differ on what would be ideal.


  • The NO campaign is playing down the ‘strong and stable government’ argument. The flip side is that an unpopular government can survive.
  • A traditional argument for FPTP was that it encourages tactical voting and hence politicking, and hence develops tough leaders, good at dealing with foreigners. We haven’t heard this, this time. Maybe the times are different?

See Also

AV or FPTP, according to wikipedia and Jenkins

The choice: FPTP or AV?

The UK has this choice on May 5th. (AV is also known as Instant Runoff Voting.) The debate so far hasn’t been particularly enlightening. Here I consider the advice from wikipedia and the UK Jenkins Commission, with a short note on tactical voting.


Wikipedia gives a comparison of AV to other voting systems. We are interested in FPTP, a variant of plurality voting. Wikipedia has a table  showing that both methods are about equally common. But which is best?

Advantages of FPTP

Wikipedia shows the following advantages for FPTP:

  • Preservation of “one person, one vote” principle.
  • Moderation
  • Fewer minority parties.

It notes that IRV (AV) is also generally regarded as satisfying ‘one  person one vote’, so this can be discounted. The other two are partially true, but one needs to consider the whole truth.

The UK ‘No’ campaign makes a number of other claims, but they don’t seem to have any validity.


Wikipedia notes:

Under a first-past-the-post system, voters are often afraid of “wasting” their vote on a candidate unlikely to win, so they cast their vote towards their most preferable choice possible of victory. Advocates of plurality voting suggest that this results in most candidates having to field a fairly moderate or centrist position.

This suggests that a moderate or centrist result is desirable. Where FPTP relies on tactical voting (which seems not to be so very common) to achieve this, AV tends to achieve it by design. It also achieves what wikipedia calls the mutual majority criterion. This is quite technical, but links to the notion of majority rule.

Majority rule

Majority rule is the binary decision criterion that if most people prefer A to B then A will be selected. For three or more choices all deterministic methods are technically vulnerable to tactical voting, so one needs to decide which desiderata are essential, and which can be compromised.

Suppose that one has a tribal society with the biggest tribe commanding 26% of the vote using its majority to repress the other tribes. Suppose that all other tribes would prefer a representative from anything other than the biggest tribe. Then majority rules demands that they get one. But if all tribes put up a candidate then the biggest tribe may win, due to vote splitting.

 AV is not liable to vote-splitting and respects the mutual majority criterion, and hence is ‘democratic’ in a different (fuller?) sense than FPTP where it seems to be considered a virtue that the opposition parties must form a pre-election agreement on a common candidate.

Fewer minority parties

FPTP encourages both tactical voting and strategic agreement, coalitions or unions of parties, to avoid gross vote splitting. This leads to fewer minority parties, either because tactical voters don’t vote for them or because they merge with other parties. Under AV voters can vote for a minority party without wasting their vote. Voting will also generally be less tactical, and thus, unlike FPTP, a minority is not disadvantaged in building to a majority over a series of elections.

Summary from FPTP perspective

FPTP rewards politicking and ‘strong parties’ by rewarding tactical voting and pre-election agreements between candidates or parties. It disadvantages minority parties, such as the UK Greens and BNP. Are these good things?

Under FPTP a minority party which was most voter’s last choice could gain or retain power by ‘divide and rule’. Is this a problem?

Advantages of AV

Wikipedia notes many advantages of AV, including the mutual majority criterion. It has this special case:

Instant-runoff voting [aka AV] also passes the Condorcet loser criterion, which requires that if a candidate would lose a head-to-head competition against every other candidate, they must not win the overall election. First-past-the-post does not meet this criterion, indeed it is usually violated in elections with more than two popular candidates.

Tactical voting

All methods are vulnerable to tactical voting,  so some compromise is required. Under FPTP vote-splitting can encourage you to vote tactically to avoid your worst-case choice winning. Wikipedia notes that, while not perfect, “alternative vote is quite resistant to strategy” (i.e., tactical voting). Under AV you are encouraged to vote tactically when:

  • There are preferences that form a cycle, as in AB, BC, CA.
  • You can be sure that the other candidates’ supporters aren’t voting tactically, or if they are, what proportion is voting tactically, and how.

The first condition is normally considered rare, and is where all methods have a problem. The second makes tactical voting much more risky: whereas under FPTP tactical voting is usually straightforward, under AV it is for from this.

Jenkins Commission

The UK’s Jenkins Commission took a broad of the political implications of FPTP versus AV and others. The defects of FPTP were:

  • A tendency to result in landslides.
  • Disadvantages third parties, even strong ones.
  • Disadvantages parties with even support across constituencies.
  • The essential contest is fought over a few ‘marginal’ seats.
  • It leads to ‘perverse’ results.
  • It advantages the ruling party.

The report noted the above advantages. Further:  

Fairness to voters is the first essential. A primary duty of an electoral system is to represent the wishes of the electorate as effectively as possible. The major ‘fairness’ count against First Past the Post is that it distorts the desires of the voters. That the voters do not get the representation they want is more important than that the parties do not get the seats to which they think they are entitled.


It [AV] would also virtually ensure that each MP commanded at least majority acquiescence within his constituency, which is far from being the case under FPTP, where as we have seen nearly a half of members have more opponents than supporters, and, exceptionally, a member can be elected (as in Inverness in 1992) with as little as 26% of the vote.

There were no criticisms of AV at the constituency level. The most significant criticism was that it isn’t proportional. AV is not always any more proportionate than FPTP, but this is not at issue in the UK referendum. In any case, some people prefer methods that tend to lead to enhanced majorities.


Wikepedia seems to provide a relatively independent summary of voting systems, including the unavoidable problems and the pros and cons of FPTP and AV (aka IRV) considered across many countries and years. A comparison with the UK’s current YES and NO campaigns seems instructive: not everything they say should be taken at face value.

One way to decide would be to rate yourself on the following scales:

  1. We need a system that gives the sitting candidate / party an advantage … or not.
  2. We need a system that gives the two leading candidates / parties an advantage … or not.
  3. We need a system that discriminates against minority parties such as the BNP … or we don’t want to discriminate against parties like the Green party.
  4. We want a system that tends to leave us with the same old two main parties … or we want a system that allows new parties to grow and potentially overtake the old parties, as long as they have support.
  5. We want a system that rewards politicking  and tactical voting … or not.
  6. We want every vote to be counted once (in a technical sense) … or we want to make sure that votes are not split.
  7. We want a system that tends to stable government, by giving the ruling party an advantage … versus we want a system that will enable us to oust the ruling party when the majority wish to do so.
  8. We want the winner to have the most first preferences … or we want to reject every candidate that belongs to a group such that some majority prefers all of those outside the group to all of those inside (as in the criterion above).

Of these alternatives, the first would indicate FPTP, the second AV. Another way to choose is to think about the current situation. If the government does well the Conservatives are likely to get a working majority next time. If the government does badly then labour should do well. The voting system should only affect things if the government performance is debatable, so if you support one of the major parties then the voting system will not matter unless your party’s policy turns out to be wrong. What might favour the liberals? If the coalition does okay and many floating voters think that it could have gone much worse without the liberals’ moderating influence. Thus, if the liberals’ manifesto was essentially correct they might do better than last time, more so under AV. This could be seen to be reasonable. I can’t think of a credible situation in which AV favours a party whose manifesto ‘line’ was clearly wrong. The longer term impacts, such as suppressing the BNP and the Greens, seem more significant. For example, labour might split into ‘new’ and ‘old’, allowing the electorate to choose. similarly, if the government does badly the Conservatives might split into ‘wet’ and ‘dry’, giving voters a choice. Under FPTP the two main parties are effectively coalitions with only party members choosing the ‘flavour’. AV would give others a say, which would presumably be a moderating influence. Which is more reasonable?

Which of all these considerations is more important? My current view is that:

  •  AV better respects the wishes of the majority
  • FPTP encourages tactical voting
  • AV gives candidates which have a lot of first preferences some advantage over ‘wishy washy’ candidates that are ranked reasonably high by most but which are the first preference of few. This is not as much of factor as for FPTP, but seems reasonable.

See also

David Marsay