Hand’s Improbability Principle

David Hand The Improbability Principle: Why Incredibly Unlikely Things Keep Happening Random House 2014

“The really unusual day would be one where nothing unusual happens”
– Paul Diaconis


This book is about … why incredibly unlikely things … keep on happening, time after time after time.
At first glance, this seems like a contradiction. …

1: The Mystery

   Understanding comes from deeper investigation. In this deeper investigation, thinkers – researchers, philosophers, scientists – have sought to devise ‘laws’ that describe the way that nature works. These laws are shorthand summaries encapsulating in simple form what observation shows about how the universe behaves.

2: A Capricious Universe

[The] basic laws of probability … form the foundations for the Improbability Principle.

3: What is Chance?

[The] three most widespread interpretations [of probability] – frequentist, subjective and classical probability – can all be described by the same mathematics.

The consensus now is that nature is indeed fundamentally driven by chance – that uncertainty lies at its very core. [Such] chance has its owns laws. Those laws are the foundations of probability.


There follow chapters on the principles forming Hand’s ‘Improbability Principle’.

9: The Human Mind

Hand discusses various fallacies that have been popularised by behavioural economics.

10: Life, the Universe and Everything

Hand describes ‘the principle of insufficient reason’, also known as  ‘the principle of indifference’ for assigning probabilities.

11: How to use The Improbability Principle

The Improbability Principle tells us that events that we regard as highly improbable occur because we got things wrong. If we can find out where we went wrong, then the improbable will become probable.

.. .In scientific terms we might say that the occurrence of the low-probability event has cast doubt on our theory … .

… The explanation which has the greatest probability of having produced the observed data will be the explanation in which we have the most confidence. Statisticians call this the law of likelihood: we prefer the explanation that is most likely to have produced the observed data.


“Whenever you have eliminated the impossible whatever remains, however improbable, must be the truth.” [Conan Doyle’s Sherlock Holmes]

If something appears sufficiently improbable, then we have grounds for doubting it, and we look for an alternative explanation. This is the law of statistical inference. Ω


“Chance has its reasons.” – Petronius

The improbability Principle is … a collection of strands which intertwine, braiding together and amplifying each other … .

  •  The law of inevitability says something must happen.
  • The law of truly large numbers says that, with a large enough number of opportunities, any outrageous thing is likely to happen.
  • The law of selection says that you can make probabilities as high as you like if you choose after the event.
  • The law of the probability lever says a slight change in circumstances can have a huge change on probabilities.
  • The law of near enough says that events that are sufficiently similar may be regarded as identical.

My Comments

Hand is a Bayesian, who notes that many people, in thinking probabilistically get things wrong, and this sometimes matters, sometimes a great deal. He provides a pragmatic codicil to Bayesianism which I find theoretically unfalsifiable and consider that some  past big mistakes might well have been avoided or ameliorated if all those involved had appreciated Hand’s subtleties. There seems to me some truth in the view that where the use of sophisticated mathematics has been implicated in various disasters, a solution could well have been more sophistication, along the lines of Hand’s approach.

But Hand’s principle does not address some of the more fundamental criticisms of Bayesianism. It seems to me to patch up an incredible theory rather than making it theoretically sound. Thus, even if I accept that:

  • ‘The best is the enemy of the good’
  • Hand’s patches would have been ‘good enough’ to avoid some of our past mistakes in constructing the modern world
  • There is no practical alternative currently to hand which would have given those helping to construct the world as we know it sufficient confidence to construct a world which would yield anything like the undoubted benefits that we currently enjoy.

I nonetheless think it worth developing some alternatives, in case we should find Hand’s approach (which is – I think – reasonably descriptive of the current, conventional, ‘state of the art’) inadequate. And in this, I find some support in the more reflective parts of this book.

A conventional Bayesian view, not challenged by Hand, is ‘the law’ that for any event there is some meaningful ‘probability’. He seeks to explain away some of the pratfalls this view seems to have led to, without challenging the basic law. Yet in his introductory chapter on ‘the mystery’ he points out that “laws are shorthand summaries encapsulating in simple form what observation shows”. I agree, apart from a quibble: the language used is as if there what was regarded as ‘simple’; and what ‘observation shows’ was always non-controversial. But in my experience many of the difficulties that arise from trying to apply conventional (Bayesian) probability theory arise when these are doubted. Certainly, I have either doubted them or come to think I should have doubted them. Thus I would rather say:

  • beliefs  are – at best – shorthand summaries of what we think credible, what we think consistent with observation/experiment, and what we think sufficiently simple, but no simpler.
  • beliefs are regarded as laws or principles when we think they have been adequately tested and are likely to apply beyond our experience.

In other words, ‘laws are conditional on conventions’. I suspect Hand would not disagree. But he seems to think that we can always proceed as if the basic ‘laws’ of probability are unconditional. Do such quibbles matter?

Hand says ‘events that we regard as highly improbable occur because we got things wrong. If we can find out where we went wrong, then the improbable will become probable’. I conjecture that one of the things we might think we ‘got wrong’ was in the laws of chance. Hand says that ‘In scientific terms we might say that the occurrence of the low-probability event has cast doubt on our theory ‘. I wonder if we don’t have enough experience of ‘low probability events’ to cast doubt on the applicability of the mathematical laws of probability to real-world chance?

Keynes, in his Treatise on Probability,  argues that doubts about the conventions are valid, and subsequently argued that they mattered. But his arguments (and others of a similar mind) are far from ‘simple’. So I shall compare Boole’s view, which contributed to Keynes’, and the views of Turing and Good, which built on Keynes’ to provide something more comprehensive and  comprehensible (to me).

Boole’s approach was to regard a probability as an algebraically constrained variable, with ‘we just don’t know’ as a possible (lack of) constraint. It would require something like a ‘principle of indifference’ or ‘principle of insufficient reason’ to reduce a range of possible values to a single ‘representative value’.  Such an approach seems appealing, but the criticisms of Keynes and others would need to be addressed. The point here is that Boole identified a particular aspect of the prevailing conventions that you may think in need of justification.

Good’s ‘modern Bayesian’ approach (consistent with Turing’s) takes Boole and Keynes seriously, but notes that decisions need to be justified and that such justification is best kept as simple as possible, and hence shouldn’t explicitly challenge conventions. Pragmatically, one can think unconventionally to develop a ‘frame’ for the problem within which the solution will appear conventional. This can be seen as a modification of Sherlock Holmes dictum and Hand’s law of statistical inference:

Whenever we have eliminated all possibilities that are not highly improbable, we should reconsider what we deem ‘impossible’.

In particular, if we hold to certain conventions which lead us to thinking that we are repeatedly faced with impossible or very highly improbable situations, we ought to reconsider our conventions.

This is what Hand has done: but maybe we could go further?

Following on from Keynes, Turing developed his theory of morphogenesis. In general terms, if a unstoppable force meets an immoveable object, something has to give, and similarly if two incompatible unstoppable forces meet, something has to give. If what is possible is defined relative to just one of the components, or if one of the conventions (or habits or methods) is appropriate to just one of the components, then something has to give. Thus, in addition to the factors which Hands notes, it may be that people form perceptions of probability that appropriate to the context as they have observed it, but that the context is changed due to conflicting forces. This is more my experience as I have observed and interpreted it, based on my own predilections.

Dave Marsay




Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google+ photo

You are commenting using your Google+ account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

This site uses Akismet to reduce spam. Learn how your comment data is processed.

%d bloggers like this: