O’Neil’s Weapons of Math Destruction

Cathy O’Neil Weapons of Math Destruction: How Big Data increases inequality and threatens democracy Crown 2016.

Weapons of Math Destruction is an urgent critique of… the rampant misuse of math in nearly every aspect of our lives.”

“We live in the age of the algorithm. Increasingly, the decisions that affect our lives—where we go to school, whether or not we get a job or a car loan, how much we pay for health insurance—are being made not by humans, but by mathematical models. In theory, this should lead to greater fairness: everyone is judged according to the same rules, and bias is eliminated. But as former Wall Street quant Cathy O’Neil shows in her revealing book, WEAPONS OF MATH DESTRUCTION: How Big Data Increases Inequality and Threatens Democracy (Crown; on sale September 6, 2016), the opposite is true.”

Certainly, a book worth reading. But where is there a credible theory in favour of algorithms? I guess that in many instances it would be possible to devise an algorithm that might do better in a particular context than some status quo, such as being subject to misguided human decision-making. But how do we know that the selection, fine tuning and implementation of the algorithm, even if well-intentioned, are not also flawed? It seems to me that with any system of decision making on should monitor the outcomes, seek to identify drawbacks and then make judgements about improving the system. Many of the examples could be seen as showing the pitfalls of dogmatically adopting methods, irrespective of whether those methods rely on human judgement about the individual case or supposedly general ‘fair’ algorithms.

The book blames the misuse and abuse of maths, not maths per se. It commends Derman and Willmott‘s [financial] modellers’ manifesto and its oath:

The Modelers’ Hippocratic Oath

~ I will remember that I didn’t make the world, and it doesn’t satisfy my equations.
~ Though I will use models boldly to estimate value, I will not be overly impressed by mathematics.
~ I will never sacrifice reality for elegance without explaining why I have done so.
~ Nor will I give the people who use my model false comfort about its accuracy.  Instead, I will make explicit its assumptions and oversights.
~ I understand that my work may have enormous effects on society and the economy,  many of them beyond my comprehension.

This can obviously be generalized. But it could go further and use ‘good maths’ to counter-act the bad ‘pseudo-math’. For example: In terms of uncertainty, it is not enough to be certain about how the algorithm operates, one would need to be certain about what it is operating on, which we never are. We may entertain assertions such as that:

P(desired effect | activity) is high and P(undesired effect | activity) is low

such as when the activity is applying an algorithm. But what do these probabilities mean? I argue in my blog that they can only be relied upon for some context C, so that we really only have:

P(desired_effect | activity : context) is high and P(undesired_effect | activity : context) is low.

This can only be relied upon when we can be sure that:

  1. The activity does not change the context.
  2. We have identified all possible undesired effects.
  3. The activity does lead to us re-valuing effects, so that what were desired may become undesired.

Often, as in the examples, these are not the case, and so the statement – even if true in some technical sense – cannot be relied upon. Typically, then, it is not that the long-term effects are beyond our comprehension, but that they have been excluded from the analysis. This should always be made explicit, which might often make the oath redundant.


While many problems with the use of maths can be blamed on naivety or worse, it seems to me that we are need of some mathmatics that allows us to at least make a ‘best effort’ to check the above issues in paryticular cases, and to articulate any residual caveats. At least, we shouyld make a best effort to understand what therev is, hence this blog:

Dave Marsay

%d bloggers like this: