O’Neil’s Weapons of Maths Destruction

Cathy O’Neil Weapons of Maths Destruction: How Big Data increases inequality and threatens democracy Crown 2016

Weapons of Math Destruction is an urgent critique of… the rampant misuse of math in nearly every aspect of our lives.” —Boston Globe

We live in the age of the algorithm. Increasingly, the decisions that affect our lives—where we go to school, whether or not we get a job or a car loan, how much we pay for health insurance—are being made not by humans, but by mathematical models. In theory, this should lead to greater fairness: everyone is judged according to the same rules, and bias is eliminated. But as former Wall Street quant Cathy O’Neil shows in her revealing book, WEAPONS OF MATH DESTRUCTION: How Big Data Increases Inequality and Threatens Democracy (Crown; on sale September 6, 2016), the opposite is true.


O’Neil commends the oath in the financial modellers’ manifesto more generally. As with financial modelling, one problem is that the decision-makers (selecting, adapting and applying the algorithm) have different interests from those who are most affected by the decisions. Another is that the methods for designing, selecting and assessing algorithms often implicitly ignore the impact that the decisions will have on the context, thus undermining any rationale. In addition, O’Neil notes that in many cases there is no feedback on the performance of the algorithms and no competition, so bad algorithms are not ‘evolved-out’.

Some good suggestions are made. Many of these can be summarised as putting a better informed and motivated humans ‘in the loop’, either directly or as supervisors. Their performance could often be improved by using the sort of ‘big data’ maths that underpins algorithms, but more wisely. This would – hopefully – lead to a better ‘big data’ eco-system.

It seems to me, though, that there is another point to be made, about classification. Big data starts with beliefs about what are relevant data sets and streams, and how to measure and classify things. Given these seminal beliefs it may (optionally) develop further measures and classification schema, and then end up with summary statistics, such as P(B|A)=p. This is all very ‘scientific’. But in science there is a further stage that is often lacking in ‘big data’, certainly in the cases identified. Normally, one checks that the deductions lead to sensible outcomes when applied. If not, one does not try to change the value of P(B|A), but one changes the classes in terms of which one reasons. For example, physicists work in terms of mass, not weight and aeronautical engineers work in terms of mach no., not miles per hour. Similarly, in a social context, while it may be true that people with some obvious characteristic (e.g. of race or gender) are over-represented in certain undesirable groups, we often regard it as discriminatory to act on this ‘information’: because the classification is not germane to the issue. More generally, wherever big data suggest an activity we need to consider whether the suggestion is based on an appropriate classifying schema. O’Neil’s point is essential that it often is not and that often this disadvantages those who are already disadvantaged, because the algorithms aren’t intended to advantage them. But they could be, if designed with ‘the public good’ in mind. Here, I think, is a positive role for big data: in revealing to the populace at large what is actually in their interest, and what is not. This book raises some key issues, but seems to resolve very little.

Dave Marsay

%d bloggers like this: