Smith’s Mad Cows and Ecstasy

Adrian F. M. Smith Mad Cows and Ecstasy: Chance and Choice in an Evidence-Based Society Journal of the Royal Statistical Society. Series A (Statistics in Society), Vol. 159, No. 3 (1996), pp. 367-383

The Address of the President, delivered to The Royal Statistical Society on Wednesday, June 12th, 1996

Outreach

From its very beginning, the Royal Statistical Society has sought to contribute to the development of what we might call an evidence-based society, in which informed quantitative reasoning is the dominant modality in public debate, as well as in the decision-making processes of government, business and individual.

However, the innumerate and confused nature of so much political, public and media reaction to major issues involving uncertainty and risk continues to serve as a reminder to us of just how far we still are from realizing this goal?

… Is it simply our own technical and public relations failure to communicate sufficiently clearly and aggressively what we all know to be the fundamental importance and relevance of our discipline? Or are there more deep-seated, social and cultural obstacles?

STATISTICS AND AN EVIDENCE-BASED SOCIETY

3.1. The Science of Statistics

An upbeat way of viewing our subject is to see statistics as ‘the science of doing science’, whose role is to provide theory and protocols to guide and discipline all forms of quantitative investigatory procedure. Such theory and protocols fall under the following kinds of heading: the framing of questions; design of experiments or surveys; drawing up protocols for data collection and recording; collection of data by sampling or observing; monitoring compliance with protocols; monitoring data quality; data storage, summarization and presentation; stochastic modelling; statistical analysis; model criticism and the assessment of assumptions; inference reporting and the use of results for prediction, decision-making or hypothesis generation.

3.2. Towards an Evidence-based Society

Most of us with rationalist pretensions presumably aspire to live in a society in which decisions about matters of substance with significant potential social or personal implications are taken on the basis of the best available evidence, rather than on the basis of irrelevant evidence or no evidence at all. Of course, the nature of what constitutes evidence in any particular instance could be a matter for significant debate. But, modulo such debate, most of us have the aspiration to live in a society which is more, rather than less, ‘evidence based’.

We are, through the media, as ordinary citizens, confronted daily with controversy and debate across a whole spectrum of public policy issues. But, typically, we have no access to any form of systematic ‘evidence base’ and, therefore, no means of participating in the debate in a mature and informed manner.

3.3. Institutional Obstacles to an Evidence-based Society

For many people, the word evidence conjures up an immediate association with the law. It is somewhat paradoxical, therefore, that the procedures and protocols of UK law-courts seem so much at odds with the kinds of disciplined scientific reasoning that many of us would see as essential in an evidence-based society.

To us, it is a commonplace that so-called ‘common sense’ is woefully inadequate for dealing with issues involving conditional probability. The ‘prosecutor’s fallacy’ the confusion of the conditional probability of A given B with that of B given A has often been the subject of correspondence in RSS News. And the kind of conundrum typified by the ‘game show problem’ can prove challenging to even the brightest students of probability.

Moreover, empirical studies have repeatedly shown that people’s intuitive processing of information tends to ignore base rates (see, for example, Eddy (1982)). There is a very real need for adherence to formal probabilistic reasoning if gross errors are to be avoided.

(The Times, 1996), reporting a judgment of the Court of Appeal, Criminal Division, … included the following:

‘Evidence of the Bayes Theorem or any similar statistical method of analysis in a criminal trial plunged the jury into inappropriate and unnecessary realms of theory and complexity, deflecting them from their proper task . . . their Lordships … had very grave doubts as to whether that evidence was properly admissible because it trespassed on an area peculiarly and exclusively within the jury’s province, namely the way in which they evaluated the relationship between one piece of evidence and another. The Bayes Theorem might be an appropriate and useful tool for statisticians, but it was not appropriate for use in jury trials
or as a means to assist the jury in its task.’


So there we have it. To hell with rationality as we know it their Lordships have pronounced!

This is bad enough, but the legal mentality displayed here has knock-on effects well beyond the confines of the courts. A not insignificant number of Members of Parliament are lawyers. …

3.4. Sociological Obstacles to an Evidence-based Society

An underlying premise in much of what I have said is that being ‘scientific’ is good and being ‘unscientific’ is bad. However, it is important for us to be aware that -in one manifestation of ‘post-modernism’ an increasing number of people are coming to see science more as part of the problem than as the pre-eminent way of finding solutions (see, for example, Beck (1992)). … Typically, the scientific and technical response is to claim that all would be well but for ‘human error’ … . This kind of response is ultimately alienating. The problem is compounded by the fact that scientific experts are increasingly perceived to be closely associated with government and industry and thus no longer seen as disinterested seekers after truth. And these perceptions are unlikely to diminish. Academics are pressured into such relationships by the current funding climate in higher education. And various plans have been mooted to privatize many of the government-funded, but currently independently operated, research agencies in the UK.

So far as the dangers of close relationships between scientists and government and industry are concerned, there may be valuable lessons to be learned from the conduct of clinical trials. Here, there are often close associations between the pharmaceutical industry and individual scientists and statisticians working in the area. But the existence of a regulatory framework, and the statutory need for monitoring and ethical committees in the conduct of trials, seems to provide sufficient checks and balances to maintain public confidence in the process.

In summary, I believe that, when necessary, we should use the perspective of statistics as the science of doing science to distance ourselves somewhat from the insensitive excesses of single-minded science and technology, and to assert rather more strongly our potential role as a resource for disinterested comment and advice.

COMMUNICATION

4.1. Relating to Other Constituencies

An evidence-based society requires not just evidence and some form of consensus about what constitutes evidence but also open access to and communication of that evidence, in a form which can be understood and acted on, either by individuals, institutions or public policy makers. The evidence must be communicated in a form which is meaningful to the intended audience.

This seems to me to be an issue which we need to take much more seriously.

… Even when the attempted communication is with educated peer groups, such as lawyers or journalists, the cultural barriers can be considerable. This is well illustrated by the following contribution from a lawyer, given at a meeting between lawyers and statisticians, which the Society organized several years ago:

‘I am the idiot boy here today … large parts of the things that have been said here are absolute gobbledegook to me. I could not make head nor tail of most of it. Bear in mind . . . that the average lawyer cannot even work out figures with a computer. We are dealing in a field which we simply do not understand. It therefore behoves you, if you want us to accept your views, to put them in a simple language which we do understand …it is not a question of the lawyers coming to terms with your field of expertise. Do not go along with the idea that we ought to conform to what you want. You … are coming into our field. If you are coming to us, unhappily, it is for you to adapt to us, and not for us to adapt to you … lawyers, in general, tend to distrust expert evidence … for the same reason that most people also distrust expert knowledge -because they do not understand it. When we do not understand something it is very difficult to accept it as the basis for forming judgement.’
(Napley, 1982).

4.2. Relating to the Individual

In part, of course, communication is bedevilled by problems of the basic lack of the requisite mathematical education – of which more later. But it is not just a matter of mathematics …

When individual members of the public are confronted by arguments about health or environmental issues, the question most likely to be asked is ‘how does this affect me or my immediate family?’. In so far as the statistician’s answer is couched in terms of ‘averages’, or frequencies of occurrence calculated by reference to membership of a ‘population’ with which the individual does not readily identify, there will be a lack of perception of relevance on the part of the individual and a failure of communication on the part of the statistician. There are no doubt contexts in which the unavailability of disaggregated data makes this inevitable, but even when -let us say -careful studies have made it possible to identify a risk factor taxonomy of a population are we really able to communicate this in a meaningful way?

CHANCE AND CHOICE

5.1. The Psychological Dimension

People are fascinated with people, particularly with themselves. By comparison, even the fascination of Bayes theorem pales into insignificance! But, as statistical educators and communicators, we do not acknowledge or exploit this sufficiently.

5.2. Decision and Utility

Psychological and technical elements combine particularly effectively in the
context of individual decision-making in situations of uncertainty. However, despite the clear pedagogical opportunities which this offers, decision-making and utility are topics which play little role in the dominant statistical paradigms as they are taught to most students.

5.3. Risk

Perhaps the main way in which members of the general public engage with the evidence-based society is in seeking guidance on ‘risks’. There is certainly a growing public concern about the ways by which risks are identified, quantified and managed. Statistics is by no means the only discipline involved here, but it certainly plays a key role.

Perhaps the Society should consider initiating a debate about the need for
appropriate forms of national ‘riskometer‘, to provide easily understood operational guidance to the public about everyday risks, and to enable new risks as they arise to be calibrated against familiar ones.

EDUCATION

6.1. Numeracy

The Society has a clear and obvious interest in numeracy and I see most of this address as being concerned with numeracy in one form or another, going well beyond just the problems of numeracy in the context of school mathematics … .

6.2. Tackling the Mathematics Problem

In his 1989 presidential address, Sir John Kingman discussed the dual role of statisticians as guardians of a mathematical theory relating to uncertainty and as collectors and commonsense interpreters of data (Kingman, 1989). He concluded – and I entirely agree with his conclusion – that ‘responsible statistical practice requires the support of a strong theoretical infrastructure‘.

MAD COWS AND ECSTASY: OUTREACH REVISITED

discussion of BSE-CJD and ecstasy, at least initially, has taken place in rather panic-stricken political circumstances, where the media sound-bite has been the primary form of access to debate.

Does the problem have a sufficiently significant statistical component? Is a statistical view possible at all without closer and deeper immersion in non-statistical scientific or other issues? Where can or should the boundary be drawn between statistical and policy analysis and advice?

… But, in a rather fast moving situation, with enormous scientific complexity, such experts do not necessarily agree on solutions, and perhaps the most useful role that the Society can then play is to press, behind the scenes, for greater awareness of the statistical issues and to encourage open debate.

EPILOGUE

.. Our discipline is vital to the honest and decent conduct of public affairs and to promoting an informed perspective on chance and choice in an evidence-based society.

VOTE OF THANKS

D. J. Bartholomew (London School of Economics and Political Science)
… The truth about ‘he who pays the piper’ is acutely felt in academic circles and beyond. It is becoming increasingly difficult to maintain independence and objectivity in what were once the bastions of free thought and opinion. But not all are so constrained. Statisticians, like everyone else, are tending to retire earlier and to live longer. There is a growing pool of experience among those, still many years short of senility, who are beholden to no one. This ‘retired’ or, more euphemistically, ‘senior’ college is something which the Society could co-ordinate and service.

T. M. F. Smith (University of Southampton)
… His final quotation about that eminent Victorian statistician, Florence Nightingale, reminded me of the fact that at one time Gladstone had in his Cabinet five Fellows of the Society, including himself. In contrast today’s Cabinet is dominated by lawyers. … There will be no simple answers but at least simplistic solutions will be demonstrated to be false. The examples of the risks from the use of nuclear power and the application of herbicides remind us that laboratory protocols are not adequate for social enquiry; external factors which cannot be controlled should feature in the design. The Taguchi approach to social studies is long overdue.

There is no easy solution but one of the contemporary methods, making mathematics appear relevant, fails dismally in the teaching of
probability, as the following examples from national examination papers demonstrate.
(a) John cycles to school on average 3 days out of 5. Bill cycles to school on average 2 days out of 5. Find the probability that on a certain day they will both cycle to school.
(b) If the probability of there being a drop in temperature tomorrow is0.8 and the probability of it raining is 0.5, what is the probability of neither event happening?
The implied assumption of independence in these examples is both mathematically naive and scientifically wrong. It undermines good teaching in both areas and instead of being relevant is misguided and misleading. Teaching probability through the old-fashioned medium of balls in urns would at least give our citizens an appreciation of their chance of winning the lottery, as well as the potential to understand that scientific models of uncertainty are analogues of these simple system.

I have to admit that I was disappointed with the President’s analysis of ‘mad cow’ disease and ecstasy. In the spirit of the title I had hoped to see a display of the relevant evidence. There were no tables of data and even more surprising no posterior distributions of the expected numbers of deaths. … My surfing skills are poor but I managed to find an entry for ‘factsheets about BSE’ from the Ministry of Agriculture, Fisheries and Food (MAFF). I was both surprised and annoyed to discover that an MAFF fact is a public relations press release, and so I retired dataless and demoralized. The
Society should heed the President’s message, and that of our founders, and have as the mission for at least one of its publications the presentation of numerical facts about contemporary issues, not just for the benefit of Fellows, but also for the benefit of society at large

Comments

As a mathematician with an interest in analysing data, I support the view that all kinds of policy and decision-making should be more informed by relevant mathematics of uncertainty, including appropriate use of statistics. There is also much of interest that I have omitted. But I am doubtful of the seeming view that Bayes’ theorem, for example, is always appropriate. (Hence this blog.) At least, the implied assumption that mathematics and uncertainty is just about numbers needs to be questioned. So I would distinguish between the state of statistical ‘best practice’ and some ideal ‘mathematics of uncertainty’, which should be the subject of investigation and debate. Thus I would respond to the highlighted sections as follows:

  • There are deep-seated, social and cultural obstacles.
  • In so far as UK law-courts are at odds with the kinds of disciplined scientific reasoning that many statisticians see as essential, a review of the appropriate mathematics of uncertainty may be helpful to both sides.
  • In particular, so-called ‘common sense’ is woefully inadequate for dealing with issues involving conditional probability, as may be some narrowly ‘academic’ statistical practice.
  • The nature of what constitutes evidence in any particular instance should be a matter for significant debate, taking more account of the mathematics of uncertainty.
  • Simplistic science now seems more as part of the problem than the pre-eminent way of finding solutions.
  • The existence of a regulatory framework, and the statutory need for monitoring and ethical committees no longer seems to provide sufficient checks and balances to maintain public confidence in the process.
  • We should develop the perspective of statistics considered mathematically as the science of doing science to distance ourselves somewhat from the insensitive excesses of single-minded science and technology.
  • When people either do not understand something or have good reason to discount it, it is very difficult to accept it as the basis for forming judgement.
  • In so far as the statistician’s answer is couched in terms of ‘averages’, or frequencies of occurrence calculated by reference to membership of a ‘population’ with which the individual does not readily identify, there will be a lack of perception of relevance on the part of the individual and a failure of communication on the part of the statistician.
  • The mathematics of uncertainty, decision-making and utility are topics which play little role in the dominant statistical paradigms as they are taught to most students.
  • There is certainly a growing public concern about the ways by which risks are identified, quantified and managed. Statistics is by no means the only discipline involved here, but it certainly plays a key role, when viewed appropriately.
  • Responsible statistical practice requires the support of a strong theoretical infrastructure, including the mathematics of uncertainty.
  • A statistical view is often not appropriate at all without closer and deeper immersion in mathematical, non-statistical scientific or other issues? The boundary be between statistical and policy analysis and advice needs to be soundly informed, including by the mathematics of uncertainty.
  • The mathematics of uncertainty is vital to the honest and decent conduct of public affairs and to promoting an informed perspective on chance and choice in an evidence-based society.

But what kind of ‘evidence’ is appropriate to issues like BSE, and how should it be evaluated?

The Bayesian approach is that one should have P(H|E) (‘the probability of the hypothesis given the evidence’) is close to 1. But it seems to me that there may still be ‘reasonable doubt’. My own practice has been to seek alternate hypotheses, H’, whose likelihood – P(E|H’) – is not much worse than that of H and which seems credible to appropriate practitioners. There are good reasons to be sceptical about any hypothesis that is suggested by the data and has not been checked, but it seems to me to be sensible to test it, and not rely on what seemed credible before considering the evidence. From a pedantic perspective one should also consider whether or not one has a context within which things have been happening with their Bayes-calculated probabilities, but in my experience, as in BSE, the problem is not so much the mis-estimation of probabilities as the tendency for what happens not to have been considered, or unduly discounted. That is, it is often unreasonable to think that practitioners (or anyone else) will have identified all necessary hypotheses in advance of considering the evidence, as the conventional scientific method demands.

The other Smith’s comments seem insightful. He suggests that probability theory should once again be taught in terms of urn problems. This would seem to establish it on surer ground. But this would leave to problems:

  • He describes the President as a ‘Bayesian’, which to me suggests the controversial notion that probability theory can be relatively straightforwardly applied, even when it goes beyond situations that resemble urn problems.
  • Most interesting and important problems concerning uncertainty, like mad cow disease and ecstasy, are not at all like urn problems, and hence the mathematics would need to be firmly established beyond that need for such problems. How is this to be taught?

I consider this by way of Smith’s examples.

Dave Marsay

%d bloggers like this: