Traffic bunching

In heavy traffic, such as on motorways in rush-hour, there is often oscillation in speed and there can even be mysterious ’emergent’ halts. The use of variable speed limits can result in everyone getting along a given stretch of road quicker.

Soros (worth reading) has written an article that suggests that this is all to do with the humanity and ‘thinking’ of the drivers, and that something similar is the case for economic and financial booms and busts. This might seem to indicate that ‘mathematical models’ were a part of our problems, not solutions. So I suggest the following thought experiment:

Suppose a huge number of  identical driverless cars with deterministic control functions all try to go along the same road, seeking to optimise performance in terms of ‘progress’ and fuel economy. Will they necessarily succeed, or might there be some ‘tragedy of the commons’ that can only be resolved by some overall regulation? What are the critical factors? Is the nature of the ‘brains’ one of them?

Are these problems the preserve of psychologists, or does mathematics have anything useful to say?

Dave Marsay

Are fananciers really stupid?

The New Scientist (30 March 2013) has the following question, under the heading ‘Stupid is as stupid does’:

Jack is looking at Anne but Anne is looking at George. Jack is married but George is not. Is a married person looking at an unmarried person?

Possible answers are: “yes”, “no” or “cannot be determined”.

You might want to think about this before scrolling down.

.

.

.

.

.

.

.

It is claimed that while ‘the vast majority’ (presumably including financiers, whose thinking is being criticised) think the answer is “cannot be determined”,

careful deduction shows that the answer is “yes”.

Similar views are expressed at  a learning blog and at a Physics blog, although the ‘careful deductions’ are not given. Would you like to think again?

.

.

.

.

.

.

.

.

Now I have a confession to make. My first impression is that the closest of the admissible answers is ‘cannot be determined’, and having thought carefully for a while, I have not changed my mind. Am I stupid? (Based on this evidence!) You might like to think about this before scrolling down.

.

.

.

.

.

.

.

Some people object that the term ‘is married’ may not be well-defined, but that is not my concern. Suppose that one has a definition of marriage that is as complete and precise as possible. What is the correct answer? Does that change your thinking?

.

.

.

.

.

.

.

Okay, here are some candidate answers that I would prefer, if allowed:

  1. There are cases in which the answer cannot be determined.
  2. It is not possible to prove that there are not cases in which the answer cannot be determined. (So that the answer could actually be “yes”, but we cannot know that it is “yes”.)

Either way, it cannot be proved that there is a complete and precise way of determining the answer, but for different reasons. I lean towards the first answer, but am not sure. Which it is is not a logical or mathematical question, but a question about ‘reality’, so one should ask a Physicist. My reasoning follows … .

.

.

.

.

.

.

.

.

Suppose that Anne marries Henry who dies while out in space, with a high relative velocity and acceleration. Then to answer yes we must at least be able to determine a unique time in Anne’s time-frame in which Henry dies, or else (it seems to me) there will be a period of time in which Anne’s status is indeterminate. It is not just that we do not know what Anne’s status is; she has no ‘objective’ status.

If there is some experiment which really proves that there is no possible ‘objective’ time (and I am not sure that there is) then am I not right? Even if there is no such experiment, one cannot determine the truth of physical theories, only fail to disprove them. So either way, am I not right?

Enlightenment, please. The link to finance is that the New Scientist article says that

Employees leaving logic at the office door helped cause the financial crisis.

I agree, but it seems to me (after Keynes) that it was their use of the kind of ‘classical’ logic that is implicitly assumed in the article that is at fault. Being married is a relation, not a proposition about Anne. Anne has no state or attributes from which her marital status can be determined, any more than terms such as crash, recession, money supply, inflation, inequality, value or ‘the will of the people’ have any correspondence in real economies.  Unless you know different?

Dave Marsay

Intelligence-led: Intelligent?

In the UK, after various scandals in the 90s, it seemed that horizon scanning for potential problems, such as the BSE crisis, ought to be more intelligent and even ‘intelligence-led’ or ‘evidence-led’ as against being prejudice or spin-led. Listening to ministerial pronouncements on the horse-meat scandal I wonder if  the current so-called ‘intelligence-led’ approach is actually intelligent.

Suppose that the house next door becomes a refuge for drug-addicts. Which of the following are intelligent? Intelligence-led?

  1. Wait until there is a significant increase in crime locally – or until you get burgled – and then up-rate your security.
  2. Review security straight away.

In case you hadn’t guessed, this relates to my blog, and the question of what you mean by ‘information’ and ‘evidence’.

Does anyone have a definition of what is meant by ‘intelligence-led’ in this context?

Dave Marsay

P.S. I have some more puzzles on uncertainty.

 

Risks to scientists from mis-predictions

The recent conviction of six seismologists and a public official for reassuring the public about the risk of an earthquake when there turned out to be one raises many issues, mostly legal, but I want to focus on the scientific aspects, specifically the assessment and communication of uncertainty.

A recent paper by O’Hagan  notes that there is “wide recognition that the appropriate representation for expert judgements of uncertainty is as a probability distribution for the unknown quantity of interest …”.  This conflicts with UK best practice, as described by Spiegelhalter at understanding uncertainty. My own views have been formed by experience of potential and actual crises where evaluation of uncertainty played a key role.

From a mathematical perspective, probability theory is a well-grounded theory depending on certain axioms. There are plausible arguments that these axioms are often satisfied, but these arguments are empirical and hence should be considered at best as scientific rather than mathematical or ‘universally true’.  O’Hagan’s arguments, for example, start from the assumption that uncertainty is nothing but a number, ignoring Spiegelhalter’s ‘Knightian uncertainty‘.

Thus, it seems to me, that where there are rare critical decisions with a lack of evidence to support a belief in the axioms, one should recognize the attendant non-probabilistic uncertainty, and that failure to do so is a serious error, meriting some censure. In practice, one needs relevant guidance such as the UK is developing, interpreted for specific areas such as seismology. This should provide both guidance (such as that at understanding uncertainty) to scientists and material to be used in communicating risk to the public, preferably with some legal status. But what should such guidance be? Spiegelhalter’s is a good start, but needs developing.

My own view is that one should have standard techniques that can put reasonable bounds on probabilities, so that one has something that is relatively well peer-reviewed, ‘authorised’ and ‘scientific’ to inform critical decisions. But in applying any methods one should recognize any assumptions that have been made to support the use of those methods, and highlight them. Thus one may say that according to the usual methods, ‘the probability is p’, but that there are various named factors that lead you to suppose that the ‘true risk’ may be significantly higher (or lower). But is this enough?

Some involved in crisis management have noted that scientists generally seem to underestimate risk. If so, then even the above approach (and the similar approach of understanding uncertainty) could tend to understate risk. So do scientists tend to understate the risks pertaining to crises, and why?

It seems to me that one cannot be definitive about this, since there are, from a statistical perspective – thankfully – very few crises or even near-crises. But my impression is that could be something in it. Why?

As at Aquila, human and organisational factors seem to play a role, so that some answers seem to need more justification that others. Any ‘standard techniques’ would need take account of these tendancies. For example, I have often said that the key to good advice is to have a good customer, who desires an adequate answer – whatever it is – who fully appreciates the dangers of misunderstanding arising, and is prepared to invest the time in ensuring adequate communication. This often requires debate and perhaps role-playing, prior to any crisis. This was not achieved at Aquila. But is even this enough?

Here I speculate even more. In my own work, it seems to me that where a quantity such as P(A|B) is required and scientists/statisticians only have a good estimate of P(A|B’) for some B’ that is more general than B, then P(A|B’) will be taken as ‘the scientific’ estimate for P(A|B). This is so common that it seems to be a ‘rule of pragmatic inference’, albeit one that seems to be unsupported by the kind of arguments that O’Hagan supports. My own experience is that it can seriously underestimate P(A|B).

The facts of the Aquila case are not clear to me, but I suppose that the scientists made their assessment based on the best available scientific data. To put it another way, they would not have taken account of ad-hoc observations, such as amateur observations of radon gas fluctuations. Part of the Aquila problem seems to be that the amateur observations provided a warning which the population were led to discount on the basis of ‘scientific’ analysis. More generally, in a crisis, one often has a conflict between a scientific analysis based on sound data and non-scientific views verging on divination. How should these diverse views inform the overall assessment?

In most cases one can make a reasonable scientific analysis based on sound data and ‘authorised assumptions’, taking account of recognized factors. I think that one should always strive to do so, and to communicate the results. But if that is all that one does then one is inevitably ignoring the particulars of the case, which may substantially increase the risk. One may also want to take a broader decision-theoretic view. For example, if the peaks in radon gas levels were unusual then taking them as a portent might be prudent, even in the absence of any relevant theory. The only reason for not doing so would be if the underlying mechanisms were well understood and the gas levels were known to be simply consequent on the scientific data, thus providing no additional information. Such an approach is particularly indicated where – as I think is the case in seismology – even the best scientific analysis has a poor track record.

The bottom line, then, is that I think that one should always provide ‘the best scientific analysis’ in the sense of an analysis that gives a numeric probability (or probability range etc) but one needs to establish a best practice that takes a broader view of the issue in question, and in particular the limitations and potential biases of ‘best practice’.

The O’Hagan paper quoted at the start says – of conventional probability theory – that  “Alternative, but similarly compelling, axiomatic or rational arguments do not appear to have been advanced for other ways of representing uncertainty.” This overlooks Boole, Keynes , Russell and Good, for example. It may be timely to reconsider the adequacy of the conventional assumptions. It might also be that ‘best scientific practice’ needs to be adapted to cope with messy real-world situations. Aquila was not a laboratory.

See Also

My notes on uncertainty and on current debates.

Dave Marsay

Avoiding ‘Black Swans’

A UK Blackett Review has reviewed some approaches to uncertainty relevent to the question “How can we ensure that we minimise strategic surprises from high impact low probability risks”. I have already reviewed the report in its own terms.  Here I consider the question.

  • One person’s surprise may be as a result of another person’s innovation, so we need to consider the up-sides and down-sides together.
  • In this context ‘low probability’ is subjective. Things are not surprising unless we didn’t expect them, so the reference to low probability is superfluous.
  • Similarly, strategic surprise necessarily relates to things that – if only in anticipation – have high impact.
  • Given that we are concerned with areas of innovation and high uncertainty, the term ‘minimise’ is overly ambitious. Reducing would be good. Thinking that we have minimized would be bad.

The question might be simplified to two parts:

  1. “How can we ensure that we strategize?
  2. “How can we strategize?”

These questions clearly have very important relative considerations, such as:

  • What in our culture inhibits strategizing?
  • Who can we look to for exemplars?
  • How can we convince stakeholders of the implications of not strategizing?
  • What else will we need to do?
  • Who might we co-opt or collaborate with?

But here I focus on the more widely-applicable aspects. On the first question the key point seems to be that, where the Blackett review points out the limitations of a simplistic view of probability, there are many related misconceptions and misguided ways that blind us to the possibility of or benefits of strategizing. In effect, as in economics, we have got ourselves locked into ‘no-strategy strategies’, where we believe that a short-term adaptive approach, with no broader or long-term view, is the best, and that more strategic approaches are a snare and a delusion. Thus the default answer to the original question seems to be ‘you don’t  – you just live with the consequences’. In some cases this might be right, but I do not think that we should take it for granted. This leads on to the second part.

We at least need ‘eyes open minds open’, to be considering potential surprises, and keeping score. If (for example, as in International Relations) it seems that none of our friends do better than chance, we should consider cultivating some more. But the scoring and rewarding is an important issue. We need to be sure that our mechanisms aren’t recognizing short-term performance at the expense of long-run sustainability. We need informed views about what ‘doing well’ would look like and what are the most challenging issues, and to seek to learn and engage with those who are doing well. We then need to engage in challenging issues ourselves, if only to develop and then maintain our understanding and capability.

If we take the financial sector as an example, there used to be a view that regulation was not needed. There are two more moderate views:

  1. That the introduction of rules would distort and destabilise the system.
  2. That although the system is not inherently stable, the government is not competent to regulate, and no regulation is better than bad regulation.

 My view is that what is commonly meant by ‘regulation’ is very tactical, whereas the problems are strategic. We do not need a ‘strategy for regulation’: we need strategic regulation. One of the dogmas of capitalism is that it involves ‘free markets’ in which information plays a key role. But in the noughties the markets were clearly not free in this sense. A potential role for a regulator, therefore, would be to perform appropriate ‘horizon scanning’ and to inject appropriate information to ‘nudge’ the system back into sustainability. Some voters would be suspicious of a government that attempts to strategize, but perhaps this form of regulation could be seen as simply better-informed muddling, particularly if there were strong disincentives to take unduly bold action.

But finance does not exist separate from other issues. A UK ‘regulator’ would need to be a virtual beast spanning  the departments, working within the confines of regular general elections, and being careful not to awaken memories of Cromwell.

This may seem terribly ambitious, but maybe we could start with reformed concepts of probability, performance, etc. 

Comments?

See also

JS Mill’s views

Other debates, my bibliography.  

Dave Marsay

The money forecast

A review of The Money forecast A Haldane New Scientist 10 Dec. 2011. On-line version is To Navigate economic storms we need better forecasting.

Summary

Andrew Haldane, ‘Andy’, is one of the more insightful and – hopefully – influential members of the UK economic community, recognising that new ways of thinking are needed and taking a lead in their development.

He refers to a previous article ‘Revealed – the Capitalist network that runs the world’, which inspires him to attempt to map the world of finance.

“… Making sense of the financial system is more an act of archaeology than futurology.”

Of the pre-crisis approach it says:

“… The mistake came in thinking the behaviour of the system was just an aggregated version of the behaviour of the individual. …

”    Interactions between agents are what matters. And the key to that is to explore the underlying architecture of the network, not the behaviour of any one node. To make an analogy, you cannot understand the brain by focusing on a neuron – and then simply multiplying by 100 billion. …

… When parts started to malfunction … no one had much idea what critical faculties would be impaired.

    That uncertainty, coupled with dense financial wiring, turned small failures into systemic collapse. …

    Those experiences are now seared onto the conscience of regulators. Systemic risk has entered their lexicon, and to understand that risk, they readily acknowledge the need to join the dots across the network. So far, so good. Still lacking are the data and models necessary to turn this good intent into action.

… Other disciplines have cut a dash in their complex network mapping over the past generation, assisted by increases in data-capture and modelling capability made possible by technology. One such is weather forecasting … .

   Success stories can also be told about utility grids and transport networks, the web, social networks, global supply chains and perhaps the most complex web of all, the brain.

    …  imagine the scene a generation hence. There is a single nerve centre for global finance. Inside, a map of financial flows is being drawn in real time. The world’s regulatory forecasters sit monitoring the financial world, perhaps even broadcasting it to the world’s media.

    National regulators may only be interested in a quite narrow subset of the data for the institutions for which they have responsibility. These data could be part of, or distinct from, the global architecture.

    …  it would enable “what-if?” simulations to be run – if UK bank Northern Rock is the first domino, what will be the next?”

Comments

I am unconvinced that archeology, weather forecasting or the other examples are really as complex as economic forecasting, which can be reflexive: if all the media forecast a crash there probably will be one, irrespective of the ‘objective’ financial and economic conditions. Similarly, prior to the crisis most people seemed to believe in ‘the great moderation’, and the good times rolled on, seemingly.

Prior to the crisis I was aware that a minority of British economists were concerned about the resilience of the global financial system and that the ‘great moderation’ was a cross between a house of cards and a pyramid selling scheme. In their view, a global financial crisis precipitated by a US crisis was the greatest threat to our security. In so far as I could understand their concerns, Keynes’ mathematical work on uncertainty together with his later work on economics seemed to be key.

Events in 2007 were worrying. I was advised that the Chinese were thinking more sensibly about these issues, and I took to opportunity to visit China in Easter 2008, hosted by the Chinese Young Persons Tourist Group, presumably not noted for their financial and economic acumen. It was very apparent from a coach ride from Beijing to the Great Wall that their program of building new towns and moving peasants in was on hold. The reason given by the Tour Guide was that the US financial system was expected to crash after their Olympics, leading to a slow-down in their economic growth, which needed to be above 8% or else they faced civil unrest. Once tipped off, similar measures to mitigate a crisis were apparent almost everywhere. I also talked to a financier, and had some great discussions about Keynes and his colleagues, and the implications for the crash. In the event the crisis seems to have been triggered by other causes, but Keynes conceptual framework still seemed relevant.

The above only went to reinforce my prejudice:

  • Not only is uncertainty important, but one needs to understand its ramifications as least as well as Keynes did (e.g. in his Treatise and ‘Economic Consequences of the Peace’).
  • Building on this, concepts such as risk need to be understood to their fullest extent, not reduced to numbers.
  • The quotes above are indicative of the need for a holistic approach. Whatever variety one prefers, I do think that this cannot be avoided.
  • The quote about national regulators only having a narrow interest seems remarkably reductionist. I would think that they would all need a broad interest and to be exchanging data and views, albeit they may only have narrow responsibilities. Financial storms can spread around the world quicker than meteorological ones.
  • The – perhaps implicit – notion of only monitoring financial ‘flows’ seems ludicrous. I knew that the US was bound to fail eventually, but it was only by observing changes in migration that I realised it was imminent. Actually, I might have drawn the same conclusion from observing changes in financial regulation in China, but that still was not a ‘financial flow’. I did previously draw similar conclusions talking to people who were speculating on ‘buy to let’, thinking it a sure-thing.
  • Interactions between agents and architectures are important, but if Keynes was right then what really matters are changes to ‘the rules of the games’. The end of the Olympics was not just a change in ‘flows’ but a potential game-changer.
  • Often it is difficult to predict what will trigger a crisis, but one can observe when the situation is ripe for one. To draw an analogy with forest fires, one can’t predict when someone will drop a bottle or a lit cigarette, but one can observe when the tinder has built up and is dry.

It thus seems to me that while Andy Haldane is insightful, the actual article is not that enlightening, and invites a much too prosaic view of forecasting. Even if we think that Keynes was wrong I am fairly sure that we need to develop language and concepts in which we can have a discussion of the issues, even if only ‘Knightian uncertainty’. The big problem that I had prior to the crisis was the lack of a possibility of such a discussion. If we are to learn anything from the crisis it is surely that such discussions are essential. The article could be a good start.

See Also

The short long. On the trend to short-termism.

Control rights (and wrongs). On the imbalance between incentives and risks in banking.

Risk Off. A behaviorist’ view of risk. It notes that prior to the crash ‘risk was under-priced’.

  Dave Marsay

 

UK judge rules against probability theory? R v T

Actually, the judge was a bit more considered than my title suggests. In my defence the Guardian says:

“Bayes’ theorem is a mathematical equation used in court cases to analyse statistical evidence. But a judge has ruled it can no longer be used. Will it result in more miscarriages of justice?”

The case involved Nike trainers and appears to be the same as that in a recent appeal  judgment, although it doesn’t actually involve Bayes’ rule. It just involves the likelihood ratio, not any priors. An expert witness had said:

“… there is at this stage a moderate degree of scientific evidence to support the view that the [appellant’s shoes] had made the footwear marks.”

The appeal hinged around the question of whether this was a reasonable representation of a reasonable inference.

According to Keynes, Knight and Ellsberg, probabilities are grounded on either logic, statistics or estimates. Prior probabilities are – by definition – never grounded on statistics and in practical applications rarely grounded on logic, and hence must be estimates. Estimates are always open to challenge, and might reasonably be discounted, particularly where one wants to be ‘beyond reasonable doubt’.

Likelihood ratios are typically more objective and hence more reliable. In this case they might have been based on good quality relevant statistics, in which case the judge supposed that it might be reasonable to state that there was a moderate degree of scientific evidence. But this was not the case. Expert estimates had supplied what the available database had lacked, so introducing additional uncertainty. This might have been reasonable, but the estimate appears not to have been based on relevant experience.

My deduction from this is that where there is doubt about the proper figures to use, that doubt should be acknowledged and the defendant given the benefit of it. As the judge says:

“… it is difficult to see how an opinion … arrived at through the application of a formula could be described as ‘logical’ or ‘balanced’ or ‘robust’, when the data are as uncertain as we have set out and could produce such different results.”

This case would seem to have wider implications:

“… we do not consider that the word ‘scientific’ should be used, as … it is likely to give an impression … of a degree of  precision and objectivity that is not present given the current state of this area of expertise.”

My experience is that such estimates are often used by scientists, and the result confounded with ‘science’. I have sometimes heard this practice justified on the grounds that some ‘measure’ of probability is needed and that if an estimate is needed it is best that it should be given by an independent scientist or analyst than by an advocate or, say, politician. Maybe so, but perhaps we should indicate when this has happened, and the impact it has on the result. (It might be better to follow the advice of Keynes.)

Royal Statistical Society

The guidance for forensic scientists is:

“There is a long history and ample recent experience of misunderstandings relating to statistical information and probabilities which have contributed towards serious miscarriages of justice. … forensic scientists and expert witnesses, whose evidence is typically the immediate source of statistics and probabilities presented in court, may also lack familiarity with relevant terminology, concepts and methods.”

“Guide No 1 is designed as a general introduction to the role of probability and statistics in criminal proceedings, a kind of vade mecum for the perplexed forensic traveller; or possibly, ‘Everything you ever wanted to know about probability in criminal litigation but were too afraid to ask’. It explains basic terminology and concepts, illustrates various forensic applications of probability, and draws attention to common reasoning errors (‘traps for the unwary’).”

The guide is clearly much needed. It states:

“The best measure of uncertainty is probability, which measures uncertainty on a scale from 0 to 1.”

This statement is nowhere supported by any evidence whatsoever. No consideration is given to alternatives, such as those of Keynes, or to the legal concept of “beyond reasonable doubt.”

“The type of probability that arises in criminal proceedings is overwhelmingly of the subjective variety, …

There is no consideration of Boole and Keynes’ more logical notion, or any reason to take notice of the subjective opinions of others.

“Whether objective expressions of chance or subjective measures of belief, probabilistic calculations of (un)certainty obey the axiomatic laws of probability, …

But how do we determine whether those axioms are appropriate to the situation at hand? The reader is not told whether the term axiom is to be interpreted in its mathematical or lay sense: as something to be proved, or as something that may be assumed without further thought. The first example given is:

“Consider an unbiased coin, with an equal probability of producing a ‘head’ or a ‘tail’ on each coin-toss. …”

Probability here is mathematical. Considering the probability of an untested coin of unknown provenance would be more subjective. It is the handling of the subjective component that is at issue, an issue that the example does not help to address. More realistically:

“Assessing the adequacy of an inference is never a purely statistical matter in the final analysis, because the adequacy of an inference is relative to its purpose and what is at stake in any particular context in relying on it.”

“… an expert report might contain statements resembling the following:
* “Footwear with the pattern and size of the sole of the defendant’s shoe occurred in approximately 2% of burglaries.” …
It is vital for judges, lawyers and forensic scientists to be able to identify and evaluate the assumptions which lie behind these kinds of statistics.”

This is good advice, which the appeal judge took. However, while I have not read and understood every detail of the guidance, it seems to me that the judge’s understanding went beyond the guidance, including its ‘traps for the unwary’.

The statistical guidance cites the following guidance from the forensic scientists’ professional body:

Logic: The expert will address the probability of the evidence given the proposition and relevant background information and not the probability of the proposition given the evidence and background information.”

This seems sound, but needs supporting by detailed advice. In particular none of the above guidance explicitly takes account of the notion of ‘beyond reasonable doubt’.

Forensic science view

Science and Justice has an article which opines:

“Our concern is that the judgment will be interpreted as being in opposition to the principles of logical interpretation of evidence. We re-iterate those principles and then discuss several extracts from the judgment that may be potentially harmful to the future of forensic science.”

The full article is behind a pay-wall, but I would like to know what principles it is referring to. It is hard to see how there could be a conflict, unless there are some extra principles not in the RSS guidance.

Criminal law Review

Forensic Science Evidence in Question argues that:

 “The strict ratio of R. v T  is that existing data are legally insufficient to permit footwear mark experts to utilise probabilistic methods involving likelihood ratios when writing reports or testifying at trial. For the reasons identified in this article, we hope that the Court of Appeal will reconsider this ruling at the earliest opportunity. In the meantime, we are concerned that some of the Court’s more general statements could frustrate the jury’s understanding of forensic science evidence, and even risk miscarriages of justice, if extrapolated to other contexts and forms of expertise. There is no reason in law why errant obiter dicta should be permitted to corrupt best scientific practice.”

In this account it is clear that the substantive issues are about likelihoods rather than probabilities, and that consideration of ‘prior probabilities’ are not relevant here. This is different from the Royal Society’s account, which emphasises subjective probability. However, in considering the likelihood of the evidence conditioned on the suspect’s innocence, it is implicitly assumed that the perpetrator is typical of the UK population as a whole, or of people at UK crime scenes as a whole. But suppose that women are most often murdered by men that they are or have been close to, and that such men are likely to be more similar to each other than people randomly selected from the population as a whole. Then it is reasonable to suppose that the likelihood that the perpetrator is some other male known to the victim will be significantly greater than the likelihood of it being some random man. The use of an inappropriate likelihood introduces a bias.

My advice: do not get involved with people who mostly get involved with people like you, unless you trust them all.

The Appeal

Prof. Jamieson, an expert on the evaluation of evidence whose statements informed the appeal, said:

“It is essential for the population data for these shoes be applicable to the population potentially present at the scene. Regional, time, and cultural differences all affect the frequency of particular footwear in a relevant population. That data was simply not … . If the shoes were more common in such a population then the probative value is lessened. The converse is also true, but we do not know which is the accurate position.”

Thus the professor is arguing that the estimated likelihood could be too high or too low, and that the defence ought to be given the benefit of the doubt. I have argued that using a whole population likelihood is likely to be actually biased against the defence, as I expect such traits as the choice of shoes to be clustered.

Science and Justice

Faigman, Jamieson et al, Response to Aitken et al. on R v T Science and Justice 51 (2011) 213 – 214

This argues against an unthinking application of likelihood ratios, noting:

  • That the defence may reasonable not be able explain the evidence, so that there may be no reliable source for an innocent hypothesis.
  • That assessment of likelihoods will depend on experience, the basis for which should be disclosed and open to challenge.
  • If there is doubt as to how to handle uncertainty, any method ought to be tested in court and not dictated by armchair experts.

On the other hand, when it says “Accepting that probability theory provides a coherent foundation …” it fails to note that coherence is beside the point: is it credible?

Comment

The current situation seems unsatisfactory, with the best available advice both too simplistic and not simple enough. In similar situations I have co-authored a large document which has then been split into two: guidance for practitioners and justification. It may not be possible to give comprehensive guidance for practitioners, in which case one should aim to give ‘safe’ advice, so that practitioners are clear about when they can use their own judgment and when they should seek advice. This inevitably becomes a ‘legal’ document, but that seems unavoidable.

In my view it should not be simply assumed that the appropriate representation of uncertainty is ‘nothing but a number’. Instead one should take Keynes’ concerns seriously in the guidance and explicitly argue for a simpler approach avoiding ‘reasonable doubt’, where appropriate. I would also suggest that any proposed principles ought to be compared with past cases, particularly those which have turned out to be miscarriages of justice. As the appeal judge did, this might usefully consider foreign cases to build up an adequate ‘database’.

My expectation is that this would show that the use of whole-population likelihoods as in R v T is biased against defendants who are in a suspect social group.

More generally, I think that anyguidance ought to apply to my growing uncertainty puzzles, even if it only cautions against a simplistic application of any rule in such cases.

See Also

Blogs: The register, W Briggs and Convicted by statistics (referring to previous miscarriages).

My notes on probability. A relevant puzzle.

Dave Marsay