Mathematical modelling

I had the good fortune to attend a public talk on mathematical modelling, organised by the University of Birmingham (UK). The speaker, Dr Nira Chamberlain CMath FIMA CSci, is a council member of the appropriate institution, and so may reasonably be thought to be speaking for mathematicians generally.

He observed that there were many professional areas that used mathematics as a tool, and that they generally failed to see the need for professional mathematicians as such. He thought that mathematical modelling was one area where – at least for the more important problems – mathematicians ought to be involved. He gave examples of modelling, including one of the financial crisis.

The main conclusion seemed very reasonable, and in line with the beliefs of most ‘right thinking’ mathematicians. But on reflection, I wonder if my non-mathematician professional colleagues would accept it. In 19th century professional mathematicians were proclaiming it a mathematical fact that the physical world conformed to classical geometry. On this basis, mathematicians do not seem to have any special ability to produce valid models. Indeed, in the run up to the financial crash there were too many professional mathematicians who were advocating some mainstream mathematical models of finance and economies in which the crash was impossible.

In Dr Chamberlain’s own model of the crash, it seems that deregulation and competition led to excessive risk taking, which risks eventually materialised. A colleague who is a professional scientist but not a professional mathematician has advised me that this general model was recognised by the UK at the time of our deregulation, but that it was assumed (as Greenspan did) that somehow some institution would step in to foreclose this excessive risk taking. To me, the key thing to note is that the risks being taken were systemic and not necessarily recognised by those taking them. To me, the virtue of a model does not just depend on it being correct in some abstract sense, but also that ‘has traction’ with relevant policy and decision makers and takers. Thus, reflecting on the talk, I am left accepting the view of many of my colleagues that some mathematical models are too important to be left to mathematicians.

If we have a thesis and antithesis, then the synthesis that I and my colleagues have long come to is that important mathematical model needs to be a collaborative endeavour, including mathematicians as having a special role in challenging, interpret and (potentially) developing the model, including developing (as Dr C said) new mathematics where necessary. A modelling team will often need mathematicians ‘on tap’ to apply various methods and theories, and this is common. But what is also needed is a mathematical insight into the appropriateness of these tools and the meaning of the results. This requires people who are more concerned with their mathematical integrity than in satisfying their non-mathematical pay-masters. It seems to me that these are a sub-set of those that are generally regarded as ‘professional’. How do we identify such people?

Dave Marsay 

 

Advertisements

What should replace utility maximization in economics?

Mainstream economics has been based on the idea of people producing and trading in order to maximize their utility, which depends on their assigning values and conditional  probabilities to outcomes. Thus, in particular, mainstream economics implies that people do best by assigning probabilities to possible outcomes, even when there seems no sensible way to do this (such as when considering a possible crash). Ken Arrow has asked, if one rejects utility maximization, what should one replace it with?

The assumption here seems to be that it is better to have a wrong theory than to have no theory. The fear seems to be that economies would grind to a holt unless they were sanctioned by some theory – even a wrong one. But this fear seems at odds with another common view, that economies are driven by businesses, which are driven by ‘pragmatic’ men. It might be that without the endorsement of some (wrong) theory some practices, such as the development of novel technical instruments and the use of large leverages, would be curtailed. But would this be a bad thing?

Nonetheless, Arrow’s challenge deserves a response.

There are many variations in detail of utility maximization theories. Suppose we identity ‘utility maximization’ as a possible heuristic, then utility maximization theory claims that people use some specific heuristics, so an obvious alternative is to consider a wider  range. The implicit idea behind utility maximization theory seems to be under a competitive regime resembling evolution, the evolutionary stable strategies (‘the good ones’) do maximize some utility function, so that in time utility maximizers ought to get to dominate economies. (Maybe poor people do not maximize any utility, but they – supposedly – have relatively little influence on economies.) But this idea is hardly credible. If – as seems to be the case – economies have significant ‘Black Swans’ (low probability high impact events) then utility maximizers  who ignore the possibility of a Black Swan (such as a crash) will do better in the short-term, and so the economy will become dominated by people with the wrong utilities. People with the right utilities would do better in the long run, but have two problems: they need to survive the short-term and they need to estimate the probability of the Black Swan. No method has been suggested for doing this. An alternative is to take account of some notional utility but also take account of any other factors that seem relevant.

For example, when driving a hire-car along a windy road with a sheer drop I ‘should’ adjust my speed to trade time of arrival against risk of death or injury. But usually I simply reduce my speed to the point where the risk is slight, and accept the consequential delay. These are qualitative judgements, not arithmetic trade-offs. Similarly an individual might limit their at-risk investments (e.g. stocks) so that a reasonable fall (e.g. 25%) could be tolerated, rather than try to keep track of all the possible things that could go wrong (such as terrorists stealing a US Minuteman) and their likely impact.

More generally, we could suppose that people act according to their own heuristics, and that there are competitive pressures on heuristics, but not that utility maximization is necessarily ‘best’ or even that a healthy economy relies on most people having similar heuristics, or that there is some stable set of ‘good’ heuristics. All these questions (and possibly more) could be left open for study and debate. As a mathematician it seems to me that decision-making involves ideas, and that ideas are never unique or final, so that novel heuristics could arise and be successful from time to time. Or at least, the contrary would require an explanation. In terms of game theory, the conventional theory seems to presuppose a fixed single-level game, whereas – like much else – economies seem to have scope for changing the game and even for creating higher-level games, without limit. In this case, the strategies must surely change and are created rather than drawn from a fixed set?

See Also

Some evidence against utility maximization. (Arrow’s response prompted this post).

My blog on reasoning under uncertainty with application to economics.

Dave Marsay

The limits of (atomistic) mathematics

Lars Syll draws attention to a recent seminar on ‘Confronting economics’ by Tony Lawson, as part of the Bloomsbury Confrontations at UCLU.

If you replace his every use of the term ‘mathematics’ by something like ‘atomistic mathematics’ then I would regard this talk as not only very important, but true. Tony approving quotes Whitehead on challenging implicit assumptions. Is his implicit assumption that mathematics is ‘atomistic’? What about Whitehead’s own mathematics, or that of Russell, Keynes and Turing? He (Tony) seems to suppose that mathematics can’t deal with emergent properities. So What is Whitehead’s work on Process, Keynes’ work on uncertainty, Russell’s work on knowledge or Turing’s work on morphogenesis all about?

Dave Marsay

 

Haldane’s Tails of the Unexpected

A. Haldane, B. Nelson Tails of the unexpected,  The Credit Crisis Five Years On: Unpacking the Crisis conference, University of Edinburgh Business School, 8-9 June 2012

The credit crisis is blamed on a simplistic belief in ‘the Normal Distribution’ and its ‘thin tails’, understating risk. Complexity and chaos theories point to greater risks, as does the work of Taleb.

Modern weather forecasting is pointed to as good relevant practice, where one can spot trouble brewing. Robust and resilient regulatory mechanisms need to be employed. It is no good relying on statistics like VaR (Value at Risk) that assume a normal distribution. The Bank of England is developing an approach based on these ideas.

Comment

Risk arises when the statistical distribution of the future can be calculated or is known. Uncertainty arises when this distribution is incalculable, perhaps unknown.

While the paper acknowledges Keynes’ economics and Knightian uncertainty, it overlooks Keynes’ Treatise on Probability, which underpins his economics.

Much of modern econometric theory is … underpinned by the assumption of randomness in variables and estimated error terms.

Keynes was critical of this assumption, and of this model:

Economics … shift[ed] from models of Classical determinism to statistical laws. … Evgeny Slutsky (1927) and Ragnar Frisch (1933) … divided the dynamics of the economy into two elements: an irregular random element or impulse and a regular systematic element or propagation mechanism. This impulse/propagation paradigm remains the centrepiece of macro-economics to this day.

Keynes pointed out that such assumptions could only be validated empirically and (as the current paper also does) in the Treatise he cited Lexis’s falsification.

The paper cites a game of paper/scissors/stone which Sotheby’s thought was a simple game of chance but which Christie’s saw  as an opportunity for strategizing – and won millions of dollars. Apparently Christie’s consulted some 11 year old girls, but they might equally well have been familiar with Shannon‘s machine for defeating strategy-impaired humans. With this in mind, it is not clear why the paper characterises uncertainty a merly being about unknown probability distributions, as distinct from Keynes’ more radical position, that there is no such distribution. 

The paper is critical of nerds, who apparently ‘like to show off’.  But to me the problem is not the show-offs, but those who don’t know as much as they think they know. They pay too little attention to the theory, not too much. The girls and Shannon seem okay to me: it is those nerds who see everything as the product of randomness or a game of chance who are the problem.

If we compare the Slutsky Frisch model with Kuhn’s description of the development of science, then economics is assumed to develop in much the same way as normal science, but without ever undergoing anything like a (systemic) paradigm shift. Thus, while the model may be correct most of the time,  violations, such as in 2007/8, matter.

Attempts to fine-tune risk control may add to the probability of fat-tailed catastrophes. Constraining small bumps in the road may make a system, in particular a social system, more prone to systemic collapse. Why? Because if instead of being released in small bursts pressures are constrained and accumulate beneath the surface, they risk an eventual volcanic eruption.

 One can understand this reasoning by analogy with science: the more dominant a school which protects its core myths, the greater the reaction and impact when the myths are exposed. But in finance it may not be just ‘risk control’ that causes a problem. Any optimisation that is blind to the possibility of systemic change may tend to increase the chance of change (for good or ill) [E.g. Bohr Atomic Physics and Human Knowledge. Ox Bow Press 1958].

See Also

Previous posts on articles by or about Haldane, along similar lines:

My notes on:

Dave Marsay

NRC’s Assessing … Complex Models

Committee on Mathematical Foundations of Verification, Validation, and Uncertainty Quantification Board on Mathematical Sciences and Their Applications Assessing the Reliability of Complex Models: Mathematical and Statistical Foundations of Verification, Validation, and Uncertainty Quantification (US) NRC, 2012

The team were tasked to “examine practices for VVUQ of large-scale computational simulations”. Such simulations are complicated. The title seems misleading in using the term ‘complex’. The summary seems like a reasonable consensus summary of the state of the art in its focus area, and of research directions, with no surprises. But the main body does provide some ammunition for those who seek to emphasise deeper uncertainty issues, considering mathematics beyond computation.

Summary

Principles

Highlighted principles include:

    1. A validation assessment is well defined only in terms of specified quantities of interest (QOIs) and the accuracy needed for the intended use of the model.
    2. A validation assessment provides direct information about model accuracy only in the domain of applicability that is “covered” by the physical observations employed in the assessment.

Comments

The notion of a model here would be something like ‘all swans are white’. The first principle suggests that we need tolerance for what is regarded as ‘white’. The second principle suggests that if we have only considered British swans, we should restrict the domain of applicability of the model.

In effect, the model is being set within a justification, much as the conclusion of a mathematical theorem is linked to axioms by the proof. This is contrary to much school science practice, which simply teaches models: we need to understand the (empirical) theory. Typically, when we read ‘all swans are white’ we should understand that it really only means ‘all British swans are white-ish’.

Swans are relatively simple. The only problem is our limited observations of them. Economics, for example, is more complex. The quantities of interest are controversial, as are the relevant observations. Such complex situations seem beyond the intended scope of this report.

Research Topics

  1. Development of methods that help to define the “domain of applicability” of a model, including methods that help quantify the notions of near neighbors, interpolative predictions, and extrapolative predictions.
  2. Development of methods to assess model discrepancy and other sources of uncertainty in the case of rare events, especially when validation data do not include such events.

Comments

These topics are easier if one has an overarching theory of which the model is a specialisation, whose parameters are to be determined. In such cases the ‘domain of applicability’ could be based on an established classifying schema, and uncertainty could be probabilistic, drawing on established probabilistic models. The situation is more challenging, with broader uncertainties, where there is no such ruling theory, as in climate science.

Recommendations

  1. An effective VVUQ [verification, validation and uncertainty quantification] education should encourage students to confront and reflect on the ways that knowledge is acquired, used, and updated.
  2. The elements of probabilistic thinking, physical-systems modeling, and numerical methods and computing should become standard parts of the respective core curricula for scientists, engineers, and statisticians.

Comments

Most engineers and statisticians will be working pragmatically, assuming some ruling theory that guides their work. This report seems most suitable for them. Ideally, scientists acting as science advisors would also be working in such a way. However, surprises do happen, and scientists working on science should be actively doubting any supposed ruling theory. Thus it is sometimes vital to know the difference between a situation where an agreed theory should be regarded as, for example, ‘fit for government work’, and where it is not, particularly where extremes of complexity or uncertainty call for a more principled approach. In such cases it is not obvious that uncertainty can be quantified. For example, how does one put a number on ‘all swans are white’ when one has not been outside Britain?

As well as using mathematics to work out the implications of a ruling theory in a particular case, one needs to be able to use different mathematics to work out the implications of a particular case for theory.

Introduction

This cites Savage,  but in his terms it is implicitly addressing complicated but ‘small’ worlds rather than more complex ‘large’ ones, such as that of interest to climate science.

Sources of Uncertainty and Error

The general issue is whether formal validation of models of complex systems is actually feasible. This issue is both philosophical and practical and is discussed in greater depth in, for example, McWilliams (2007), Oreskes et al. (1994), and Stainforth et al. (2007).

There is a need to make decisions … before a complete UQ analysis will be available. … This does not mean that UQ can be ignored but rather that decisions need to be made in the face of only partial knowledge of the uncertainties involved. The “science” of these kinds of decisions is still evolving, and the various versions of decision analysis are certainly relevant.

Comment

 It seems that not all uncertainty is quantifiable, and that one needs to be able to make decisions in the face of such uncertainties.

In the case of ‘all swans are white’ the uncertainty arises because we have only looked in Britain. It is clear what can be done about this, even if we have no basis for assigning a number.

In the case of economics, even if we have a dominant theory we may be uncertainty because, for example, it has only been validated against the British economy for the last 10 years. We might not be able to put a number on the uncertainty, but it might be wise to look for more general theories, covering a broader range of countries and times, and then see how our dominant theory is situated within the broader theory. This might give us more confidence in some conclusions from the theory, even if we cannot assign a number. (One also needs to consider alternative theories.)

Model Validation and Prediction

Comparison with reality

In simple settings validation could be accomplished by directly comparing model results to physical measurements for the QOI  …

Findings

  1. Mathematical considerations alone cannot address the appropriateness of a model prediction in a new, untested setting. Quantifying uncertainties and assessing their reliability for a prediction require both statistical  and subject-matter reasoning.
  2. The idea of a domain of applicability is helpful for communicating the conditions for which predictions (with uncertainty) can be trusted. However, the mathematical foundations have not been established for defining such a domain or its boundaries.

Comment

I take the view that a situation that can be treated classically is not complex, only at most complicated. Complex situations may always contain elements that are surprising to us. Hence bullet 1 applies to complex situations too. The responsibility for dealing with complexities seems to be shifted from the mathematicians to the subject matter experts (SMEs). But if one is dealing with a new ‘setting’ one is dealing with dynamic complexity, of the kind that would be a crisis if the potential impact were serious. In such situations it may not be obvious which subject is the relevant one, or there may be more than one vital subject. SMEs may be unused to coping with complexity or with collaboration under crisis or near-crisis conditions. For example, climate science might need not only climatologists but also experts in dealing with uncertainty.

My view is that sometimes one can only assess the relevance and reliability of a model in a particular situation, that one needs particular experts in this, and that mathematics can help – but it is a different mathematics.

Next Steps in Practice, Research, and Education for Verification, Validation, and Uncertainty Quantification

 For validation, “domain of applicability” is recognized as an important concept, but how one defines this domain remains an open question. For predictions, characterizing how a model differs from reality, particularly in extrapolative regimes, is a pressing need. … advances in linking a model to reality will likely broaden the domain of applicability and improve confidence in extrapolative prediction.

Comment

As Keynes pointed out, in some complex situations one can only meaningfully predict in the short-term. Thus in early 2008 economic predictions were not in error, as short-term predictions. It is just that the uncertain long-term arrived. What is needed, therefore, is some long-term forecasting ability. This cannot be a prediction, in the sense of having a probability distribution, but it might be an effective anticipation, just as one might have anticipated that there were non-white swans in foreign parts. Different mathematics is needed.

My Summary

The report focusses on the complicatedness of the models. But I find it hard to think of a situation where one needs a complicated model and the actual situation is not complex. Usually, for example, the situation is ‘reflexive’ because the model is going to be used to inform interaction with the world, which will change it. Thus, the problem as I see it is how to model a situation that is uncertain and possibly complex. While the report does give some pointers it does not develop them.

The common sense view of modelling is that a model is based on observations. In fact – as the report notes – it tends to be based on observations plus assumptions, which are refined into a model, often iteratively. But the report seems to suppose that one’s initial assumptions will be ‘true’. But one can only say that the model fits one’s observations, not that it will continue to fit all possible observations, unless one can be sure that the situation is very constrained. That is, one cannot say that a scientific theory is unconditionally and absolutely true, but only ‘true to’ ones observations and assumptions.

The report is thus mainly for those who have a mature set of assumptions which they wish to refine, not those who expect the unexpected. It does briefly mention ‘rare events’, but it sees these as outliers on a probability distribution whereas I would see these more as challenging assumptions.

See Also

The better nature blog provides a view of science that is complimentary to this report.

My notes on science and uncertainty.

Dave Marsay

Making your mind up (NS)

Difficult choices to make? A heavy dose of irrationality may be just what you need.

Comment on a New Scientist article, 12 Nov. 2011, pg 39.

The on-line version is Decision time: How subtle forces shape your choices: Struggling to make your mind up? Interpret your gut instincts to help you make the right choice.

The article talks a lot about decision theory and rationality. No definitions are given, but it seems to be assumed that all decisions are analogous to decisions about games of chance. It is clearly supposed, without motivation, that the objective is always to maximize expected utility. This might make sense for gamblers who expect to live forever without ever running out of funds, but more generally is unmotivated.

Well-known alternatives include:

  • taking account of the chances of going broke (short-term) and never getting to the ‘expected’ (long-term) returns.
  • taking account of uncertainty, as in the Ellsberg’s approach.
  • taking account of the cost of evaluating options, as in March’s ‘bounded rationality’.

The logic of inconsistency

A box claims that ‘intransitive preferences’ give mathematicians a head-ache. But as a mathematician I find that some people’s assumptions about rationality give me a headache, especially if they try to force them on to me.

Suppose that I prefer apples to plums to pears, but I prefer a mixture to having just apples. If I am given the choice between apples and plums I will pick apples. If I am then given the choice between plums and pears I will pick plums. If I am now given the choice between apples and pears I will pick pears, to have a good spread of fruit. According to the article I am inconsistent and illogical: I should have chosen apples. But what kind of logic is it in which I would end up with all meat and no gravy? Or all bananas and no custard?

Another reason I might pick pears was if I wanted to acquire things that appeared scarce. Thus being offered a choice of apples or plums suggests that neither are scarce, so what I really want is pears. In this case, if I was subsequently given a choice of plums to pears I would choice pears, even though I actually prefer plums. An question imparts information, and is not just a means of eliciting information.

In criticising rationality one needs to consider exactly what the notion of ‘utility’ is, and whether or not it is appropriate.

Human factors

On the last page it becomes clear that ‘utility’ is even narrower than one might suppose. Most games of chance have an expected monetary loss for the gambler and thus – it seems – such gamblers are ‘irrational’. But maybe there is something about the experience that they value. They may, for example, be developing friendships that will stand them in good stead. Perhaps if we counted such expected benefits, gambling might be rational. Could buying a lottery ticket be rational if it gave people hope and something to talk about with friends?

If we expect that co-operation or conformity  have a benefit, then could not such behaviours be rational? The example is given of someone who donates anonymously to charity. “In purely evolutionary terms, it is a bad choice.” But why? What if we feel better about ourselves and are able to act more confidently in social situations where others may be donors?

Retirement

“Governments wanting us to save up for retirement need to understand why we are so bad at making long-term decisions.”

But are we so very bad? This could do with much more analysis. With the article’s view of rationality under-saving could be caused by a combination of:

  • poor expected returns on savings (especially at the moment)
  • pessimism about life expectancy
  • heavy discounting of future value
  • an anticipation of a need to access the funds before retirement
    (e.g., due to redundancy or emigration).

The article suggests that there might also be some biases. These should be considered, although they are really just departures from a normative notion of rationality that may not be appropriate. But I think one would really want to consider broader factors on expected utility. Maybe, for example, investing in one’s children’s’ future may seem a more sensible investment. Similarly, in some cultures, investing one’s aura of success (sports car, smart suits, …) might be a rational gamble. Is it that ‘we’ as individuals are bad at making long-term decisions, or that society as a whole has led to a situation in which for many people it is ‘rational’ to save less than governments think we ought to have? The notion of rationality in the article hardly seems appropriate to address this question.

Conclusion

The article raises some important issues but takes much too limited a view of even mathematical decision theory and seems – uncritically – to suppose that it is universally normatively correct. Maybe what we need is not so much irrationality as the right rationality, at least as a guide.

See also

Kahneman: anomalies paper , Review, Judgment. Uncertainty: Cosimedes and Tooby, Ellsberg. Examples. Inferences from utterances.

Dave Marsay

The End of a Physics Worldview (Kauffman)

Thought provoking, as usual. This video goes beyond his previous work, but in the same direction. His point is that it is a mistake to think of ecologies and economies as if they resembled the typical world of Physics. A previous written version is at npr, followed by a later development.

He builds on Kant’s notion of wholes, noting (as Kant did before him) that the existence of such wholes is inconsistent with classical notions of causality.  He ties this in to biological examples. This complements Prigogine, who did a similar job for modern Physics.

Kauffman is critical of mathematics and ‘mathematization’, but seems unaware of the mathematics of Keynes and Whitehead. Kauffman’s view seems the same as that due to Bergson and Smuts, which in the late 1920s defined ‘modern science’. To me the problem behind the financial crash lies not in science or mathematics or even in economics, but in the brute fact that politicians and financiers were wedded to a pre-modern (pre-Kantian) view of economics and mathematics. Kauffman’s work may help enlighten them on the need, but not on the potential role for modern mathematics.

Kauffman notes that at any one time there are ‘adjacent possibles’ and that in the near future they may come to pass, and that – conceptually – one could associate a probability distribution with these possibilities. But as new possibilities come to pass new adjacent possibilities arise. Kauffman supposes that it is not possible to know what these are, and hence one cannot have a probability distribution, much of information theory makes no sense, and one cannot reason effectively. The challenge, then, is to discover how we do, in fact, reason.

Kauffman does not distinguish between short and long run. If we do so then we see that if we know the adjacent possible then our conventional reasoning is appropriate in the short-term, and Kauffman’s concerns are really about the long-term: beyond the point at which we can see the potential possibles that may arise. To this extent, at least, Kauffman’s post-modern vision seems little different from the modern vision of the 1920s and 30s, before it was trivialized.

Dave Marsay