Complexity, causality, sense making and strategy
How does our conception of the complexity of the world influence how we should approach it?
Commonsense theories of action, such as seeking to maximise utility while taking account of risk, presuppose a lot about the real world and our ability to make the appropriate assessments. Such notions seem questionable, both in theory and in practice. It has been noted that the world and our knowledge of it are complex, and that our activity in the world needs to reflect this complexity. But there seems little agreement on what complexity is, much less how to act effectively in complex situations. It seems useful, therefore, to identify some different types of complexity in the hope that we can identify sound ways of acting within a particular type of complex situation.
Types of Complexity
Complexity involves inter-dependency, as against a things being constructed sequentially.
A situation is relatively simple when it has a state with a dynamics such that:
- In principle, from an approximation to its current state one can predict the future state with an accuracy that is well-behaved (‘linear’) with respect to the accuracy of the initial state and the extent of the prediction. Prediction is just extrapolation.
- Possible behaviours can be classified in such a way that all possible sequences of behaviours can be seen as permutations of elements drawn from a fixed palette, albeit perhaps with random or pseudo-random variation.
Thus behaviours can be seen as ‘coded up’ in the dynamic, and simply unfolding from a given initial state, with time, like an idealised machine. Small changes have small impact.
Even with such simple systems there are complexities. For example the ‘three body problem’, where three masses move under mutual influence, is difficult. But such systems are ‘mechanistic’ and typically lend themselves to good approximations.
Deterministic, State-based worlds
We often view what happens as a consequence of dynamical laws of nature acting on the state of the world. Complexity can be generated by bounded divergent dynamics (where a small change can make a big difference that is recycled), making the future unpredictable in detail. However there are typically attractors, perhaps ‘strange attractors’, to which in the long run states tend to converge. The situation resembles the weather, which can be forecast in the short term (based on imprecise measurements) and which (absent global warming) tends to repeat familiar patterns endlessly, if somewhat unpredictably.
Here the attractors, such as cyclones, typically correspond to complexes and can mostly be dealt with as interacting objects. Where more complex emergence is an issue, it is mostly routine.
Probabilistic dynamics, State-based worlds
If the dynamics are probabilistic, the very strangest behaviours tend to get ‘smeared out’, but the overall result is the same: one has attractors that may not be points, one can forecast in the short-term, and can identify repeating patterns in the long term. Complexes tend to be relatively simple and comprehensible with repeating behaviours.
This and the previous are the most complex situation that many people are relatively comfortable thinking about. They are often assumed to be quite general. This model may be preferred for practical reasons (it may be easier than the former).
Alternatively, the state may be itself be a probability distribution (as in quantum mechanics) which has a deterministic dynamics. What is measured in a sample from the probability distribution. One thus has a deterministic state-based world, but limited observability of any supposed underlying state and hence an uncertainty principle which means that the more precise one can be about the underlying state, the less precise one can be about the dynamic, and vice-versa.
Complexes arise as local equilibria in interdependent conditional probability distributions. They thus tend to be regular and repeating in character.
Summary of regular complexes
Minimal complexes unfold from the dynamic in familiar ways. The life-cycle of a complex may not be calculable, but it will be statistical regular. There is no meaningful emergence, in the sense of genuine novelty.
Kant introduces to classical concepts the notion of emergence, as the appearance of an overlooked (or previously absent) influence. This particularly applies to communities, where influence may be reciprocal. This suggests that fundamental change is always possible: what have always been attractors may suddenly fail to be attractors.
One can view planetary systems, the weather and particle physics as having communities and emergence, so that Kant is simply providing a unifying taxonomy. But one can also have situations in which there is continual genuine novelty, as in evolution. There may not be common enduring patterns that unfold from the dynamics. One can have disorder, instability, genuine diversity and disequilibrium.
Whitehead shows that activity can be considered in stable epochs, typically bounded (e.g. in time and space). Each epoch has its own logic, and hence values and probability distributions. Thus Kantian emergence is a change of epoch. Such changes are not necessarily arbitrary: one can have temporarily heighted potential for change – the ‘nexus’.
Whitehead adds to Kant a notion of structure. Each epoch may have elements that appear random, but which may be a ‘tactical’ epoch in its own right. Similarly, successive epochs may be part of the same ‘strategic’ supra-epoch, so that sudden changes of ‘the rules’ are not arbitrary but are subject to superior rules. Epochs and their rules are associated with self-reinforcing process cycles, and constitute something like ‘games’.
Whitehead’s epochs correspond to significant complexes. One can have order, stability and equilibrium amidst disorder, instability and disequilibrium, and vice-versa. But also genuine diversity may be the key to the stability of a complex, and too little can undermine it.
Prigogine describes operators acting on operators, in what may be seen as generalising quantum mechanics’ probability distributions and as an interpretation of Whitehead, with explicit models of complexes and of emergence. Causality is classical or probabilistic within an epoch, but from time to time new structures can emerge that can alter ‘the rules of the game’ at higher or lower levels. Even with full knowledge, it cannot be predicted ahead of time whether at a nexus a new epoch will emerge or the system will descend into chaos. Thus emergence appears to be spontaneous.
Prigogine treats ‘chaos’ as a special case, but we can think of it as a limiting case, when we have a ‘foam’ of epochs, too small to be made sense of.
Prigogine shows how macro-scale trends create the conditions under which the micro-scale matters. Moreover, the micro-scale cannot be represented by a point or distribution in a fixed landscape, or as a bounded set of something: instead there is scope for genuine innovation.
We should both watch out for it, and seek opportunities. We should also, strategically, seek to manage the opportunities for innovation and other micro-scale changes to have an impact.
Prigogine cites Turing’s work on morphogenesis, which was the first to model the emergence of (relatively static) complexity. Turing was a grand-student of Whitehead whose fellowship to King’s was backed by Keynes.
Strictly speaking a world in which novelty continually emerged but where the new things had no impact on the existing things would be complex. We are typically interested in situations where emergence of novelty can lead to the collapse of established systems and networks, so we might (as some do) reserve the terms ‘complex’ for such cases. But then we are also interested in cases where evolutionary pressure favours systems that are resilient, ready to work together and perhaps ‘altruistic’. We would not wish to exclude such cases.
Complexity and causality
Something is said to be the cause of something else when the second thing would not have happened without the former. More generally, something is said to be causal if the probability of its happening would typically have been less.
Deterministic causality is straightforward, simpler than a mouse springing a trap. Behaviour is comprehensible and potentially knowable in the short term and in the long-term pseudo-randomised over attractors. Such attractors are generally identifiable in principle and so apparent causality is subjectively probabilistic. As in weather forecasting the mid-term is a more problematic situation. The delineation of the mid-term can also be problematic, and may vary dramatically with both state and time. Such a mid-term is often regarded as a time of chaos that one just has to muddle through.
Probabilistic causality is like a drunken mouse springing an unreliable trap. ‘Real’ causality is probabilistic in all time frames, but in the mid-term it is still impossible to determine useful probability distributions, unless the dynamics are so random as to ‘smear out’ the complexity of the dynamics. Where the ‘state’ is itself a probability distribution one imagines that real causality, within the world of probability distributions, is governed by the dynamic, but has the difficulty of only observing samples. There is no causality as such between samples, only between probability distributions. But one does still have universal laws. The classical concepts of stability, order, uniformity and equilibrium are modified, but still present.
Kant recognises that some things just seem to happen, so that one seems to have ‘uncaused causes’. He supposes that one always has a chain of causes going back in time, but that such chains may have times that are bounded below. Thus an event now may have no cause in the events of yesterday. He identifies this type of causality with communities, inter-dependent entities: complexes.
We can resolve Kant’s view with the world of dynamics by supposing that sometimes a new dynamic is created. Thus the dynamic is not a static thing, but may be changed, such as when someone invents something. It is implicit that laws are local and temporary, not universal.
For Whitehead, a world of regular dynamics is an epoch. The way that an epoch is changed may be subject to the dynamics of its supra-epoch. Alternatively, an epoch may move to a condition which upsets the viability of a supra-epoch, or which gives it a positive opportunity to change.
An example is a hierarchical organisation in which each part behaves routinely, until there is a need for change which may have a cascading impact.
Thus one has causality due to the current within-epoch dynamics, plus that which is a response to changes in supra or sub epochs.
Prigogine gives a mathematization of Whitehead that is much broader than classical models. Instead of thinking as people within an organization as having states and following rules, we may think of people as more complex ‘operators’, potentially with their own concerns and with scope for innovation. They don’t just follow rules, but may seek ways around any problems, perhaps showing more insight than we, the modellers, have.
Causality is associated with epochs, and may be either routine (applying the existing dynamics) or an innovation, which may change the structure. Such innovations can not always be represented as taking place within some pre-existing epoch, but may be genuinely novel.
Complexity and sense making
A situation is made sense of when one understands enough about it to make a decision. This may be just establishing some parameter or ‘the state’, or may be more challenging. Sense making depends on observation. Rather than develop a theory of a system being observed we can consider a larger system, including the observing. Psychologically we may only consider sense to have been made when there is no significant residual uncertainty, but here we allow the sense made to include an appreciation of the residual uncertainty.
A tamed situation is one that has fully been made sense of. In the simple case the evidence arises from a definite dynamic and a prior state. The interpretation of evidence as a likelihood is thus an inverse problem: would the hypothesized state have produced the evidence? There may be many states that would have produced a single piece of evidence, but as more evidence is collected one can determine which states were credible at some reference time, and thus identify a reducing set of credible states. The evidence is said to be diagnostic if this set reduces to a single candidate, which may then be said to be ‘true’. More generally, even in this simple case, one can only say that some one of a set of credible hypotheses is ‘true’. Thus uncertainty, if present, is possibilistic in the short term.
In ‘chaotic’ systems we may not practically be able to determine which states would have led to the evidence. This may spoil the diagnosticity of the evidence beyond some ‘event horizon’. Beyond this, we may necessarily be subjectively uncertain about the original state even if it is actually determined.
Where the dynamic is probabilistic the likelihood is given directly by the probabilistic dynamic. For a novel situation, the likelihood may be all that can be said about uncertainty. If the system has reached some sort of equilibrium then we may be able to estimate a ‘prior’ probability and hence (via Bayes’ rule) a probability distribution.
Where the states of the system are probability distributions we can either consider evidence production to be a part of the dynamic of the system (extended if necessary) or consider it to be a separate sampling process. One needs to be careful to distinguish between the state and likelihoods, both of which are probability distributions, but this is complicated rather then complex.
In the simplest cases problems can be solved. There are no real tactics or strategy in any meaningful sense, and sense making can be routine. More generally one cannot be sure of the impact of one’s activity. A pragmatic approach is to model (make sense of) the world as best one can, identify short-term utilities (measure of merit etc) and goals, and to adapt the model, utilities or goals, when particular problems arise. Thus one distinguishes between ‘good’ and ‘bad’, supporting the former while inhibiting the latter. Too much support or inhibition may be more effective than is needed, and may be inefficient. Thus even if one is not clear about the details of the situation, one can generally start to act and adjust or titrate as required. Sense making can be routine. If a situation involves genuine probability distributions that are broad enough, then it may not be possible to do any better. The uncertainty principle means that a given amount of sense making effort has to be split between levels, with continual engagement at both ‘the current situation’ and the ‘bigger picture’. For all variants, the scope for non-routine, strategic, sense making lies in situations that involve more complex complexes.
In Kantian worlds the dynamic can suddenly change. We can determine a likelihood conditioned on what we think the dynamic is, but we should also consider the dynamic as being variable and hence look for evidence about the actual dynamic. If the evidence is impossible (or extremely unlikely) for what we think is the current dynamic then we may consider this to be evidence that the dynamic has changed. Thus we may consider evidence ’surprising’ when not only it had been considered improbable but also it suggests an emergent dynamic.
Given that a new dynamic may emerge imperceptibly, it may also be necessary to re-assess old evidence in the light of a newly detected dynamic: non-monotonic reasoning.
Total uncertainty can be divided into two parts. The first is about the interpretation assuming that the epoch is persisting, the second is that the epoch is persisting. This latter will typically consider scenarios corresponding to anticipated new epochs, or a chaotic regime.
Classical information theory concerns the interpretation of evidence within a set context. In times of emergence information about the changing context may be more important than information about the ‘state’ within the assumed context.
The emergence of novelty means that what was helpful may become unhelpful, and vice-versa. Thus one cannot simply view things as ‘good’ (to be supported) and ‘bad’ (to be suppressed). Sense making involves more judgement than that. Moreover, providing excessive support or inhibition to something may backfire when things change. One approach is to have short-term goals, but to also watch out for emergence, or opportunities for emergence, particularly of new communities that may upset the stability. Thus sense-making should always be outward looking and open to novelty, never closed.
One may seek to inhibit or foster new communities or new behavioural complexes within communities in order to safeguard or overcome a status quo. Either way, tactical outcomes can lead to a need to re-think strategic conditions. Where different communities may be in competition (as is often the case) the eventual dominance of a particular group of compatible communities form a possible scenario to be considered strategically, each scenario typically being stable etc. Looking ahead, then, one sees a set of possible scenarios, each with its own values and probabilities. If one considers such scenarios probabilistically then the probability is very conditional, even reflexive: it depends on you. Thus sense-making is largely about communities and potential, embracing ambiguity.
Whitehead shows how evidence needs to be considered on multiple epochs or ‘levels’. For a classical approach to apply, an epoch needs to have stabilised. Thus if the rules of a higher-level game is changing or has changed, one needs to reconsider how one interprets lower-level evidence and may need different methods, typically seeking to identify a higher-level epoch that is stable, and focussing on those aspects of the evidence that can still be interpreted reasonably.
Whitehead’s student, Keynes, has given formulae, developed by Turing and Good, which show how to interpret evidence against hypotheses, and hence notions of weight of evidence, likelihood and information. These assume the law of large numbers for the hypotheses. That is, any results are clearly conditioned on assumptions about epoch. The apparent failure of the law of large numbers (‘the data going haywire’) is an indication that there may be a new epoch. A key part of operating under unstable conditions is identifying which measures / statistics will be relatively reliable. If one is working at a given level of abstraction, in a given epoch, one is ignoring both the detail of the sub-epochs and the context of the supra-epoch. The detail is only logically consistent if one regards it as ‘random’. As soon as one pays attention to it in its own right one can get logical contradictions, because each epoch has its own logic, constrained by but not determined by the other epochs. As one operates at different levels one trades off detail against context, giving rise to an uncertainty principle analogous to Heisenberg’s.
Whitehead shows that activity can be considered in stable epochs, typically bounded in time and space. Each epoch has its own logic, and hence values and probability distributions, and hence its own tactics. An epoch may have elements that appear random, but each may be a ‘tactical’ epoch in its own right. It is a choice we make as to whether to treat something as random, or whether we try to understand it, perhaps engage with it. Similarly, successive epochs may be part of the same ‘strategic’ supra-epoch, so that sudden changes of ‘the rules’ are not arbitrary but are subject to superior rules. Epochs and their rules are associated with self-reinforcing process cycles, and constitute something like ‘games’. We can thus see the distinction between tactics and strategy. Sense making seeks to understand these ‘levels’ and the links and potential links between them.
This suggests that as well as trying to establish the rules of the current ‘game’ and considering alternative possibilities in terms of communities, one needs to consider the more general rules that could endure, and not just accept an given ideology as a fixed part of one’s culture. Elements within an epoch that have seemed to be random may become correlated, e.g. when previously separate communities commune), which could undermine the status quo. Thus, strategically, one needs to watch out for, or encourage, such things.
Working at different levels of abstraction is a trade-off between detail and context. One wants to identify the best compromise as ‘the level of the fight’, while still paying attention to adjacent levels.
Prigogine shows how macro-scale trends create the conditions under which the micro-scale matters, and so novel emergence can occur. We should both watch out for it, and seek opportunities. We should also, strategically, seek to manage the opportunities for innovation and other micro-scale changes to have an impact.
It is often thought that the higher the level of abstraction one works at, the greater the time-frame before interventions can have an impact. Whilst this is generally so, if one can identify ‘the level of the fight’ and shape conditions to create a nexus, an intervention (or accident) may have an effect that quickly creates a new structure, cutting across the old ones. One should be on the look-out for both opportunities and threats.
Complexity and strategy
In this part we compare the above insights into the ‘strategic viewpoints’ associated with complexity with more action-oriented accounts of coping with complexity.
Moving from a status quo
The ideal is typically to move from order to order without ‘losing control’. An established status quo, from the viewpoint of tameable complexities, is just about identifying the dynamic and changing the state so that a more desired state will follow. In complex situation Cynefin recommends an approach of probe-sense-respond, where the probes are ‘safe to fail’ so that if things do not work out one can recover to stability. The concern, then, is to avoid stimulating ‘out of control’ emergence.
For tameable complexities a probe can be something that one thinks will have a desired effect but which one uses tentatively, checking for error or divergent dynamics. In effect, complexity has merely contributed to uncertainty, which necessitates caution.
For more challenging complexities, Kant shows that while our probes might disturb existing communities we do not want them to change the communal structure. Whitehead notes the importance of rules and (with Keynes) expectation, and that there are generally multiple levels to consider. Our probes will often be highly disruptive on some scale, so we need to be clear about the levels that matter, and about contagion between them. Sometimes it will be imposible to change the status quo in an orderly fashion as viewed from its level. It will be necessary to identify a higher level vantage point from which a transition can be seen to be relatively orderly.
Prigogine shows how macro-scale changes can set the scene for micro-scale issues to erupt emergently at higher levels in a non-classical way. Those probing will need to be aware of any such nexus, or to probe for such before attempting more macro-scale probes.
Thus while probe-sense-respond remains appropriate as a strategic description, the nature of probing, sensing and responding is very different between tameable and more wicked complexities.
Sometimes one starts with an unstable situation, which one seeks to stabilize. Even from a tameable view this will rarely mean trying to take the system back to a previous, apparently stable, state, as the problems that led to the instability are likely to recur. Instead one seeks to identify a new potentially stable state.
From a tameable point of view one has a predisposition to suppose that one knows all the attractors, or at least all the acceptable ones. Attractors and their basins can be labelled relatively good or bad. One tries to nudge the state into the basins of better attractors.
From a Kantian perspective one also needs to worry about identifying potentially critical actual and potential communities, and to encourage or inhibit them. One also needs to look out for novel emergence, and to consider its merits in terms of the situation as it actually is (as distinct from how you thought it would be when you made your plans). Considering Whitehead, one needs to consider rules and other behavioural traits as well as communities. To achieve stability at a given level one may first need to achieve more stability – or perhaps less – at levels above and below. Thus while strategically one may be seeking to stabilise, tactically one may need to destabilise, or vice-versa. In addition, Prigogine shows how one needs to watch out for any nexus, where effects can cascade horizontally or vertically.
Complicatedness and bounded rationality
Dictionary definitions often confuse complexity and complicatedness. Both challenge our thinking, but something that is just complicated could be understood and described in principle, but there are problems of scale. Complexity, however, has more fundamental challenges. Thus we hope that an aircraft is complicated but not complex. In terms of bounded rationality, there are clearly bounds on the degree of complicatedness of objects or networks that can be comprehended, described and reasoned about. But there may also be more fundamental bounds, such as if we were constrained to view things as networks of objects when they are actually more complex.
Levels of abstraction
The levels in Whitehead’s theory are similar to levels of abstraction but, just as in holistic theory biology is not derived from physics, or in Kant where our conception of objects are not just abstracted from any reality, the higher levels have content of their own, which are not derived from the lower levels. If one works solely at some supposedly real level one is missing something out.
As WR Ashby noted, multi-level games have a similar structure to Whitehead’s epochs. New tactics can arise as the result of new teams forming or of practice from different areas coming together synergistically. New plays can result in rule changes, and rule changes can result in new plays. Thus one has influence up and down the epochs. Also, teams could get bored with an existing game and develop a new variant. A situation where the rules are fixed and the styles of play stable is tame. One where everything is, or potentially is, changing, is complex.
Cybernetics is often thought of as a branch of engineering and hence associated with tameable problems. But as a general theory it is much more expressive. Fixed programmes or tightly bounded learning give rise to tameable behaviours. Complexity arises when the learning is totally unbounded, and perhaps when the learning is unbounded at the level that we have visibility of.
Wicked problems and messes
Social planners and operational researchers have noted a distinction between ‘tame’ and ‘wicked’ problems, and this is now widely seen to be important. A similar concept is of a mess. It is clear from their descriptions that wicked problems and messes appear to be complex but that there are significant differences, with less emphasis on emergence and on levels. Whereas for Kant and Whitehead every surprise at one level is an opportunity for learning at a higher level, outside Britain social planners and operational researchers have been more pragmatic, dealing with each new problem anew, with only a fixed ideologies to provide ‘top cover’. They view their problems as appearing to be complex but as needing to be tameable if there is to be progress.
Mathematics and chess are cited as examples of problems that are not wicked. But if one looks at the criteria of Rittel and Webber or of Conklin the only one that they do not meet is that they are not ‘one shot’ operations, because one can improve one’s performance at mathematics and chess by doing them. Thus Whitehead’s notion of a complex problem has this critical distinction: due to the existence of levels, there is no one-size-fits all ideology, set of rules of the game, strategies or problem structuring methods to be learnt and then applied: there is always scope for improvement.
Keynes went on to apply his mathematics to economics, noting that according to the then conventional economics recessions could only result from external factors (such as wars) and that economies were inherently resilient, and that taking emergence into account led to the opposite conclusions. Indeed, after a shock one might need to establish new rules of the game for healthy economies.
Complexity is often considered from a very human viewpoint, as something that strains our capacity or capability to reason. From such a viewpoint, we can think of tameable complexity as something that we might be able to deal with if only we could increase our capacity, perhaps by combining with others. We might then think of true complexity as that which strains our capability, but this would be to make generalizations about capability that might not be justified and which one might seek to overcome.
By identifying complexity with the theories of Kant and Whitehead one can be clear, in a relatively culturally independent fashion, about what is creating the complexity. It is emergence, particularly the emergence of new communities of belonging, of activity or of ideas. To cope with or even exploit complexity one needs to be good at general reasoning, having a clear appreciation of some of the limits of pragmatism and other culturally-dependent short-cuts. One needs to be able to distinguish between an extrapolation (based on the current rules) and a wider appreciation of the opportunities, for ourselves and others.
A problem with Whitehead has been that its logic notion of emergence has been seen as too abstract by ‘pragmatic’ mathematicians. Prigogine is useful in giving more explicit models. Keynes is also useful in showing why the distinction matters. Economies may settle down for long periods while people make fortunes out of playing the game with amazing refinement, but they are ultimately not tameable. The imposition of false authority or consensus can make matters worse, by prolonging a false stability and thereby increasing the over-hang. Nor is it necessarily satisfactory to exploit the current situation to the fullest and then switch to any emergent new reality, re-starting from scratch. Instead one should be mindful of potentially enduring realities and have something credible ready to be the basis for a subsequent building of a new, viable, order.