Rigid Theories and Adaptive Markets

It is no secret that the economics profession is struggling to learn the lessons taught by the recent crisis. Two new books show where that thinking is headed. The End of Theory, by Richard Bookstaber, emphasizes the importance of focusing analysis on the world as it is, rather than on a more formal universe that is easier to model.  Adaptive Markets, by Andrew Lo, explores how much we could benefit from learning to tap the potential of modern finance.

Let’s start with Bookstaber’s argument. Imagine two worlds. In one world, people with the best intentions nevertheless interact with one-another in ways that no-one enjoys; they try to learn from their mistakes; but they accept that they cannot predict the future; and they realize that sometimes there are no short cuts to getting things right. In the other world, people never interact in ways that differ from their underlying motives; they don’t learn from their mistakes because they don’t make ‘mistakes’ (in the conventional sense of the word); they know if their heart of hearts that they can predict the future if only they can make people’s motives more transparent to one-another, which includes revealing any institutional incentives; and they believe that such predictions can be achieved through mathematical shortcuts so that we do not have to rely on lived experience. The first world is the one we live in. The other is the one economists imagine. And that explains, as Bookstaber argues, why economists systematically fail to anticipate when and why things go horribly wrong.


The World as It Is

The solution, Bookstaber insists, is to embrace the world as it is rather than imagining something that is more tractable for mathematical modelling. The goal is not to submit ourselves to the tyranny of the lived experience. Rather it is to embrace a strategy for modelling the world that takes advantage of, rather than ignoring, the basic elements of the human condition. The models would start with real-world agents, it would describe how they navigate the world in terms of their relationships to one-another and the simple decision rules they deploy in the face of changing circumstances, and it would look at many different iterations of their interaction to suggest the universe of possibilities – both intended and unintended. By analysing these scenarios as unfolding narratives, we can isolate the crucial elements that bring our real-world agents into jeopardy. Then we can tweak the institutional environment in ways that will help avoid potential calamity.

There is a lot of promise in this line of argument. Economic theory does rest on unrealistic assumptions and it did fail to anticipate the most recent economic crisis – not to mention crises past. Meanwhile, evolutionary biologists have made great strides in showing how simple, rule-based models that focus on individual agents as they interact with one-another over time in a constrained environment can reveal patterns that mirror what we see in the world around us and yet would be hard to identify using another set of modelling conventions. If we could find a way to use this kind of agent-based modelling to illustrate dynamics that would otherwise remain hidden from view until we experienced them, then this kind of re-engineering of the way we study finance and economics would be worth the effort.


From Assumptions to Modelling

Contrary to the title, however, Bookstaber’s argument does not replace mainstream economics. Sure, economists have their failings, but they also have their successes. Any reasonable replacement should be able to acknowledge and replicate those insights. Bookstaber’s analysis does not and the agent-based modelling used in his illustrations cannot. As for the ability of these models to spot the next crisis, I remain dubious. We do not need a multi-agent adaptive model to spot the problem of portfolio contagion, where crisis spreads from one market to another because portfolio managers need to meet ever increasing liquidity requirements. Bookstaber had a more compelling account of that phenomenon in his last book, A Demon of Our Own Design, which antedated the recent crisis. Much of the emphasis he places on liquidity management can be justified using lived experience as well. That experience was expensive in terms of jobs lost and capital destroyed. If the argument is that we need some sophisticated model to convince policymakers to learn from such a painful past, then I might have to reconsider what world we live in.

Andrew Lo’s Adaptive Markets Hypothesis shows the promise of evolutionary biology and multi-agent adaptive modelling more fully than Richard Bookstaber’s account. Like Bookstaber, Lo begins with a critique of the economic fascination with mathematical modelling and the strong professional consensus surrounding the belief that financial markets are efficient. The difference is that Lo is more willing to admit that the Efficient Markets Hypothesis works. Financial market participants can absorb huge amounts of information in vanishingly short periods of time; their pricing decisions can communicate that information in a streamlined format; and the results can be verified independently through comparisons with other information-harvesting techniques like in-depth surveys (for consumer products) or public opinion polling (for voting intentions and hence election outcomes).


Moving Beyond Existing Accomplishments

Just because the efficient markets hypothesis works, however, does not mean it is ‘right’ – anymore than Newtonian physics is a true depiction of the material world. Human behaviour departs from the assumptions underpinning the financial markets hypothesis in a number of different ways. Many of these deviations, like the influence of framing effects or loss aversion, have been catalogued by behavioural economists. The problem is that observed behaviour does not constitute a predictive theory. We can see bias, but it is hard to see how that bias contributes both to the ‘normal’ functioning of the economy between crises (as anticipated by the efficient markets hypothesis) or the sudden onset of crisis (when predictions based on the efficient markets hypothesis break down). As Lo insists: ‘it takes a theory to beat a theory’ (p. 75). That new theory should be able to capture all that its predecessors have to offer and then add more that they cannot reveal.

Lo’s adaptive markets hypothesis is that replacement. The essence of the theory is the same as that outlined by Bookstaber. Human beings make decisions based on shortcuts; those shortcuts evolve to fit a particular context; and when people use those shortcuts out of context, bad things happen. Lo uses this simple framework to show the conditions under which the efficient markets hypothesis would appear to hold – thus validating the contributions of the mainstream economics literature as a special case. He then shows how the same evolutionary framework can be used to explain the evolution of the various quirks identified by behavioural economists in terms of loss aversion, risk aversion, framing effects, and the like. Critically, Lo shows how that such quirks are not ‘irrational’ but they may be ‘maladaptive’ depending upon the environment (p. 189). Finally, Lo shows how that context varies depending upon whether everyone in the markets experiences the same shocks or whether the experience of individual participants is idiosyncratic.


Idiosyncrasy and Analysis

The notion of idiosyncrasy is important insofar as it makes it possible for Lo to draw on the experience of hedge funds as a kind of natural experiment. According to the efficient markets hypothesis, hedge fund managers should not be able to beat the market. Experience shows, however, that some fund managers are successful for extended periods. Lo uses his adaptive markets hypothesis to explain why market pricing does not follow a random walk. He shows how hedge funds have emerged to exploit systematic deviations using different strategies and techniques. And he shows how the success of those funds invites both imitation and competition until the original deviation no longer exists. The process is not always a smooth one; sometimes whole populations collapse. What matters is that we understand the underlying pathology. Such understanding is not only the key to fixing finance but also to harnessing financial markets to tackle the world’s most significant problems. Lo’s book is essential reading.

These reviews were originally published in the IISS journal Survival.  You can access the review section of the latest issue here.

The End of Theory: Financial Crises, the Failure of Economics, and the Sweep of Human Interaction. By Richard Bookstaber. Princeton: Princeton University Press, 2017.

Adaptive Markets: Financial Evolution at the Speed of Thought. By Andrew W. Lo. Princeton: Princeton University Press, 2017.