On the use of models for prediction

A perceptive and well-argued essay in JASS sets a sceptical threshhold for the use of models as predictions.

The latest JASSS has an essay, Alternative Approaches to the Empirical Validation of Agent-Based Models, by Scott Moss, which contrasts two different schools of modelling. One is based on prior theory – in other words, model the laws already derived by someone else – and the other is “the French school of companion modelling associated with Bousquet, Barreteau, Le Page and others, [..which..> engages stakeholders in the modelling and validation process.”

Sounds good when you put it that way, though I suppose a lot depends on which stakeholders you engage and whether you get a representative sample.

My main interest in this paper is its level of scepticism about modelling results. First theres unpredictable volatility:
“Between volatile episodes, individuals will mainly be engaging in routine behaviour so that econometric analysis might (this has not been proved) yield reasonable forecasts. Environmental scientists have a notion of an x-year event so that a 50-year event would be one that occurs on average every 50 years. But this does not mean that there is a reliable two per cent probability of a 50-year event occuring this year. We might not see such an event for several centuries and then observe a rash of them. The two floods of the Rhine in the early 1990s and none since is an example. Time series of water levels in the Rhine are a good example of distributions of relative changes with heavy tails – just as we observe not only in financial markets but also in supermarket sales of tea, biscuits, shaving preparations, alcoholic beverages and probably any brand of any fast moving consumer good.”

The problem is forecasting the step change.
“The position we face is that models with socially embedded, cognitively plausible agents cannot be used reliably to forecast the consequences of corresponding social processes. The reason is that such models produce unpredictable episodes of volatility in macro level time series and, where we have appropriately fine grain evidence, a similarly unpredictable episodic volatility is found in actual time series. Such social simulation models are therefore better candidates as representations of the real social data generating process than are models that do not produce such episodic volatility. Evidently, such models cannot be used to forecast turning points in trade cycles or financial market prices or other characteristics of social volatility. Whilst it is certainly possible to condition models on data covering such episodes, even models designed econometrically to capture episodes of volatility (based on Engle, 1982; Bollerslev, 1986) have yet to provide a correct forecast to such an episode. Moreover, it has long been known that models that perform well on data sets available at the time of their publication typically perform less well or badly when applied to post-publication data (Mayer, 1975)…..However, it is not possible to know when the next volatile episode will become manifest.”

The problem is that turning points in trade cycles (for example) are exactly what we do want to forecast! Or, in a sense, this is the Millenium Challenge 2002 problem. How do you cope with a maverick result/ agent?

Moss goes on: “However, forecasting over periods long enough to include volatile episodes cannot be reliable and, as far as I know, has never been observed. This conclusion is at odds with the conventional economic view (adopted by many social simulation modellers) that a model that is well validated against existing data can be expected to provide good forecasts in the sense that they would be equally well validated against future data. There is no evidence to justify that presumption and increasing simulation evidence to reject it.”

His conclusion then is that models developed by the companion modelling proces are more useful because: “The formal purpose of validation therefore, can only be for purposes of calibration and not for forecasting. The purpose of the models themselves is to introduce precision into policy and strategy discussions. The validation exercise integrates the models into the discussions of longer term processes helping the engaged stakeholders to disambiguate the terminology they use and to clarify their specification of the social processes generating the outcomes — the data — they anticipate. The models are no more likely to be in any sense true than are narrative scenarios and they lack the richness of the narratives. By integrating the modelling process into the development of narrative scenarios, policy and strategy analysts obtain the benefits of formal precision and the benefits of the rich expressiveness of storylines and scenarios.”

Hmm. If you unpack this, to use Mosss favourite expression, Im not sure that it really means much more than hey, lets talk to some guys before we decide.

But this is a well-thought-out and interesting essay, partly because (yet again) its a reminder that modelling and simulation isnt the same thing as prediction – despite the naive assumption by most users that it is. General Schwarzkopf is quoted as saying: “the movements of Iraq’s real-world ground and air forces [during Desert Storm> eerily paralleled the imaginary scenario of the {Internal Look> game…”. Hes perhaps typical of a generation of modellers and model users who expect that they will get usable predictions from their models, and who are prepared to stake a lot on them.

My own view is that theres a sort of arms race: as our models get better, the areas of uncertainty slowly diminsih. However. Moss is right to point out that the uncertainty level is still enormous, particularly in social and soft sciences: it will take a long time before we understand what causes everything.

Leave a Reply

Your email address will not be published. Required fields are marked *