How can the experience of drinking an espresso in Surrey simulate the same action in Milan? How can we judge whether the simulation is effective?
Since I was sipping espresso in Milan this week, I was reminded of an essay, by Petra Ahrweiler and Nigel Gilbert, which was published in JASSS in 2005. Called “Caffè Nero: the Evaluation of Social Simulation”, it is a meditation on how we know that a simulation “works”. The essay uses the case of the Caffe Nero in Guildford, which claims to serve “the best espresso this side of Milan”. The authors ask how one can judge whether the Guildford cafe is a good simulation of an Italian cafe – and, by extension, how one can judge whether a simulation designed to test a social science situation is a “good” simulation?
The essay concludes that “It is good if it works” – in other words, “The evaluation of the simulation is guided by the expectations, anticipations and experience of the community that uses it — for practical purposes (Caffè Nero), or for intellectual understanding and for building new knowledge (science simulation).” This judgement is partly based on the philosopher John Searle’s work on “social reality”: as the essay puts it, Searle shows “how conventions are “real”: they are not deficient for the support of a relativistic approach because they are constructed.”
When I first blogged this essay in 2005, I said:
“The essay is fundamentally sceptical, quoting Quine on the underdetermination problem: that is, that most theories are underdetermined by the facts. (Which is why two people often hold different theories based on the same set of facts. This is not just that they don’t have enough facts; it is a more profound discontinuity between facts and what we make of them.) So if you ask two people what makes a good or a ‘typical’ Venetian cafe, one will say that it is restful, one that it is noisy and alive. (Partly of course this is a problem of definition: which Italian cafe are you trying to simulate? Make it Florian’s, please…)
So even the observations we use to test the model are based on a theory about what we should be looking for. Testing the simulation designed to validate our underlying theory depends on our second-order theories. Some things cannot be tested at all (eg learning – you may be able to test the results, but not the action itself) and simulations, like real life, have too wide a range of inputs and outputs for realistic testing to be possible.” You can’t just say “It is good if it works” without recognising that “it works” can mean many things.
The Italian cafe simulation is less demanding: it works if I as a “user” am happy with it, and that may be enough. (In other words, if I can sit in Surrey and feel as if I am in Milan: the satisfaction of a scenario that “works” is largely subjective.)
But using simulations to build social science theory is perhaps another issue entirely. In this case. “it works” ought to mean that it delivers a defensible theory. But a news photograph, for example, can be a good source of contrasting underdetermined theories about the event is simulates. (eg see John Berger’s essay on a photograph of Che Guevera dead: ‘in the face of this photograph, we must either dismiss it or complete its meaning for ourselves’.)
The problem is that any simulation is necessarily a shortened version of reality – otherwise we get into the Borges mapproblem, or perhaps what Steven Wolfram calls computational irreducibility. In other words, are all simulations, and all maps, always underdetermined, by definition?
PS: Left hand image: NH-Machiavelli Hotel, Milan. Right hand: Caffe Nero, Richmond.