Making simulations available and shareable

An article in JASSS makes some very sensible points about the difficulties of sharing simulations.

In “Open Access for Social Simulation”, J. Gary Polhill and Bruce Edmonds pint out that:

“Even the best social simulations are complex objects whose meaning and import are difficult to discern. One may read a description of a simulation and get a vague idea of its properties but, like mathematics, one only thoroughly understands a simulation when one has pulled it apart and played with it. Despite this fact, simulations have become part of the scientific discourse and are starting to enter the more general, public, discourse. In public domains simulations are often communicated without deep hands-on understanding, relying instead upon the skill and integrity of the community who created them. In this sense they are like mathematical models, simultaneously opaque to most and relied upon by many. However, unlike mathematical models, they facilitate attractive and accessible animations of their results—making them seem more transparent than they are and persuasive to a degree quite separate from their validity….

“For these reasons it is important that simulations are openly accessible to others so that they can be checked, compared and improved upon. In this way they can be passed on and developed by a series of independent researchers resulting in objects that are reliable and understood as far as possible. Such simulations could be said to be evolved by the community. The easier it is to download, reimplement, compare, alter and try out the simulations, the more productive this process will be and the better its results.”

They go on to discuss various licensing systems and other means of sharing simulations, and a list of what you need to compare or even implement a simualtion. This includes a good description (or the code) and they add:
“The following are suggested as being potentially helpful in this regard:
* A description in narrative form of the sort of behaviour that is observed when running the simulation;
* Graphs of simulation outcomes;
* Both summary results concerning behaviour that seems to be general to a wide range of parameter settings as well as detailed examples of specific runs and settings;
* A set of numerical summary results that can be used to check the accuracy of any reimplementation (in which case how the statistics are obtained needs careful description);
* A summary description of the expected effects of changing various parameters, for example, which parameters are particularly sensitive;
* Ideally the simulation should be available to be rerun in a manner that makes it easy to obtain more results using parameters chosen by the reader;
* Often it is necessary to be able to change the original code to make fresh simulation experiments in order to track down differences in behaviour, for example by changing the selection method in an evolutionary simulation or to eliminate a source of chance. ”

as well as

“The following are suggested as being potentially helpful in this regard:

* A description in narrative form of the sort of behaviour that is observed when running the simulation;
* Graphs of simulation outcomes;
* Both summary results concerning behaviour that seems to be general to a wide range of parameter settings as well as detailed examples of specific runs and settings;
* A set of numerical summary results that can be used to check the accuracy of any reimplementation (in which case how the statistics are obtained needs careful description);
* A summary description of the expected effects of changing various parameters, for example, which parameters are particularly sensitive;
* Ideally the simulation should be available to be rerun in a manner that makes it easy to obtain more results using parameters chosen by the reader;
* Often it is necessary to be able to change the original code to make fresh simulation experiments in order to track down differences in behaviour, for example by changing the selection method in an evolutionary simulation or to eliminate a source of chance. ”

I quote this at length bcause it emphasises – and puts more elegantly – the point I was trying to make about US military simulations in my previous posting.

Its no good just building a lot of black boxes.

Leave a Reply

Your email address will not be published. Required fields are marked *