'Improving the Quantitative Interpretation of Simulation Models'
Workshop
Date: 13-18 March 2016
Venue: Banff International Research Station for Mathematical Innovation and Discover
Organised by: Leonard Smith and Robert Rosner (University of Chicago)
Objectives
Despite the fact that mathematical simulations are now common place in science, decision making, forward planning, policy, regulation, and litigation, the fundamental connections between a mathematical simulation and the empirical quantities of the world it targets remain vague and wishy-washy. The focus of this meeting is not so much how science and policy draw insights from the general behaviour of complicated simulation modelling, but rather on the mathematical foundation and empirical justification for quantitative interpretation of simulations for use in the real world, the design of computational experiments and model inter-comparisons, the quantitative comparison of simulation and observations under controlled lab conditions, and the end-to-end communication of information between all those involved. The final attendance list will determine the precise selection of topics in the small (~20) intense program, however we expect to include (a) clarifying the relationship between temperature measured by a thermometer or inferred from a satellite, and the variable of the same name in the numerical solution of a set of partial differential equations (on a finite grid), (b) the extent to which ensembles of simulations (whether over sets of initial conditions, parameter values, mathematical model structures, …) can be deployed/interpreted rationally as a probability distribution for the future quantitative measurements, (c) the improvement (specialization) of experimental design for “grand” ensemble experiments and model inter-comparison projects, and how that design might best differ depending on the target of the exercise, and (d) the quantitative contrast of simulation with observations of laboratory experiments under carefully controlled conditions.
The meeting would have very few lectures, the traditional talks we do have would focus on common jargon, goals, and targets. It will include at least one panel discussion per day, homework assignments (the statistics of which will be summarised in the following day’s discussions ), and several structured discussions each of which will use Turningpoint “clickers” to gather and display anonymous response in real time. The lead organiser has used this method in the past (but with students and young career researchers, never with senior researchers). Where there is general agreement topics will be wound up, while on topics where there is a strong diversity of views the origins of those differences will be followed up and identified. A major aim of the meeting would be to achieve precision in understanding exactly where individuals disagree on the quantitative relations of simulations and the world, and to determine if those disagreements can be resolved, at least in special cases. Each panel will target a few specific questions, one panel on setting and exploring the values used for model parameters, another for the interpretations of model diversity as probabilistic forecasts, another on the design of large model inter-comparison experiments which a present is largely hodgepodge. Panels will each consist of individuals with similar interests from different fields (crop forecaster, financial risk regulator, climate modeller, nonlinear dynamical systems mathematician, Verification and Validation expert). Most of the panellists do not know each other at present, although almost all of them know one or more of the organisers. The meeting is timely in several respects, one being that the climate community will be preparing “CMIP6” , a large model inter-comparison project in support of the next IPCC report. The location and environment of BIRS is important to allow relaxed, constructive, civil discussion of complex questions which sometimes result in heated debate. By showing similar challenges exist across a number of fields we plan to achieve significant progress in the underlying mathematical, computational and somewhat philosophical issues, insight into which is key to the more effective use of computational models in society and in science.
Relevance: Simulation by complicated computer modelling is common place in a growing variety of fields and applications. Better, more accessible guidance on uncertainty quantification, validation and verification, and the incorporation of considerations of model inadequacy in large experimental design and model inter-comparison projects is relevant to a host of mathematical modelling and applied research programs.
Importance: The lack of clarity and guidance regarding how to connect (quantitatively) the world with our models of the world is one of the most important issues in modern simulation. This meeting would allow the sharing of lessons learned and of concerns across disciplines. Model inadequacy is often side-lined, mischaracterised in a naïve manner as merely a question of statistical post-processing. By focusing on how to best use what exists, this meeting will provide an important foundation, by summarising approaches and methods used in different fields, each being critiqued mathematically, important cross-pollination can occur.
Timeliness: Arguably, this workshop is two decades overdue. Naïve interpretations of simulation experiments (both individual and ensembles) abound. Some of those expressing interest in attending clearly stated open challenges back in the mid-1990s; many of those challenges remain open and would be targeted at BIRS. Two of the climate scientists approached noted that the year 2016 falls at an opportune time in terms of the preparation for the next IPCC report. Perhaps most importantly, issues of model inadequacy are widely acknowledged and even more widely ignored, largely due to the lack of a positive framework for their discussion, this meeting will not resolve the challenge of model inadequacy, but it will raise the level of debate and the profile of a critical challenge.
Final Report (pdf)