Principle 10

Basic "good practice" for extrapolation tasks (science in the dark) differs from that of more straightforward science in the light tasks where the system is thought stable and a large archive of forecast-outcome pairs are available. Nevertheless, violations of good practice remain well defined and, if allowed, the impacts of knowingly bad-practice elements of an analysis must be clearly identified along with their implications for the relevance of those results for decision support. 

WHY?

Good practice, when a system is stable and large datasets can be obtained, consists of robust out-of-sample evaluation of forecasts against real-world outcomes. In extrapolation tasks, this may simply be unachievable, either because few data are available for evaluation, or because the underlying system is changing in such a way that past performance may not be a reliable guide to future success. Good practice here includes acknowledgement of the limitations of statistical methods. Where "bad practice" is employed, for reasons which might include time and resource limitations and the incremental scientific value of conducting an incomplete analysis, it is important to identify what the impact may be on the results and signpost the nature of the incompleteness to any downstream customers.

TELL ME MORE

As an example, the IPCC's headline projections of global mean temperature change in 2100, reported in their Summary for Policymakers, are based on climate models. Instead of reporting the 90% model range as a 90% ("very likely") range for the real-world outcome, they report the 90% model range as a 66% ("likely") range for the real-world outcome. This is good practice: acknowledging the limitations of models and giving a rough steer about the potential implications for decisions and further analysis based on these results. Unfortunately, it is rarely propagated into further analysis such as downscaling or impact modelling.

References and further reading

IPCC, 2013: Summary for Policymakers. In: Climate Change 2013: The Physical Science Basis. Contribution of Working Group I to the Fifth Assessment Report of the Intergovernmental Panel on Climate Change [Stocker, T.F., D. Qin, G.-K. Plattner, M. Tignor, S.K. Allen, J. Boschung, A. Nauels, Y. Xia, V. Bex and P.M. Midgley (eds.)]. Cambridge University Press, Cambridge, United Kingdom and New York, NY, USA.


Leave a comment