Uncertainty Guidances in Science and Public Policy

[1] Bamber, Jonathan L., and W. P. Aspinall. An expert judgement assessment of future sea level rise from the ice sheets. Nature Climate Change 3.4 (2013): 424-427.

[2] Barkemeyer, R., Dessai, S., Monge-Sanz, B., Renzi, B. G., and Napolitano, G. (2016). Linguistic analysis of IPCC summaries for policymakers and associated coverage. Nature Climate Change, 6(3), 311-316.

[3] Beyth-Marom R. (1982). How probable is probable? A numerical translation of verbal probability expressions. Journal of Forecasting, 1, 257-269.

[4] Bradley S. (2011). Scientific uncertainty: a user’s guide. Centre for Climate Change Economics and Policy Working Paper No. 65 / Munich Re Programme Technical Paper No. 9.

[5] Budescu, D. V., and Wallsten, T.S. (1985). Consistency in interpretation of probabilistic phrases. Organizational Behavior and Human Decision Processes, 36, 391-405.

[6] Budescu, D. V., Broomell S., and Por, H. (2009). Improving communication of uncertainty in the reports of the intergovernmental panel on climate change. Psychol. Sci. 20:299-308

[7] CCSP, 2009: Best Practice Approaches for Characterizing, Communicating, and Incorporating Scientific Uncertainty in Decision-making. [M. Granger Morgan (Lead Author), Hadi Dowlatabadi, Max Henrion, David Keith, Robert Lempert, Sandra McBride, Mitchell Small, and Thomas Wilbanks (Contributing Authors)]. A Report by the Climate Change Science Program and the Subcommittee on Global Change Research. National Oceanic and Atmospheric Administration, Washington, DC, 96 pp.

[8] Dessai, S., and Hulme, M. (2004). Does climate adaptation policy need probabilities?. Climate policy, 4(2), 107-128.

[9] Dessai, S., Hulme, M., Lempert, R., and Pielke Jr, R. (2009). Climate prediction: a limit to adaptation. Adapting to climate change: thresholds, values, governance, 64-78.

[10] Dhami, M.K. (2017). Towards an Evidence-Based Approach to Communicating Uncertainty in Intelligence Analysis. Intelligence and National Security (in press).

[11] Dhami, M. K., Lundrigan, S., and Mueller-Johnson, K. (2015). Instructions on reasonable doubt: Defining the standard of proof and the juror's task. Psychology, Public Policy and Law, 21, 169-178.

[12] Dhami M. K. and Wallsten T. S. (2005) Interpersonal comparison of subjective probabilities:Toward translating linguistic probabilities. Memory and Cognition, 33 (6), 1057-1068.

[13] Fischhoff, B. (1995). Risk perception and communication unplugged: twenty years of process. Risk analysis, 15(2), 137-145.

[14] Fischhoff, B., and Davis, A. L. (2014). Communicating scientific uncertainty. Proceedings of the National Academy of Sciences, 111(4), 13664-13671.

[15] Frigg, R., Bradley, S., Du, H., and Smith, L. A. (2014). Laplace's Demon and the Adventures of his Apprentices. Philosophy of Science, 81(1), 31-59.

[16] Frigg, R., Smith, L. A., and Stainforth, D. A. (2015). An assessment of the foundational assumptions in high-resolution climate projections: the case of UKCP09. Synthese, 192(12), 3979-4008.

[17] Funtowicz, S. O., and Ravetz, J. R. (1990). Uncertainty and quality in science for policy. Springer Science & Business Media.

[18] Communicating uncertainty and change: Guidance for official statistics producers. Government Statistical Service, 16pp.

[19] Houghton, J. T., Jenkins G. J., and Ephraums J. J. (1990). Climate Change: the IPCC Scientific Assessment. Report prepared for IPCC by Working Group 1. Cambridge University Press.

[20] Houghton, J. T. (1996). Climate change 1995: The science of climate change: contribution of working group I to the second assessment report of the Intergovernmental Panel on Climate Change. Cambridge University Press.

[21] Ho, E. H., Budescu, D. V., Dhami, M. K., and Mandel, D. R. (2015). Improving the communication of uncertainty in climate science and intelligence analysis. Behavioral Science & Policy, 1(2), 43-55.

[22] Hogarth, R. M. (1975). Cognitive processes and the assessment of subjective probability distributions. Journal of the American statistical Association, 70(350), 271-289.

[23] Hohle, S. M. and Teigen, K. H. (2015). Forecasting forecasts: The trend effect. Judgment and decision making. 10(5):416- 428.

[24] Inter-Academy Council (2010). Climate change assessments: Review of the processes and procedures of the IPCC. Committee to Review the Intergovernmental Panel on Climate Change.

[25] Guidance Notes for Lead Authors of the IPCC Fourth Assessment Report on Addressing Uncertainties, IPCC, 4pp.

[26] Janssen, P. H., Petersen, A. C., van der Sluijs, J. P., Risbey, J. S., and Ravetz, J. R. (2005). A guidance for assessing and communicating uncertainties. Water science and technology, 52(6), 125-131.

[27] Juanchich, M., Teigen, K. H. and Gourdon, A. (2013). Top scores are possible, bottom scores are certain (and middle scores are not worth mentioning): A pragmatic view of verbal probabilities. Judgment and decision making. 8(3):345-364.

[28] Kandlikar, M., Risbey, J., and Dessai, S. (2005). Representing and communicating deep uncertainty in climate-change assessments. Comptes Rendus Geoscience, 337(4), 443-455.

[29] Knutti, R., G. Abramowitz, M. Collins, V. Eyring, P. J. Gleckler, B. Hewitson, and L. Mearns, 2010: Good Practice Guidance Paper on Assessing and Combining Multi Model Climate Projections. In: Meeting Report of the Intergovernmental Panel on Climate Change Expert Meeting on Assessing and Combining Multi Model Climate Projections [Stocker, T. F., D. Qin, G.-K. Plattner, M. Tignor, and P. M. Midgley (eds.)]. IPCC Working Group I Technical Support Unit, University of Bern, Bern, Switzerland.

[30] Lempert, R. J., and Schlesinger, M. E. (2000). Robust strategies for abating climate change. Climatic Change, 45(3-4), 387-401.

[31] Løhre, E. and Teigen, K. H. (2016). There is a 60% probability, but I am 70% certain: Communicative consequences of external and internal expressions of uncertainty. Thinking and Reasoning. 22(4):369-396.

[32] Lynn, J., M. Araya, O. Christophersen, I. El Gizouli, S.J. Hassol, E. M. Konstantinidis, K. J. Mach, L. A. Meyer, K. Tanabe, M. Tignor, R. Tshikalanke, J.-P. van Ypersele (eds.) (2016). Meeting Report of the Intergovernmental Panel on Climate Change Expert Meeting on Communication. World Meteorological Organization, Geneva, Switzerland.

[33] Manning, M.R. (2006). The Treatment of Uncertainties in the Fourth IPCC Assessment Report. Adv. Clim. Change Res. 2 (Suppl. 1): 13-21.

[34] Manning, M. R., M. Petit, D. Easterling, J. Murphy, A. Patwardhan, H-H. Rogner, R. Swart, and G. Yohe (eds.) (2004) IPCC Workshop on Describing Scientific Uncertainties in Climate Change to Support Analysis of Risk and of Options: Workshop Report. Intergovernmental Panel on Climate Change (IPCC), Geneva, Switzerland.

[35] Mastrandrea, M. D., Field, C. B., Stocker, T. F., Edenhofer, O., Ebi, K. L., Frame, D. J., and Plattner, G. K. (2010). Guidance note for lead authors of the IPCC fifth assessment report on consistent treatment of uncertainties.

[36] Mastrandrea, M. D., Mach, K. J., Plattner, G. K., Edenhofer, O., Stocker, T. F., Field, C. B., and Matschoss, P. R. (2011). The IPCC AR5 guidance note on consistent treatment of uncertainties: a common approach across the working groups. Climatic Change, 108(4), 675-691.

[37] Mastrandrea, M. D., and Mach, K. J. (2011). Treatment of uncertainties in IPCC Assessment Reports: past approaches and considerations for the Fifth Assessment Report. Climatic Change, 108(4), 659-673.

[38] Morgan, M. G. and Keith, D. W. (1995). Subjective judgements by climate experts. Environ. Sci. Technol. 29, 468-476.

[39] Morgan, M. G. (2014). Use (and abuse) of expert elicitation in support of decision making for public policy. Proceedings of the National Academy of Sciences, 111(20), 7176-7184.

[40] Morgan, M. G. and Henrion M. (2010). Uncertainty: A Guide to Dealing with Uncertainty in Quantitative Risk and Policy Analysis. Cambridge University Press, ISBN 978-0521427449.

[41] Melillo, J. M., Richmond T.C., and Yohe, G.W. (Eds.) (2014). Climate Change Impacts in the United States: The Third National Climate Assessment. U.S. Global Change Research Program, 841 pp. doi:10.7930/J0Z31WJ2.

[42] Moss, R. H. and Schneider, S. H. (2000). Uncertainties in the IPCC TAR: Recommendations to lead authors for more consistent assessment and reporting. In: Guidance Papers on the Cross Cutting Issues of the Third Assessment Report of the IPCC [eds. R. Pachauri, T. Taniguchi and K. Tanaka], World Meteorological Organization, Geneva, pp. 33-51.

[43] Moss, R.H. (2011). Reducing doubt about uncertainty: Guidance for IPCC’s third assessment. Climatic Change 108:641.

[44] Moss, R. H. (2016). Assessing decision support systems and levels of confidence to narrow the climate information “usability gap”. Climatic Change 135:143-155.

[45] National Research Council Governing Board Committee on the Assessment of Risk. (1981). The handling of risk assessments in NRC Reports. Washington, DC.

[46] Nicholls, N. (1999). Cognitive illusions, heuristics, and climate prediction. Bulletin of the American Meteorological Society, 80(7), 1385-1397.

[47] Nordhaus, W. D. (1994). Expert opinion on climatic change. American Scientist, 82(1), 45-51.

[48] Nuclear Decommissioning Authority (2017). Geological disposal: Methods for management and quantification of uncertainty. NDA Report no. NDA/RWM/153, ISBN 978-1-84029-579-5.

[49] O’Brien, B. J. (1989). Words or numbers? The evaluation of probability expressions in general practice. J R Coll Gen Pract. 39(320): 98-100.

[50] O’Neill, B. C., Oppenheimer, M., Warren, R., Hallegatte, S., Kopp, R. E., Portner, H. O., and Mach, K. J. (2017). IPCC reasons for concern regarding climate change risks. Nature Climate Change, 7(1), 28-37.

[51] Oppenheimer, M., O'Neill, B. C., Webster, M., and Agrawala, S. (2007). The limits of consensus. Science Magazine's State of the Planet 2008-2009: with a Special Section on Energy and Sustainability, 317, 1505-06.

[52] Oppenheimer, M., Little, C. M., and Cooke, R. M. (2016). Expert judgement and uncertainty quantification for climate change. Nature Climate Change, 6(5), 445-451.

[53] Parker, W. (2010). Whose Probabilities? Predicting Climate Change with Ensembles of Models. Philosophy of Science 77(5): 985-997.

[54] Parker, W. (2011). When Climate Models Agree: The Significance of Robust Model Predictions. Philosophy of Science 78(4): 579-600.

[55] Parker, W. (2014). Values and uncertainties in climate prediction, revisited. Studies in History and Philosophy of Science Part A 46: 24-30.

[56] Patt, A. G., and Schrag, D. P. (2003). Using specific language to describe risk and probability. Climatic change, 61(1-2), 17-30.

[57] Patt, A, and Dessai, S. Communicating uncertainty: lessons learned and suggestions for climate change assessment. Comptes Rendus Geoscience 337.4 (2005): 425-441.

[58] Petersen, A. C. (2011). Climate simulation, uncertainty, and policy advice - the case of the IPCC. In Climate Change and Policy (pp. 91-111). Springer Berlin Heidelberg.

[59] Petersen, A. C. (2012). Simulating Nature: A Philosophical Study of Computer-Model Uncertainties and Their Role in Climate Science and Policy Advice. (2nd ed.). CRC Press, ISBN 978-1466500624.

[60] Pidgeon, N., and Fischhoff, B. (2011). The role of social and decision sciences in communicating uncertain climate risks. Nature Climate Change, 1(1), 35-41.

[61] Pittock, A. B., Jones, R. N., and Mitchell, C. D. (2001). Probabilities will help us plan for climate change. Nature, 413(6853), 249.

[62] Risbey, J. S., and Kandlikar, M. (2007). Expressions of likelihood and confidence in the IPCC uncertainty assessment process. Climatic change, 85(1-2), 19-31.

[63] Seneviratne, S. (2012). Changes in climate extremes and their impacts on the natural physical environment. IPCC presentation, Geneva, March 2012.

[64] Smith, L. A. (2002). What might we learn from climate forecasts? Proceedings of the National Academy of Sciences, 99(1), 2487-2492.

[65] Smith, L. A. and Petersen, A. C. (2013) Variations on reliability: Connecting climate predictions to climate policy. In: Boumans, M. and Hon, G. and Petersen, A. C., (eds.) Error and Uncertainty in Scientific Practice. (pp. 137-156). Pickering & Chatto: London, UK.

[66] Swart, R., Bernstein, L., Ha-Duong, M., and Petersen, A. (2009). Agreeing to disagree: uncertainty management in assessing climate change, impacts and responses by the IPCC. Climatic Change, 92(1-2), 1-29.

[67] Teigen, K. H. (2014). When very likely is not so likely. Nature Climate Change, 4:421-422.

[68] Teigen, K. H., Juanchich, M. and Filkukova, P. (2014). Verbal probabilities: An alternative approach. Quarterly Journal of Experimental Psychology. 67(1):124-146.

[69] Teigen, K. H., Juanchich, M. and Riege, A. H. (2013). Improbable outcomes: Infrequent or extraordinary? Cognition. 127(1):119-139.

[70] Thompson E. L., Frigg R. P. and Helgeson C. (2016). Expert Judgement for Climate Change Adaptation. Philosophy of Science, 83:1110-1121.

[71] Van Der Sluijs, J. P., Craye, M., Funtowicz, S., Kloprogge, P., Ravetz, J., and Risbey, J. (2005). Combining quantitative and qualitative measures of uncertainty in model-based environmental assessment: the NUSAP system. Risk analysis, 25(2), 481-492.

[72] Wallsten, T. S., Budescu, D. V., Rapoport, R. Z., and Forsyth, B. (1986). Measuring the vague meanings of probability terms. Journal of Experimental Psychology: General, 115, 348?365.

[73] Wallsten, T. S., Fillenbaum, S., and Cox, J. A. (1986). Base rate effects on the interpretations of probability and frequency expressions. Journal of Memory and Language, 25, 571?587.

[74] Wardekker, J. A., van der Sluijs, J. P., Janssen, P. H., Kloprogge, P., and Petersen, A. C. (2008). Uncertainty communication in environmental assessments: views from the Dutch science-policy interface. Environmental Science & Policy, 11(7), 627-641.

[75] Weber, E. U., and Hilton, D. J. (1990). Contextual effects in the interpretation of probability words: Perceived base rate and severity of effects. Journal of Experimental Psychology: Human Perception and Performance, 16, 781-789.

[76] Winsberg, E., Huebner, B. and Kukla, R. (2014). Accountability and values in radically collaborative research. Studies in History and Philosophy of Science Part A, 46:16-23.

[77] Zickfeld, K., Levermann, A., Morgan, M. G., Kuhlbrodt, T., Rahmstorf, S., and Keith, D. W. (2007). Expert judgements on the response of the Atlantic meridional overturning circulation to climate change. Climatic Change, 82(3-4), 235-265.