Errors in estimates of the aggregate economic impacts of climate change – Part II

In a previous commentary, I outlined my interactions with Professor Richard Tol about some errors in his work. I provide here further details of those errors and my interactions with Professor Tol, with the journals in which the papers containing the errors were published, and with Working Group II of the Intergovernmental Panel on Climate Change.

On 17 October 2013, a new print edition of ‘The Spectator’ was published, featuring an article by Viscount Ridley, advertised on the front cover, under the headline ‘Why climate change is good for the world’.

Viscount Ridley’s article included several references to an academic paper by Professor Tol which had been published in the ‘Journal of Economic Perspectives’ in 2009. When I examined the paper, I discovered a number of errors and inconsistencies.

Errors in Tol (2009)

The Economic Effects of Climate Change’, by Richard S.J. Tol, was published in the ‘Journal of Economic Perspectives’ in 2009 (Volume 23, Number 2, Spring 2009, pages 29–51).

Table 1 (page 31) and Figure 1 (page 35) contain a number of errors.

i) Table 1 of Tol (2009) refers to a study by “Nordhaus (1994b)”, which, according to the references listed in Tol (2009), is a paper by William D. Nordhaus on ‘Expert Opinion on Climatic Change’, published in the journal ‘American Scientist’ in 1994 (Volume 82, Number 1, January-February 1994, pages 45-51).

Table 1 of Tol (2009) lists Nordhaus (1994b) as concluding that the welfare impact of warming of 3°C would be -4.8 per cent of GDP, with an “uncertainty” of -30.0 to 0.0 per cent.

However, Scenario A in Figure 2 on page 48 of Nordhaus (1994b) shows that it concluded a 3°C rise in global average temperature would result in a loss ranging from 0 to 21 per cent of gross world product with a mean loss of 3.6 per cent.

In fact, the figures attributed by Tol (2009) to Nordhaus (1994b) correspond to Scenario A in Figure 3 on page 48 which indicates a range of 0 to 30 per cent, with a mean of 4.8 per cent, for estimates of the likelihood of a high-consequence event (a loss of at least 25 per cent in global output) from global warming of 3°C by 2090.

It appears, therefore, that Professor Tol did not read properly Nordhaus (1994b) and mixed up Figure 2 and Figure 3.

This error also occurs in Figure 1 on page 35 of Tol (2009) where the value of welfare impact attributed to Nordhaus (1994b) is -4.8 per cent.

ii) Table 1 of Tol (2009) refers to a study by “Plambeck and Hope (1996)”, which, according to the references listed in Tol (2009), is a paper by Erika L. Plambeck and Chris W. Hope on ‘PAGE95 – An Updated Valuation of the Impacts of Global Warming’, published in the journal ‘Energy Policy’ in 1996 (Volume 24, Issue 9, September 1996, pages 783-793).

Table 1 of Tol (2009) lists Plambeck and Hope (1996) as concluding that the welfare impact of warming of 2.5°C would be +2.5 per cent of GDP, with an “uncertainty” of -0.5 to -11.4 per cent.

However, Plambeck and Hope (1996) do not cite any value of global losses for warming. The paper states that the tolerable temperature rise before any impacts occur would be 2°C, and Figure 5 on page 790 shows a cubic function and linear function both estimating GDP losses of about 1 per cent of GDP for a warming of about 4.5°C.

In his more recent papers, Professor Tol lists Plambeck and Hope (1996) as concluding that the welfare impact of warming of 2.5°C would be -2.5 per cent of GDP, with a range of uncertainty of -0.5 to -11.4 per cent, but it is not clear how these figures were calculated. However, as they are more recent, it would appear that the figure of +2.5 per cent cited in Table 1 of Tol (2009) is an error. Figure 1 on page 35 of Tol (2009) includes a value attributed to Plambeck and Hope (2006) for welfare impact of -2.5 per cent.

iii) Table 1 of Tol (2009) refers to a study by “Hope (2006)”, which, according to the references listed in Tol (2009), is a paper by Chris Hope on ‘The marginal impact of CO2 from PAGE2002: An integrated assessment model incorporating the IPCC’s five reasons for concern’ published in the ‘Integrated Assessment Journal’ in 2006 (Volume 6, Number 1, pages 19-56).

Table 1 of Tol (2009) lists Hope (2006) as concluding that the welfare impact of warming of 2.5°C would be +0.9 per cent of GDP, with an “uncertainty” of -0.2 to +2.7 per cent.

However, Hope (2006) does not cite any value of global losses for warming. The paper estimates the marginal impacts of a 10 per cent reduction in carbon dioxide emissions. These are calculated from the PAGE2002 model which incorporates regional impact factors listed in Table 5 on page 24 as percentage GDP loss due to global warming of “2.5°C above the tolerable level in each impact sector in the EU, with regional multipliers for other regions”. Apart from the EU, regional weight factors are provided for seven other regions, with mean values ranging from -0.35 for Eastern Europe and the Former Soviet Union (the only regional impact factor implying a positive change in GDP) to 2.5 for India.

iv) Table 1 of Tol (2009) refers to a study by “Nordhaus (2006)”, which, according to the references listed in Tol (2009), is a paper by William D. Nordhaus on ‘Geography and macroeconomics: new data and new findings’, published in the journal ‘Proceedings of the National Academy of Sciences’ in 2006 (Volume 103, Number 10, pages 3510-3517).

Table 1 of Tol (2009) lists Nordhaus (2006) as concluding that the welfare impact of warming of 2.5°C would be -0.9 per cent of GDP, with an “uncertainty” of 0.1 per cent.

However, Nordhaus (2006) presents an estimate of impacts from two scenarios, one of which considers warming only and one which includes mid-continental drying as well. On page 3516 of the paper, Nordhaus (2006) states that the scenarios are drawn from the IPCC TAR and “have been rescaled to correspond to a 3°C global average equilibrium increase”, not a warming of 2.5°C as Table 1 of Tol (2009) states.

Figure 1 on page 35 of Tol (2009) includes a value attributed to Nordhaus (2006) for welfare impact of -0.9 per cent for a warming of 2.5°C, which also appears to be an error.

In summary, 13 studies, yielding 14 data points, are listed in Table 1 on page 31 and plotted in Figure 1 on page 35 of Tol (2009). At least four of the data points are inaccurately represented in Table 1, and three of the studies are inaccurately represented in Figure 1. Of the remaining 10 data points, I have been able to verify that five values (for Nordhaus (1994a), Fankhauser (1995), Tol (1995), Nordhaus and Boyer (2000), and Tol (2002)) listed in Table 1 and plotted in Figure 1 of Tol (2009) are correct. However, the five remaining data points (for Nordhaus and Yang (1996), Mendelsohn et al. (2000), Maddison (2003), and Rehdanz and Maddison (2005)) were derived by Tol using his own calculations based on the other authors’ work, so I have been unable to verify their accuracy.

Given the three known errors in Figure 1, the least squares fit to the 14 data points, and the 95 per cent confidence intervals, to the 14 data points should be re-calculated. In addition, the commentary on pages 33-37 of Tol (2009) should also be revised.

With the correction of the errors in Table 1 and Figure 1, only two of the 14 data points suggest net positive welfare impacts from warming.

One of these studies was listed by Tol (2009) as “Mendelsohn, Schlesinger and Williams (2006)”, which, according to the references listed in Tol (2009), is a paper by Robert Mendelsohn, Michael Schlesinger and Larry Williams on ‘Comparing impacts across climate models’, published in the journal ‘Integrated Assessment in 2000 (Volume 1, Issue 1, pages 37-48). Mendelsohn et al. (2000) provided two data points for Tol (2009), listed as welfare impacts of +0.1 per cent and 0.0 per cent of GDP for a warming of 2.5°C. However, these data points are based on aggregations by Tol (2009) of the figures published by Mendelsohn et al. (2000), and cannot be verified. It should be noted that Mendelsohn et al. (2000) warned that their estimates of the impacts was incomplete, and the final paragraph of the paper states:

“Finally, we have measured only market effects from predicted climate changes. Preliminary research indicates that climate change is also likely to impact the quality of life. Effects on ecosystems, health, and aesthetics have not been taken into account in this analysis. Impacts from changes in extreme events or catastrophes should also be measured. As research in these areas develops, the model can be revised to include these more complete measures of impacts.”

The other study suggesting a net positive welfare impact is another paper by Professor Tol on ‘Estimates of the Damage Costs of Climate Change. Part 1: Benchmark Estimates’, published in the journal ‘Environmental and Resource Economics’ in 2002 (Volume 21, Issue 1, pages 47-73).

Tol (2002), which found an impact of +2.3 per cent of GDP from a warming of 1°C, also acknowledges that its estimate is incomplete, and states:

“The list of omitted impacts is long. It includes amenity, recreation, tourism, extreme weather, fisheries, construction, transport, energy supply, morbidity and so on. The reason for omitting is that no comprehensive, quantified impact studies have been reported.”

Tol (2009) acknowledges some of the shortcomings of the data used in Table 1 and Figure 1, stating that “the level of uncertainty here is large, and probably understated – especially in terms of failing to capture downside risks”. However, the three mistakes in Figure 1, particularly the mis-plotting of Hope (2006) as a welfare impact of +0.9 instead of -0.9, appears to have prevented Professor Tol from recognising that his 2002 paper is an outlier because it is the only one of the 14 data points that indicated significant net benefits from warming, and excluded many of the potential impacts. Hence, the section on ‘Findings and implications’ on pages 33-37 of Tol (2009) require revision, particularly the assertion on page 34 that “some estimates, by Hope (2006), Mendelsohn, Morrison, Schlesinger, and Andronov (2000), Mendelsohn, Schlesinger and Williams (2000) and myself (Tol, 2002b) point to initial benefits of a modest increase in temperature, followed by losses as temperatures increase further”.

I also checked a more recent paper by Professor Tol which provided an updated analysis of the aggregate economic impacts of climate change.

Errors in Tol (2012)

‘On the Uncertainty About the Total Economic Impact of Climate Change’, by Richard S.J. Tol, was published in the journal ‘Environmental and Resource Economics’ in 2012 (Volume 53, Issue 1, September 2012, Pages 97-116).

Table 1 on page 99, Figure 1 on page 101, and Figure 3 on page 105 use exactly the same 14 data points that are cited in Tol (2009), and include most of the same errors – the only difference is that Plambeck and Hope (1996) is represented in Table 1 as yielding a value of -2.5 per cent of GDP for the welfare loss due to a warming of 2.5°C.

In addition, much of the accompanying text in Tol (2012) is identical to that of Tol (2009), and repeats the same assertions that require revision in light of the errors. For instance, on page 100, Tol (2012) states that “some estimates (Hope 2006; Mendelsohn et al. 200a,b; Tol 2002b) point to initial benefits of a modest increase in temperature, followed by losses as temperatures increase further”. However, the three errors among the 14 data have more serious implications for Tol (2012) because they were used to carry out an uncertainty analysis which is described on pages 102-108. These analyses need to be repeated using the correct values for the data.

My interactions with Professor Tol

After discovering the errors in Tol (2009) and Tol (2012), I contacted Professor Tol by e-mail on 18 October 2013. I exchanged a number of messages with him, and he reluctantly admitted to three of the four errors in Table 1 and two of the three errors in Figure 1. When I pointed out that Tol (2009) had misrepresented the findings of Hope (2006), he replied “shame on me”. My last e-mail to him asked if he would be publishing a corrigendum to correct the errors. Professor Tol did not reply. On 25 October 2013, I sent another e-mail to Professor Tol asking him to make available to me the details of the calculations he had made in aggregating the work of other authors for Tol (2009) and Tol (2012), so that I could check them for further errors. He did not reply. On 5 November 2013, I sent another e-mail to Professor Tol to ask him what plans he had to correct the errors in Tol (2009) and Tol (2012), and to request that he make his calculations available so that they could be checked for more errors. He did not reply. At this point, I decided not to pursue the issue further as Professor Tol was refusing to respond to my requests.

Then in January 2014, I discovered that a leaked version of the Final Draft of the contribution of Working Group II of the Intergovernmental Panel on Climate Change to the Fifth Assessment Report had been posted on a blog for climate change ‘sceptics’. I was a registered reviewer for the report and had seen the First Order and Second Order Drafts which had been made available to the research community for comment. I had paid particular attention to Chapter 10 of the report because the draft versions cited a paper on which I had been a lead author (Robert E.T. Ward, Celine Herweijer, Nicola Patmore and Robert Muir-Wood:‘The role of insurers in promoting adaptation to the impacts of climate change’, published in ‘The Geneva Papers on Risk and Insurance: Issues and Practice’, Volume 33, January 2008, pages 133-139). Chapter 10 had two Co-ordinating Lead Authors: Professor Douglas Arent and Professor Tol.

I had not previously seen the Final Draft because it had not been made available for comment by the research community and had only been distributed to governments on 28 October 2013, seeking comments by 20 December 2013. The Final Draft version of Chapter 10 was posted on the IPCC website on 31 March 2014.

When I examined the leaked version of Chapter 10 in January 2014, I discovered that a new section 10.9.2 on ‘Aggregate impacts’ had been added on page 34, along with a new Table 10.B.1 on pages 82-83 and a new Figure 10-1 on page 84, all of which appeared to be based on a paper by Professor Tol which was published in 2013. The Table, Figure and text had not been included in any Chapters of the Second Order Draft.

Errors in Tol (2013)

‘Targets for global climate policy: An overview’,by Richard S.J. Tol, was published in the ‘Journal of Economic Dynamics and Control’ (Volume 37, Issue 5, May 2013, pages 911–928).

I checked Tol (2013) and found that it includes among its 17 data points the 14 data points that are used by Tol (2009) and Tol (2012), together with data from two other studies. Of the 14 data points, three (labelled as Nordhaus, 1994a, Hope, 2006, and Nordhaus, 2006) are misrepresented in Table 1 on page 914, in Figure 1 on page 912, and in Figure 8 on page 921. Hence all three of the erroneous data points that are included in Tol (2012) also appear in Tol (2013).

In addition, the text in Tol (2013) needs to be revised to take account of the three errors among the 17 data points that are considered. For instance, page 912 of Tol (2013) paraphrases Tol (2009) and Tol (2012) with the statement that “the initial benefits of a modest increase in temperature are probably positive, followed by losses as temperatures increase further”, even though only one data point, from Tol (2002), suggests any significant positive net impact due to warming.

Errors in the Final Draft of Chapter 10 of the contribution of IPCC Working Group II to the Fifth Assessment Report

The text contained in Section 10.9.2 of the Final Draft of Chapter 10 of the contribution of IPCC Working Group II to the Fifth Assessment Report is similar to that of Tol (2013), even though the source is not cited:

“Estimates agree on the size of the impact (small relative to economic growth) but disagree on sign (Figure 10-1). Climate change may be beneficial for moderate climate change but turn negative for greater warming.”

However, Figure 10-1 and the accompanying Table 10.B.1 show that only one of 20 data points indicates any significant net positive impacts of warming: Tol (2002).

Table 10.B.1 and Figure 10-1 use the 17 data points from Tol (2013), and reproduces one of the errors that appear in Tol (2009), Tol (2012) and Tol (2013), labelled as Nordhaus (1994a). But Hope (2006a) is cited as providing an estimate of the welfare impact, due to a warming of 2.5°C, of -0.9 per cent of GDP, with an “uncertainty” of -0.2 to +2.7. Nordhaus (2006) is cited as providing two estimates of the welfare impact, due to a warming of 3.0°C, of -0.9 and -1.1 per cent of GDP.

There is also an error in one of the other data points listed in Table 10.B.1 and plotted in Figure 10-1, but which was not included in Tol (2013). Table 10.B.1 refers to a study by “Roson and van der Mensbrugghe (2012)”, which, according to the references listed in Chapter 10, is a paper by Roberto Roson and Dominique van der Mensbrugghe on‘Climate change and economic growth: impacts and interactions’, published in the ‘International Journal of Sustainable Economy’ in 2012 (Volume 4, Number 3, pages 270-284).

Table 10.B.1 lists Roson and van der Mensbrugghe (2012) as concluding that the welfare loss of warming of 2.3°C would be equivalent to -1.8 per cent of GDP, and a warming of 4.9°C would create a welfare loss equivalent to -4.6 per cent of GDP. In fact, Roson and van der Mensbrugghe (2012) states on page 282: “According to our preliminary estimates, at the global level, the most serious consequence from climate change will be changes to labour productivity that would induce 84% of the global damage in 2050 (-1.8% of global GDP) and 76% in 2100 (-4.6% of global GDP)”. The authors do not state how much warming they assume to occur by 2050 and 2100. I contacted the authors by email on 28 January 2014 for clarification and they sent me a breakdown showing a warming of 2.3°C in 2050 and 4.8°C in 2100, compared with the 2.3°C and 4.9°C cited in Table 10.B.1 of Chapter 10.

My interactions with the journals

Having discovered the errors in Tol (2009), Tol (2012) and Tol (2013), and finding that Professor Tol would not respond to my e-mail messages seeking assurances that he would correct them and make available his calculations so that their accuracy could be verified, I decided to contact the journals directly to draw attention to the problems. I felt that this was necessary because the errors were being propagated by others who were citing the papers by Professor Tol.

On 28 January 2014, I sent an e-mail message to the ‘Journal of Economic Perspectives’, which published Tol (2009), and I have had numerous exchanges with Ann Norman, the Assistant Editor. The last e-mail message from her on 14 March 2014 stated: “Professor Tol was told to publish and correction and give us data and the editors are deciding whether what they have is an acceptable response to the complaint”.

I also sent an e-mail message on 28 January 2014 to the journal ‘Environmental and Resource Economics’, which published Tol (2012). After some delay, I received a response from Professor Ian Bateman, the Editor-in-Chief. On 4 March, Professor Bateman sent me an e-mail indicating that Professor Tol would not be asked to provide an errata, or to allow access to his calculations, but instead suggesting that I could submit a ‘Comment’ to the journal. I agreed as it seemed to be the only way to correct the errors in Tol (2012). I asked if the ‘Comment’ might also deal with the errors in Professor Tol’s other papers, but the last e-mail message from Professor Bateman on 14 March 2014 stated: “I would prefer you just criticise what’s in our journal rather than others”.

In addition, I sent an e-mail message on 28 January 2014 to the ‘Journal of Economic Dynamics and Control’, which published Tol (2013). After some delay, I received a response from Professor Christopher Otrok, one of the Editors. The last e-mail message from him on 17 March 2014 stated: “From my reading of the results it seems that the only error in the paper is that a minus sign was dropped from Table 1. I will look into having this noted/changed/add an erratum in the online version of the paper to fix this”.

Hence I have yet to receive assurances from any of the journals that all of the errors by Professor Tol will be corrected in the papers that they published. None of the journals have made available the calculations that Professor Tol used to derive the aggregations of other authors’ work.

My interactions with Working Group II of the Intergovernmental Panel on Climate Change

On 31 January 2014, I sent an e-mail to Professor Douglas Arent and Professor Tol, as Coordinating Lead Authors of Chapter 10 of the contribution of Working Group II of the Intergovernmental Panel on Climate Change to the Fifth Assessment Report. The message was copied to Professor Chris Field and Professor Vicente Barros, the two Co-Chairs of Working Group II, and to David Dokken, the Executive Director of the Technical Support Unit for Working Group II. My message drew attention to the errors in Table 10.B.1, Figure 10-1 and the text in Section 10.9.2. I have received responses from Professor Arent. His email message on 8 March 2014 stated: “The data has been double and triple checked, and corrected if in error, and the chapter revised”. I have not received any indication that all of the errors will be corrected in the final version of Chapter 10.

On 4 April, I sent another e-mail message to the same group of individuals to draw their attention to an article by Professor Tol which was published by ‘The Conversation’ on 2 April 2014. In the article, Professor Tol describes ‘The Economics of Climate Change: The Stern Review’ as “grey literature” and declares that it “has no place in the IPCC’s work”. I pointed out in my e-mail that Professor Tol has apparently ensured that the Stern Review is not cited at all in Chapter 10, even though the corresponding chapters in the Fourth Assessment Report include multiple references to the Review. I expressed surprise at Professor Tol’s declaration as it seems inconsistent with the ‘Procedure on the use of literature in IPCC reports‘, which states:

“Priority should be given to peer-reviewed scientific, technical and socio-economic literature if available. It is recognized that other sources provide crucial information for IPCC Reports. These sources may include reports from governments, industry, and research institutions, international and other organizations, or conference proceedings.”

I have yet to receive any reply to the e-mail message.

Professor Tol’s reaction

Professor Tol has not responded to any of my e-mail messages to Working Group II. Indeed, I have not received an e-mail message or any other form of direct communication from Professor Tol since 25 October 2013. Professor Tol did, however, comment on the errors in Chapter 10 on his blog on 4 March 2014 and on Twitter on 5 March. On his blog on 4 April, Professor Tol continued to dispute a number of the errors in his papers.

Bob Ward is Policy and Communications Director at the Grantham Research Institute on Climate Change and the Environment at London School of Economics and Political Science.