LSE research into how facts travel with integrity has helped to improve the quality of scientific evidence in public policymaking.
What was the problem?
In February 2010, the Dutch Government and Parliament were rocked by a serious factual error contained in a report from the United Nations Intergovernmental Panel on Climate Change (IPCC) about the danger of flooding in the Netherlands.
The report erroneously claimed that 55 per cent of the Netherlands is below sea level, when the true figure is actually 26 per cent.
This incident, referred to as “climategate”, naturally created a serious political embarrassment for the Dutch Minister for Environment and called into question the reliability and integrity of the whole report.
Faced with the serious error in the Intergovernmental Panel's report, the Dutch Minister for Environment asked the Netherlands Environmental Assessment Agency to check the report for any further errors and to report their findings to her and to the Dutch Parliament.
The Agency's Head of Methodology and then Chief Scientist, Dr Arthur Petersen, sought advice from Professor Mary Morgan of LSE's Department of Economic History.
What did we do?
The underlying questions the Dutch problem raised was: just how well do facts travel, and how do we know when to trust them?
Between 2004 and 2010, Morgan had led LSE's 'Travelling Facts' project, which developed a framework for understanding how reliable facts travel within and across the humanities and sciences. The project was a team effort involving two other faculty members, Peter Howlett and Patrick Wallis, a number of doctoral and postdoctoral researchers and several visiting international researchers.
The team developed a framework for understanding how reliable facts circulate which centred on two issues.
The first concerned clarifying what 'travelling well' means, both in terms of whether facts maintain sufficient 'integrity' as they travel and in terms of the 'fruitfulness' in the way they are used by others in different contexts.
The second focussed on the many kinds of 'good company' required to ensure that facts travel well: 'chaperones' such as named scientists producing the facts, the packaging of facts such as in case studies, and the vehicles by which facts travel, for example through computer models or hidden in material objects.
To understand how these two issues came together so that factual knowledge could ‘travel well’, individual scholars in the team investigated historical case studies from everyday culture and from academic disciplines in humanities and the social and natural sciences. The case studies covered a broad range: from the circulation of facts about how to construct buildings in the early modern period; to facts about how rats behave in crowded conditions, used to re-design college dormitories and prisons in the mid-twentieth century; to the use of statistical models to circulate urgent facts about modern epidemics from scientists to policymakers.
Out of this research came innovative methodological recipes for chasing facts back to their production and forwards to their eventual use, as well as generic recipes for providing a 'quality assurance' regime for scientific facts. These recipes could prove extremely valuable for any public or policy body relying on scientific work for its credibility and legitimacy, as the approach from the Netherlands Environmental Assessment Agency so clearly demonstrated.
What happened?
In responding to the Dutch request for advice, Professor Morgan was able to draw on LSE's Travelling Facts project to offer a conceptual approach and a methodology to check the integrity of scientific 'facts' contained in the faulty report.
As she suggested, checking the validity of all the climate science facts it contained was impossible without re-doing all the science. Instead, she proposed a quality assurance check of how the integrity of the report's scientific facts was maintained as they travelled into the final report. This strategy involved a methodology for three sorts of checks: whether the facts produced by scientists were accurately reported, the integrity of the process by which they found their way into the report, and an assessment of any expert judgements used in reporting them.
On the basis of this advice, the Agency set up a process of checking the integrity of facts in the Intergovernmental Panel's report, under the co-leadership of Dr Petersen and involving some 30 staff members over five months. They were able to clarify how the specific error arose, when two separate facts were conflated into one – that 26 per cent of the Netherlands is under sea level and 29 per cent subject to river flooding – and then reported inaccurately.
Although the checkers were able to reassure the Dutch Government and Parliament that the report contained no errors that materially affected its conclusions about the regional impacts of climate change, they found several failures of integrity in the way facts had travelled into summaries.
Crucially for the future, the Agency has used this process to develop a practical taxonomy of likely errors, and it has undertaken to conduct similar quality-assurance exercises for the current round of reports from the IPCC.
Climate science is vitally important to society. Equally vital is maintaining public confidence in the integrity of scientific evidence placed in the public domain.
Search all impact case studies.