LSE research produced a new analysis of why generalising evidence from one context to another is always risky
What was the problem?
Medicine has long upheld the virtues of randomised controlled trials when testing the efficacy of any particular treatment. But how reliably can we generalise the evidence gained from the very precise and controlled conditions of a clinical trial to other, invariably messier, situations?
Even supposing that such randomised research gives us perfect, unbiased information about the effectiveness of an intervention in a particular trial, it does not follow — either in medicine or in the social realm — that the intervention will work in the ‘real world’ outside the trial.
Investigating the philosophical basis for generalising evidence from one context to another becomes even more pressing in non-clinical fields such as social sciences, where practitioners and policy makers are clamouring for evidence-based policies.
What did we do?
LSE's Philosophy Department has a long-standing tradition of investigating the methodology of clinical trials, the virtues and limitations of randomised controlled trials and the general issue of evidence in medicine.
Nancy Cartwright has added to work already undertaken at LSE by Peter Urbach and by John Worrall, developing her own slant on the issues and extending the discussion into the social sciences and the impact of evidence on policy.
Cartwright's original contribution is best understood in the light of her detailed and influential analysis of causation in terms of 'capacities'. She has shown that the results of an experiment can be applied to other contexts only if they include evidence of a 'stable capacity to promote some outcome'. For example, when you dip blue litmus paper in acid it always turns red because the chemical constituents of litmus make this happen, thus representing a 'stable capacity'.
As no randomised controlled trial, or set of such trials, can, when taken alone, provide strong evidence for the existence of such stable capacities, background theories are needed to explain and therefore justify clinical or policy interventions. This conclusion runs counter to the prevailing view that randomised controlled trials provide evidence in and of themselves, independent of theory.
Cartwright's work has important implications for developing policies and practices in both medicine and social sciences. Since no amount of randomised controlled trials or other research methods will provide incontestable evidence to support any particular course of action, policy makers and practitioners must learn how to handle uncertainty. And the way they evaluate 'evidence' must become necessarily much more nuanced.
Cartwright's thinking has directly influenced US and UK bodies concerned with practical issues of evidence and policy-making. Here are three examples drawn from the UK.
In 2011, the Department for International Development commissioned a study aimed at broadening the range of designs and methods for evaluating the impact of aid initiatives. Cartwright acted as an adviser to the study team and her views were sought throughout. An Appendix to the report on models of causality and causal inference draws almost exclusively on her approach, as does the central question posed in one section of the main report: will the intervention work elsewhere?
Donor agencies such as the World Bank and the Organisation for Economic Co-operation and Development (OECD) have organised conferences to discuss the report's findings and used them in staff training. Cartwright's views have also influenced Michael Woolcock, Lead Social Development Specialist at the World Bank's Development Research Group, and other World Bank researchers.
In the field of mental health, the House of Commons Select Committee on Health raised concerns in 2007 about the methodology used by the National Institute for Clinical Excellence (NICE) to evaluate 'talking therapies'. This issue was taken up in 2011 by the New Savoy Partnership, a group of organisations aimed at improving mental health care provision in the UK. Partnership members include the Royal College of Psychiatrists, the United Kingdom Council for Psychotherapy and the mental health charity, Mind.
The Partnership invited Cartwright to join their working party. Roundtable discussion produced a series of questions for a Keynote Panel discussion chaired by Sir Michael Rawlins, then head of the NICE. The Panel included Cartwright as a member.
In their report on the issue, the Partnership's working party endorsed Cartwright's view that an over-reliance on randomised-controlled-trial evidence is “likely to impair and distort guideline recommendations for psychological therapies”. It further suggested that “studies need to focus much more on how treatments work (mechanisms of change) and the factors that support or hinder them in different ... contexts” — in effect, the core of Cartwright's work.
The group subsequently submitted to NICE a consensus statement on evidence (http://www.newsavoypartnership.org/consensus-statement.htm) as a starting point for its review of evaluation methodologies for 'talking therapies'. Cartwright was one of nine initial signatories. This statement in turn fed into a Health Select Committee Inquiry held in Autumn 2012, at which NICE's incoming head, Professor David Haslam, committed himself to reviewing the way his organisation assesses the effectiveness of talking therapies. These discussions are ongoing and changes to the Public Health Guidelines are expected.
Cartwright's views have also influenced the work of LSE Professor of Social Policy Eileen Munro. The pair produced a joint paper on “The limitations of randomized controlled trials in predicting effectiveness”. Its publication coincided with Munro's appointment by the Secretary of State for Education to review the UK's child protection system in the wake of the 'Baby P' scandal and other cases involving vulnerable children.
Munro's approach to evidence in producing her final Report reflected Cartwright's influence. According to Munro, Cartwright's work “helped form ... the whole approach [taken in the Report] to the real role of research findings in [the] area and especially to the role of expertise and expert reasoning”. Munro’s 15 recommendations advocated major changes in the way social workers involved with children are educated and appraised and in the child-protection system in general. All were accepted by government.
Nancy Cartwright subsequently left the LSE to go to the University of Durham. However work on evidence and policy is still very much at the forefront of work in the LSE Philosophy Department, through the work of John Worrall, Katie Steele and others.
Search all impact case studies.