Can randomised controlled trials help develop effective green innovation policies?

Randomised controlled trials are a proven tool for understanding the causal effects of policies. Esin Serin asks whether they can play an increased role in informing the development of effective green innovation policies in the UK at this critical time for climate action.

Technology, alongside behaviour change, is essential for the UK to bring its emissions to net zero by 2050. Progress is needed to develop new technologies for reducing emissions where current technologies fall short. The UK is already implementing policies to drive the required innovation in low-carbon technologies, with significant capital deployed along the way. But how can policymakers know whether their policies are working – or working better than alternatives, or that they are creating additional benefits and not just subsidising innovation that would have happened anyway? Randomised controlled trials (RCTs) offer a rigorous way of answering these questions.

Understanding the efficacy of green innovation policies

Evidence on the efficacy of green innovation policies is needed to identify the most cost-effective pathways to net zero, ensuring that public resources are not wasted but are spent on policies that create real positive impacts. There may be many policies not working as intended. For example, a review of studies evaluating direct funding programmes for innovation finds eight papers showing positive impacts of such programmes on patent outputs but another six showing either mixed or zero impact. Another paper – drawing on evidence from RCTs that go beyond a sole focus on innovation – goes further to suggest “80% of things don’t work”.

Evidence on what works can help unlock significant growth opportunities for the UK if it can build comparative advantage in and capture a valuable share of the growing global market for green technologies. The UK government has already adopted innovation-led growth as a long-term ambition and embedded it in its net zero strategy. Research from the LSE’s Centre for Economic Performance and Grantham Research Institute has shown that the UK has a strong base, being specialised in green technologies overall and in some specific technologies where the estimated returns on public investment in innovation are especially high, such as in offshore wind, tidal stream and carbon capture and storage.

Using RCTs to inform green innovation policies

RCTs involve assigning subjects such as individuals, households or firms (in a randomised manner) to either a treatment group that receives the policy being studied (e.g. a research and development grant) or a control group that does not.

Given subjects are randomly assigned to one group or the other, the control group should, in theory, be similar to the treatment group, and so the difference in average outcomes between the two groups (e.g. R&D spending, patent applications) can be associated with the policy itself rather than individual characteristics or wider trends or influences.

Despite their potential, RCTs have seen very limited use in the green innovation context so far. Indeed, we identified only 29 trials globally (five being in the UK) that are either planned, conducted or under way. This is unsurprising given the challenges of implementing RCTs, ranging from cost and time requirements to the ethical implications of allocating a policy on a randomised basis, even if it is for learning purposes.

Views on RCTs are also varied: some argue they represent a methodological ‘gold standard’ for proving causal impact, while others view them as insensitive to the importance of context as they can measure impact with high confidence but on their own, tend to provide little information about how and why the measured impact came about. There are also unique challenges to implementing RCTs in the specific context of green innovation.

Can RCTs have a greater role?

Among the priority areas of green innovation for the UK, we found that energy storage and flexibility, homes and buildings, and hydrogen may present fertile grounds for RCT-based evidence building on innovation policies. This is primarily because these areas involve some domestic-scale, consumer-facing technologies that are likely to be developed by a large number of innovators, making them conducive to evidence-building from RCTs.

RCTs’ contribution to achieving green outcomes can be maximised if they are used as a way to make innovation policy more innovative, shifting the focus from ‘evaluating’ if a single policy design is working to approaching the policy as an experiment in its entirety – that is, exploring a range of policy ideas, testing promising ones on a small scale and basing subsequent decisions on these lessons.

The Business Basics Fund from the Department for Business, Energy and Industrial Strategy uses an experimental approach, by supporting trials of different ideas that encourage small and medium-sized enterprises (SMEs) to adopt productivity-boosting technologies and management practices. Larger trials seeking access to the fund need to have an evaluation methodology such as an RCT that will make it possible to draw robust conclusions on what does or does not work. The recently launched Green Homes Finance Accelerator takes a similar approach, allocating a portion of its £20 million funding to pilot a range of innovative financing mechanisms that encourage domestic energy efficiency and low-carbon heating retrofits.

RCTs as part of a broader culture of evidence-based policymaking

Of course, RCTs are not the only source of evidence. Policymakers can draw on years of learning by doing and other rigorous methods of policy evaluation on innovation broadly (see this toolkit) as well as on green innovation specifically. For example, we know that progress made to date with some key green technologies was driven not by technology-neutral, generic research and development investments or carbon pricing but instead by innovation policy packages involving deliberate technology choices.

The UK’s Contracts for Difference mechanism was not tested under an RCT, but no one can deny its role in bringing down the costs of offshore wind in the UK by around 70% over a decade. The investment signal that the policy sent also had wider and longer-term impacts on the development of the sector that may not have been captured under an RCT design.

Randomised controlled trials are one important enabler of evidence-based policymaking rather than an end in themselves. We cannot randomise every green innovation policy, but we can be more proactive about recognising opportunities for using RCTs to learn about the role that different policy instruments can play for achieving specific technological objectives. As a result, we may learn that not every policy is having its intended impact, but in so doing, we might build an evidence base to accelerate net zero-aligned innovation and growth.

Read the full report: Randomised controlled trials: Can they inform the development of green innovation policies in the UK? by Esin Serin, Nils Handler, Lewis Morey and Apurva Mujual.

This commentary has been co-published with the LSE British Politics and Policy Blog.