Forecasting the grid: how statistics keeps the lights on

QiWei Yao sets out LSE research carried out with energy provider EDF on electricity load forecasting, including the creation of a new model which has led to reductions in forecasting errors.

Geopolitical and supply pressures, pricing mechanics and faster market trading speeds are all making electricity markets more volatile, competitive and uncertain. Adding weather-dependent renewables such as wind and solar to the grid brings even more unpredictability. Accurate load forecasting (predicting how much electricity people will use, and when) is therefore essential for deciding how much power to generate, how to move it around the grid, and how to manage the risk of shortages.

Researchers at the LSE Department of Statistics have worked with EDF R&D in Paris on load forecasting since 2010. Our first project looked at the electricity loads in whole France between 1996 and 2009 recorded every half hour, giving 48 readings a day. Plotted together, these form a daily curve: low overnight, rising in the morning and peaking in the evening.

The challenge is that this pattern keeps changing. Winter days look different from summer days, and weekdays differ from weekends. Many traditional forecasting methods assume that the data behave in the same way over time, so these shifts cause problems. Our approach was to treat each whole day's curve as a single object and forecast the shape of the next day as a whole. Seen this way, the changes within a day become much easier to model from one day to the next (Cho et al. 2013, JASA, 108, 7–21).

Our second project focused on forecasting uncertainty. Most forecasts give a single best guess, such as 'demand at 7 pm tomorrow will be 70 gigawatts'. Yet the future is uncertain, and the most expensive problems come from extreme events such as cold snaps. A more useful forecast gives a range: 'we are 95% confident that demand at 7 pm will be between 66 and 74 gigawatts'. Ranges like this help grid operators prepare for unusual situations and avoid high costs and supply crises.

Producing a range for a single time of day is a well-studied problem. Producing one for a whole day is much harder. Suppose we make a separate 95% range for each of tomorrow's 48 half-hours. Each range is very likely to be right on its own, but the chance that all 48 are right at the same time is much lower, because there are 48 separate chances for something to fall outside. Grid managers need a single band that contains the entire day's curve with a level of confidence they choose in advance.

We developed a method to do exactly this. It breaks the difficult problem of predicting a whole curve into a handful of simpler, standard predictions, then combines them to produce a large set of plausible curves for tomorrow. These curves are ranked from the most typical to the most extreme, much as the median picks out the middle value in a list of numbers. The outer edges of the set form the forecast band. To our knowledge, this was the first method to give a properly calibrated range for a whole day's curve (Xu et al. 2021, Applied Energy, 117465).

Our latest project looks at forecasting when the rules suddenly change. Forecasts are made one step at a time, before the outcome is known, and the method learns from each new reading as it arrives. The COVID-19 lockdown is a vivid example of sudden change: as offices, schools and shops closed, French electricity use changed almost overnight. Models built on normal years suddenly found that the past was a poor guide to the future.

This raises a basic question: how much should a forecaster trust old data? Relying on a long history gives stable forecasts that are slow to notice change. Focusing only on recent data reacts quickly but gets thrown around by random noise. The right balance is not known in advance, and it can change over time.

Our method, ‘MELO’ (Memory-hedged Exponentially Weighted Least-Squares Online aggregation), avoids having to choose. It can sit on top of whatever forecasting models are already in use. Alongside them, it runs a panel of extra forecasters, each with a different length of memory: some pay most attention to the last few days, others to a much longer history. After each new reading, MELO checks which forecasters have been doing well and gives them more weight in the next forecast, much like listening more closely to the advisers whose advice has recently been paying off.

We have proved mathematically that MELO performs nearly as well as an all-knowing 'oracle' that could pick, with hindsight, the best forecaster or the best changing combination of forecasters. Importantly, this guarantee holds for any sequence of data, not just on average.

We tested MELO on French national electricity demand through the COVID-19 lockdown. It was given no information about lockdowns or their dates; it had to notice the change for itself. It cut overall forecasting error by 34.7% compared with combining the original models without MELO's extra forecasters. It also outperformed leading deep learning and AI models, even when one of them was told about the lockdown policies directly. MELO needs only a few quick calculations each time new data arrives, rather than retraining a large model from scratch (Wang et al. 2026, arXiv:2605.06541). This makes it a lightweight, energy-efficient approach in the spirit of green computing.

By Professor Qiwei Yao, LSE Department of Statistics