Despite the continuous improvement of weather forecasts over the last few decades, uncertainties due to meteorological measurements and models mean that ensemble forecasts remain critical to weather forecasting. Ensemble forecasts estimate this uncertainty by running multiple simulations over the same forecast horizon. Comparing the different outcomes then paints a more detailed picture of the future.
In this post, we introduce tools for producing ensembles in a fast and cost-effective way.
NVIDIA Earth-2 is a scientific AI platform that provides tools to easily access and deploy data-driven weather prediction models. Among the value propositions of Earth-2 are tools for the accelerated generation of ensemble weather forecasts. These ensembles generate a multitude of possible weather scenarios, offering a more detailed representation of potential weather outcomes, of interest to many industries.
MetDesk, a leading professional weather services company based in the UK, operationalizes AI forecast ensembles using the NVIDIA Earth-2 platform to serve accelerated weather data to the energy trading market. MetDesk’s operational workflow marks a significant leap forward in producing actionable weather data, powered by NVIDIA technology.
Ensemble weather forecasts
Ensemble forecasts with traditional numerical methods are extremely compute-intensive, even on some of the largest HPC clusters. AI weather models, accelerated by the NVIDIA software and hardware stack, can handle similar workloads in seconds. This is especially important for applications, for example in the energy trading sector, that depend on a quick adaptation to changing weather conditions.
Weather governs the generation and consumption of energy, making fast and accurate forecasts vital for anticipating market fluctuations, optimizing trading decisions, and managing risks. Using the NVIDIA Earth-2 platform, MetDesk developed an operational workflow for AI-driven ensemble forecasting, which provides value to traders in practice.
How to create an AI ensemble forecasting system
NVIDIA Earth2Studio is the package for creating AI weather modeling workflows in Python. In the following example, we show you how to make an ensemble forecast in Earth2Studio using the NVIDIA FourCastNet (FCN) AI model.
The example begins by downloading an analysis—the best estimate of the state of the atmosphere—on September 13, 2023, when Hurricane Lee was active off the East Coast of the United States. The analysis is automatically pulled from the data repository of NOAA’s Global Forecasting System (GFS) and cached to disk for later reuse.
Continue by applying perturbations to the analysis using noise sampled from a spherical Gaussian distribution, which causes each ensemble member to produce a slightly different forecast. The forecasts are stored in a Zarr format archive for analysis and visualization. These choices are customizable.
Currently, Earth2Studio offers a range of models, data sources, perturbation methods, and output formats. Earth2Studio will offer more advanced functionalities and optimized pipelines for scale through NVIDIA AI Enterprise.
import numpy as np
from earth2studio.data import GFS
from earth2studio.io import ZarrBackend
from earth2studio.models.px import SFNO
from earth2studio.perturbation import SphericalGaussian
from earth2studio.run import ensemble
# Load the SFNO model package, which downloads the checkpoint from NGC
model = SFNO.load_model(SFNO.load_default_package())
# Use the spherical Gaussian perturbation method
sg = SphericalGaussian(noise_amplitude=5e-5)
# Use the GFS analysis as the data source
data = GFS()
# Use a Zarr archive to store the outputs
chunks = {"ensemble": 1, "time": 1}
io = ZarrBackend(file_name="output.zarr", chunks=chunks)
nsteps = 10 # the number of 6-hour time steps
nensemble = 8 # the number of ensemble members
io = ensemble(
["2023-09-13T00:00"], # start the forecast on 13 September 2023, 00:00 UTC
nsteps,
nensemble,
model,
data,
io,
sg,
# Run 2 ensemble members simultaneously by batching
batch_size=2,
# Save 2 meter temperature and total column vertically-integrated water vapor
output_coords={"variable": np.array(["t2m", "tcwv"])},
)
For more information about the complete original example including visualizations of the results, see Earth2Studio Examples.
Figure 1 shows part of the output data for four of the ensemble members. The visualizations were predicted 60 hours ahead for four of the ensemble members generated with the example script.
MetDesk’s operational workflow
AI ensemble forecasting workflows similar to the one shown earlier are already finding their way to real-world business applications, such as that of MetDesk. Through the course of 2023, it became clear to MetDesk that a significant change was brewing in the world of weather prediction. A string of new machine learning (ML) weather models from some of the world’s largest companies were now showing deterministic skill levels to rival that of the best physics-based numerical weather prediction (NWP) model from the European Centre for Medium-Range Weather Forecasts (ECMWF).
While deterministic forecasts are useful in the short forecast range, ensemble-based systems show more skill and give better guidance from 5–7 days onwards. This is why MetDesk used the NVIDIA Earth-2 platform tools to create ML ensemble output to feed into its range of Trading Weather products.
A selection of perturbation methods and the ability to tune various settings enables MetDesk to create its own unique set of 51 initial conditions based on a single analysis field of the operational ECMWF run (EC-OP). The perturbation method was tuned by hindcasting (producing forecasts for previously occurred weather) over a year to reduce model error and improve ensemble spread. Common measures of weather model skill were computed on the hindcast output, with comparisons made to the EC-OP and ECMWF ensemble forecast (EC-ENS) and the GFS forecast model.
Figure 2 shows the performance of MetDesk’s currently operational 51-member FCN ensemble implementation (MD-FCNE) using Root Mean Square Error (RMSE) and Anomaly Correlation Coefficient (ACC) at the 500 hPa geopotential height.
Both measures show MD-FCNE to have improved skill compared to the EC-OP from day 7 and comparable skill to the GFS ensemble throughout.
In addition to traditional metrics such as the RMSE and ACC, synoptic regime analysis was performed to look at how often the MD-FCNE system provides good guidance on the likely overall weather regime compared to the EC-ENS. Using a combination of the first and second most likely regimes predicted by the ensemble members as a representation of “good guidance,” MD-FCNE performs only slightly worse than the EC-ENS ensemble in the first 10 days and is comparable between days 10 and 15 (Figure 3).
The skill highlighted earlier is one of the reasons that MetDesk’s trading clients incorporate MD-FCNE into their forecasts when considering risks. It is a skillful prediction system in its own right and, when combined with other systems, helps to inform decisions.
Another reason is speed. Using MetDesk’s in-house NVIDIA GPU hardware, a full 15-day, 51-member ensemble prediction can be created before the full set of EC-OP data is available and hours ahead of the full release of the EC-ENS.
This early arrival of data can be used as a useful early indicator of the change in the weather prediction, and when many models are showing similar outputs, MetDesk clients have greater confidence in the predicted scenario. Conversely, when models such as MD-FCNE have different outputs from the ECMWF and NOAA models, forecast confidence is reduced.
There are four main weather parameters that feed into one of MetDesk’s core energy trading products:
- Wind
- Temperature
- Precipitation
- Solar radiation
Wind and temperature are readily available in the core FCN output.
Meanwhile, variables that are not produced directly by FCN can be estimated with diagnostic models that estimate additional variables from FCN output. Earth2Studio offers a catalog of diagnostic models and recipes to train custom diagnostic models, and MetDesk can obtain precipitation with PrecipitationAFNO. For solar radiation, MetDesk leveraged the ability to create custom diagnostics. They worked with the humidity levels native to the FCN output to create cloud diagnostics and then from there radiation.
Medium-range forecasting
Using MetDesk’s operational workflow based on NVIDIA technology, it is possible to perform both medium-range and sub-seasonal weather forecasting. The MD-FCNE system runs 4x per day out to a forecast horizon of 15 days.
As soon as the latest EC-OP analysis data is received, the Earth-2 workflow is started on MetDesk’s NVIDIA GPUs to generate a set of 50 perturbed initial conditions. Along with the original EC-OP analysis, these perturbed states are then used to each initialize their own FCN 15-day forecast, which in turn feeds into the creation of diagnostic parameters.
Within the first 5 minutes of receiving the EC-OP analysis file, MetDesk can produce the 15-day deterministic forecast, including diagnostic parameters that are then streamed into respective trading products and APIs.
Over the course of the following 40 minutes, the 50 members of the ensemble forecast are generated. These are post-processed to create statistics such as the ensemble mean and fed into products such as weather forecast maps, country-weighted predictions, and wind and solar power generation models.
Sub-seasonal forecasting
In addition to the 4x daily 15-day ensemble predictions, MetDesk also creates a daily 50-day ensemble forecast (MD-FCN50) comprising 50 ensemble members for customers who are looking into the sub-seasonal range.
Figure 5 shows that the FCN skill is comparable with that of the EC46 system (with bias corrections). One benefit of the FCN model is its stability when run to longer lead times. The huge speed advantages of GPU-accelerated ML forecast systems compared to traditional NWP enable MetDesk to serve MD-FCN50 predictions to clients nearly 12 hours earlier than the ECMWF’s 46-day sub-seasonal system. This means that they deliver data within the main European daytime trading period rather than later the same evening after markets have closed.
Accelerate and scale ensemble inference with NVIDIA NIM
Improvements in speed and resource efficiency are two of the main drivers for the adoption of AI models for weather forecasting.
NVIDIA NIM is a set of easy-to-use microservices designed for secure, reliable deployment of high-performance AI model inferencing on-prem and in the cloud. NIM offers enterprise-grade inference performance and scalability while giving you complete control over the integration of the latest AI models into production workflows. Offered through NVIDIA AI Enterprise, NIM is provided with enterprise support, rigorous validation, and regular security updates.
NVIDIA NIM accelerates ensemble AI weather forecasting workflows while leaving full control and the ability to customize in your hands.
Figure 7 shows the AI ensemble forecast workflow at MetDesk with NIM handling the core forecast step.
- The process starts by downloading and pre-processing analysis data from ECMWF.
- NIM then completes an efficient ensemble forecast using SFNO and calculates additional variables through pre-trained and custom diagnostic models.
- The output data is post-processed and ingested into downstream systems for insights generation.
Using Earth2Studio and NIM, combined with NVIDIA GPUs, we reduced the operational MetDesk workflow runtime for a 15-day forecast with 50 ensemble members from 45 minutes down to 2 minutes on four NVIDIA H100 GPUs.
NIM makes scaling to more GPUs trivial. In fact, the same workload can be processed in seconds when submitted to 50 NVIDIA H100 GPUs in parallel.
NIM provides a production setup for Earth2Studio-like workflows in the form of a container that is easy to deploy using Kubernetes. Inference is triggered through a standardized API, with a configurable number of forecast steps and the set of required output variables. As the data volume to be handled can be considerable, NIM has optimized I/O capabilities with reads and writes directly from and to disk.
In addition to the models already provided through Earth2Studio, NIM enables the integration of custom diagnostic models. Using NVIDIA Triton Inference Server, NIM retains the benefits of the NVIDIA Triton feature set, including dynamic batching, advanced scheduling, Prometheus logging, and more.
Key benefits for the energy trading business
Access to accurate weather forecasts is paramount in the energy trading business, especially with the continued expansion of renewable energy production. Weather not only directly impacts the generation of wind, solar, and hydroelectric energy but also energy consumption. Extreme weather events can further have a disruptive effect on energy infrastructure and supply chains.
With AI weather forecast models now rivaling the accuracy of numerical predictions, the magnitudes of improvement in speed allow traders to react earlier to imminent weather conditions than previously thought possible. Where minutes and seconds can make all the difference, any production inference setup must work at peak performance.
NVIDIA NIM provides a robust and easy solution for exactly this purpose. It can produce a 15-day ensemble forecast in seconds.
MetDesk, as an early adopter of this technology, brings immense value to the energy trading sector. MetDesk’s infrastructure team can rely on standard interfaces and deployment workflows provided by NIM. Instead of building an inference system from scratch, MetDesk’s developers gain time and focus on customizing the workflow to their customers’ needs.
If you are interested in trying out an early-access version of the Earth-2 ensemble NIM in your own proprietary workflows, reach out to the Earth-2 team.
