Minisymposium

AtmoRep is a first example of probabilistic multi-purpose model that extends the concept of representation learning to Earth system science, and in particular to atmospheric dynamics. Starting from a single pre-trained architecture used as a backbone, AtmoRep is able to achieve skilful results in multiple zero-shot applications such as nowcasting, temporal interpolation and scenario generation, compared to the state-of-the-art approaches. Thanks to a novel definition of the loss, the model is also probabilistic by design, as it outputs a set of ensemble members for each task, with well calibrated distributions as proven for weather forecasting. The first part of the talk will focus on the innovative aspects of the model architecture, with particular attention to the local space-time approach and the flexible plug-and-play design. The second part of the talk will show the current results for the most relevant applications, while the last part of the talk will focus on the future foreseen developments.

Significant progress in the development of highly adaptable and reusable Artificial Intelligence (AI) models is expected to have a significant impact on Earth science and remote sensing. Foundation Models (FMs), AI models designed to replace task-specific models, are increasingly being recognized for their versatility across numerous downstream applications. These models, trained using self-supervised techniques on any type of sequence data, circumvent the need for large annotated datasets, a major bottleneck in traditional AI model development. FMs can be applied to downstream tasks using few-shot learning and fine-tuning, significantly reducing the need for large labeled training datasets and computational resources. In contrast to task-specific models, large-scale FMs facilitate the processing of multi-modal data from different satellites and additional data modalities in order to obtain improved model skills in the Earth observation domain.

Foundation models have demonstrated great success in the field of natural language processing (NLP) and for other vision-based tasks (e.g., DALL-E). With the rise of data-driven, global weather forecast models, researchers have begun to create foundation models for the Earth System. There several foundation models in development (e.g., ClimaX) to allow for rapid fine-tuning to specific tasks such as weather forecasting or climate. However, several open questions remain on how well this foundation model approach will work with such a complex and diverse set of tasks typically needed for weather and climate. Here, we evaluate the ability for ClimaX to perform downstream tasks not seen during the pre-training phase. Specifically, we look at two tasks using ClimaX fine-tuned on ERA5 to 1) perform data assimilation using real, in-situ observations of the atmosphere and 2) replace the output layer of the foundation model with one that is parameterized by a Gaussian to perform uncertainty quantification (UQ). We also use the lessons learned from these experiments to develop a state-of-the-art weather forecasting model called Stormer. Stormer is a simple transformer-based model that achieves state-of-the-art performance on weather forecasting with minimal changes to the standard transformer backbone.

The last part of the workshop will be an interactive discussion on the following open questions: “Where foundation models could be useful in Earth system science?”, “What would be the advantages and limitations of foundation models in Earth system science?” and “Can we start building an agenda?”. Given the diverse audience of the event, we foreseen this part as a brainstorming occasion also across different scientific communities. For this reason, the discussion will focus on more general questions like “To which extent can we trust and utilize a foundation model for scientific predictions?” and “Will these models be able to replace the current numerical approaches?”. For this interactive Q&A part the speakers will be invited as panelists, together with experts like Karthik Kashinath from NVIDIA and Sebastian Schemm from ETH Zurich to ensure a broader overview of the ongoing efforts in the field.