.svg)
.svg)
.svg)
.svg)
.svg)
.svg)
.png)
Paper
Reducing the Impact of I/O Contention in Numerical Weather Prediction Workflows at Scale Using DAOS
Presenter
Nicolau Manubens obtained a licentiate in informatics from the Universitat Autònoma de Barcelona in 2014, and has since worked as a software engineer at various weather and climate forecasting and research institutions. He is currently employed at the European Centre for Medium-Range Weather Forecasts and is a Ph.D. candidate at EPCC, The University of Edinburgh, doing research work on HPC storage for km-scale Numerical Weather Prediction in the context of the Destination Earth initiative.
Description
Operational Numerical Weather Prediction (NWP) workflows are highly data-intensive. Data volumes have increased by many orders of magnitude over the last 40 years, and are expected to continue to do so, especially given the upcoming adoption of Machine Learning in forecast processes. Parallel POSIX-compliant file systems have been the dominant paradigm in data storage and exchange in HPC workflows for many years. This paper presents ECMWF's move beyond the POSIX paradigm, implementing a backend for their storage library to support DAOS --- a novel high-performance object store designed for massively distributed Non-Volatile Memory. This system is demonstrated to be able to outperform the highly mature and optimised POSIX backend when used under high load and contention, as per typical forecast workflow I/O patterns. This work constitutes a significant step forward, beyond the performance constraints imposed by POSIX semantics.