Button Text
Back

P34 - Ionbeam: Scalable IoT Streaming Infrastructure for Meteorology

This is some text inside of a div block.
This is some text inside of a div block.
-
This is some text inside of a div block.
CEST
Climate, Weather and Earth Sciences
Chemistry and Materials
Computer Science, Machine Learning, and Applied Mathematics
Applied Social Sciences and Humanities
Engineering
Life Sciences
Physics
This is some text inside of a div block.

Description

The European Centre for Medium-Range Weather Forecasts’ (ECMWF) relies on extensive meteorological observations, sourced from ground-based stations, aircraft, and satellites. Low-cost Internet-of-Things (IoT) devices present an opportunity to access observations at higher frequency, higher spatial resolution with more parameters. While much higher in volume, IoT data cannot be expected to be curated, standardised or reliable. We present the design of a prototype system for ingesting, standardising, quality assessing, encoding storing and serving these novel data into a high-performance scientific infrastructure. ECMWF’s data infrastructure and workflows are all driven by access to data according to semantically and scientifically meaningful data. This novel infrastructure is based on the same principles, carefully bringing the highly heterogenous IoT data into this curated, metadata driven data ecosystem. Further design goals include scalability to high data throughput, fault tolerance to invalid data, system configurability and FAIR accessibility of data. The prototype system adopts a message-driven architecture. Messages are self-describing object carrying their own metadata, according to a domain-specific language. This description is used to route data through the system and enable the system to decide which specific transformations are required. THE encoded objects are stored in the FDB, ECMWF’s domain-specific object store for meteorological data.

Presenter(s)

Presenter

Thomas
Hodson
-
ECMWF

Tom Hodson is a research software engineer working on IoT data ingestion at ECMWF.

Authors