Minisymposium Presentation

In modern experimental facilities, advanced detectors gather data at rates of multiple gigabytes per second. To manage this massive influx of data effectively, experiment-time analysis techniques that allow tailoring of experiment run (e.g. filter out irrelevant data elements ) are required. Achieving such automation at scale, necessitates the development of distributed computing pipelines, which connect instruments, computers (for analysis, simulation, and AI model training), edge computing systems (for analysis), data repositories, and metadata catalogs. Further, resulting data or information needs to be delivered in a form that can be easily discovered and accessed by end users. The Globus platform (globus.org) offers various capabilities to that end: reliable and secure data and compute management, and managed task orchestrations system for automation, all underpinned by sophisticated standards compliant security to bridge across distributed systems. In this talk, we’ll describe these capabilities and their application for accelerating time to science. Drawing from our experiences processing data from various beamlines at Argonne National Lab’s Advanced Photon Source by utilizing powerful capabilities at the Argonne Leadership Computing Facility and other services, we discuss the common patterns and implementation, along with the implications of these methods for both facility operators and scientists.