National Center for Atmospheric Research A New Parallel Python Tool for the Standardization of Earth System Model Data Kevin Paul Sheri Mickelson John M. Dennis National Center for Atmospheric Research Application Scalability And Performance (ASAP) I/O & Workflow Applications (IOWA)
Big Data in Earth System Modeling NCAR’s Community Earth System Model: Massively parallel (MPI-based) Higher resolution simulations … “Big Data”! Coupled Model Intercomparison Project: CMIP5 (2010-2013): 20 different institutions from around the world! CESM: 2.5 PB generated 170 TB published “First to produce, last to publish!” CMIP6 (2017-2020): EXPECT: Up to 25 PB generated from CESM alone! A New Parallel Python Tool for Standardization of ESM Data
CESM Publication Workflow Time Slices Time Series CESM Run Time-Series Generation Standardization Publication Time-Slice Format: (synoptic or history-file) All variables from model in every file, spanning a short time Time-Series Format: (single-field) Each variable from model in single file, spanning a long time Better for analysis! Standardization: Produce the data requested by the MIP Standard variable names, standard dimension names, etc. Publication: Obtaining DOI, “tagging” data, publication data location, etc. A New Parallel Python Tool for Standardization of ESM Data
CESM Publication Workflow Time Slices Time Series CESM Run Time-Series Generation Standardization Publication NCO NCO Pre-CMIP6: All performed using the NetCDF Operators (NCO) App performed in serial! A New Parallel Python Tool for Standardization of ESM Data
CESM Publication Workflow Time Slices Time Series CESM Run Time-Series Generation Standardization Publication PyReshaper PyConform Pre-CMIP6: All performed using the NetCDF Operators (NCO) App performed in serial! CMIP6: All performed using parallel (MPI) Python tools Time-Series Generation PyReshaper (12x Speedup) Standardization PyConform A New Parallel Python Tool for Standardization of ESM Data
MIP Data Standards Every MIP requests specific data: File formats (e.g., NetCDF4) Names of files and directory structure File attributes (e.g., institution, MIP name, …) Names of dimensions (e.g., lat, lon, …) Names of variables (e.g., psl, ta, tas, …) Dimensions of variables Variable data types (e.g., float, double, …) Attributes of variables (e.g., units, …) Ranges of time (e.g., 2006 to 2100) Lots of information! Very complicated! Could be very different from model output! A New Parallel Python Tool for Standardization of ESM Data
The Limitation of Standards MIP Standards say “what” you need MIP Standards don’t say “how” to get it Suppose “v” is requested by the MIP What is its data? v = ? Easy case… v = v or v = V Also, pretty easy… v = X Harder… v = X1 + somefunction(X2 - X3) Also need a set of rules, or “definitions”, which require special knowledge about the model itself! A New Parallel Python Tool for Standardization of ESM Data
MIP Data Standardization Standardization Process: Get the MIP-specific “standards” Get the model-specific “definitions” Combine! “standards” + “definitions” = “standardization” Perform the operations needed by the “standardization” Specification Computation PyConform: 2 utilities Specification: creates a “standardization file” Computation: parallel graph-based computation A New Parallel Python Tool for Standardization of ESM Data
PyConform: Specification Get the MIP-specific “standards”: CMOR (from PCMDI/LLNL) No clear separation of “specification” from “computation” DreqPy (from UK Met Office) Database-like collection of MIP standards Get the model-specific “definitions”: Requires input from the scientists! Slow in producing, but no other way? Combine! Serial Python command-line tool Creates a single file used as input in… DreqPy x = X1 + X2 y = X1 - X2 z = Z u = function1(U,V) v = function2(U,V) definitions.txt A New Parallel Python Tool for Standardization of ESM Data
PyConform: Computation “x = X1 + X2” Read: X1[i] Validate: > minimum < maximum dimensions = [j] et cetera Evaluate: (X1+X2)[i] Map: ij Read: X2[i] Write: x[j] y[j] File Read: X1[i] Validate: > minimum < maximum dimensions = [j] et cetera Evaluate: (X1-X2)[i] Map: ij Read: X2[i] “y = X1 - X2” A New Parallel Python Tool for Standardization of ESM Data
Evaluate Nodes: (Almost) anything can be done! All data passed along the “Data Flow” has… Units allows for implicit conversion! Dimensions allows for implicit transposition! Basic element-wise arithmetic e.g., +, -, *, /, ** Element-wise math functions e.g., sqrt(x) User-defined functions: Subclass from PyConform’s Function class Input: numbers, strings, variable-names, … Many already written for testing: e.g., vinth2p(T,…), invert_dim(T,”dim”) A New Parallel Python Tool for Standardization of ESM Data
PyConform: Testing Super-linear speedups! CESM Case Name CMIP5 Table Input Dataset Size Output Dataset Size Original Serial Runtime PyConform Parallel Runtime (16 Procs) SPEEDUP b40.rcp4_5.1deg.006 Amon 84 GB 62 GB 4313 sec 113 sec 38x b40.20th.track1.1deg.012 135 GB 102 GB 7157 sec 456 sec 16x 3hr 540 GB 506 GB 22267 sec 645 sec 34x Super-linear speedups! NOTE: …Serial performance much better Much more testing to be done! Achieving bit-for-bit agreement is time-consuming Coming up with the “definitions” can be hard! A New Parallel Python Tool for Standardization of ESM Data
Thank you! Questions? A New Parallel Python Tool for Standardization of ESM Data