“curator” DB design Curator meeting, GFDL, Sep 20

2 Why RDBMS  A lot of information:  Model metadata  Experiments metadata  Institution/user metadata  Data metadata  Mostly it’s in textual form  Information is internally linked tightly that can be easy to express by means of relational databases.  Relational databases have well developed means for searching and extracting procedures (SQL query language and program interfaces for any language) as for local as well as for remote user.  Very reliable, safety technology. Curator meeting, GFDL, Sep 20

3 Desirable Features of Model Data Factory  Relational Database storing metadata, containing description of  model components and model configuration  scenarios  postprocessing (model output and CMOR) directives  experiments  variables  formalized rules of Quality Control  data locations  task scheduler  users and groups accounts  XML as data exchange format  for compliance with FRE  working format of existing third party software  good fitted for hierarchical metadata description  prevalent in world, easy to exchange with others Data Portals  Model Builder (FMS Runtime Environment in GFDL)  checks out available model components from DB  chooses model datasets from DB  sets postprocessing directives  checks components and configurations compatibility  builds executable application and runs it  write metadata about experiment into DB (model configuration, scenario, project, organization/user, postprocessing) Curator meeting, GFDL, Sep 20

4 Desirable Features of Model Data Factory (continue)  Climate Model Output Rewriter (CMOR) subsystem  prepares data consistently with specific project requirements  Data Publisher  transfer data to Data Portal storage in accordance to settings from DB  Data Portal Software Package  Configuration Manager (configures Aggregation Server and Data Portal Interface)  Search Catalog Engine  Data Subsampling Engine  Data Computation Engine  Data Visualization  Data Delivery Manager Curator meeting, GFDL, Sep 20

5 Standard scenario of functioning Model Data Factory (ideal picture)  Scientist builds model in FRE using available model components, datasets and forcing scenario.  FRE puts metadata about built model, scenario, experiment into “curator” DB and runs experiment;  Postprocessing subsystem extracts metadata about postprocessing plan from “curator” DB and executes it, and on finish puts metadata about processed experiment back into DB.  Data Publisher (DP) regularly checks “curator” DB for new experiments marked as “public” and if finds any invokes CMOR.  CMOR goes to “curator” DB for metadata and processes needed data following metadata instructions.  DP calls QAC and then transfers data to Data Portal storage.  Configuration Manager configures Aggregation Server and Data Portal Interface and puts records about new public data in “curator” DB.  End of process, data is ready to go. Curator meeting, GFDL, Sep 20

6 Common functionality schema of ‘Model Data Factory’ Curator meeting, GFDL, Sep 20

7 Database Compartments:  Model Metadata Compartment contains models’ descriptions, allows to build coupled model of needed configuration contains models’ descriptions, allows to build coupled model of needed configuration  Variables Compartment List of all related physical variables List of all related physical variables  Workflow Compartment contains scenarios, experiments, institutions, projects and users info contains scenarios, experiments, institutions, projects and users info  Postprocessing Compartment defines postprocessing plan for conducting experiment defines postprocessing plan for conducting experiment  Data Portal Compartment contains info about experiments data contains info about experiments data Database ‘curator’ design Database ‘curator ’ design Curator meeting, GFDL, Sep 20

8 MySQL DB CURATOR Curator meeting, GFDL, Sep 20

9 Model Metadata Compartment (in development) Coupled_Models Model_List Component_Medias Models Experiments Workflow Compartment Variables Variables Compartment Curator meeting, GFDL, Sep 20

10 Data Samples from Model Compartment Components_Medias Coupled_Models Model_List Models Curator meeting, GFDL, Sep 20

11 Variables Compartment Projects Workflow Compartment Variables Variable_Bundles Variable_Lists Variable_List_Contents Proj_Var_Names Curator meeting, GFDL, Sep 20

12 Variable_Lists Variable_List_Contents Data Sample from Variables Compartment Proj_Var_Names Variables Variable_Bundles Curator meeting, GFDL, Sep 20

13 Workflow Compartment (in development) InstitutionsGFDL_USERS Experiment_Status Realization Projects Experiments Scenarios Curator meeting, GFDL, Sep 20

14 Data Samples from Workflow Compartment Experiments Scenarios Curator meeting, GFDL, Sep 20

15 Coupled_Models Postprocessing Compartment PP_Units Post_Proc PP_Content Data Samples from Postprocessing Compartment PP_Units PP_Content Variable_Lists Projects GFDL_USERS Average_Periods Curator meeting, GFDL, Sep 20

16 Data Portal Compartment MissedData_Descriptors Data_GridsData_Files Variables Experiments Variable_Bundles Coupled_Models Curator meeting, GFDL, Sep 20

17 Data Samples from Data Portal Compartments Data_Files Data_Grids MissedData_Descriptors Curator meeting, GFDL, Sep 20

18 “curator” DB is in use now: CM2.0 CM2.0 CM2.0 CM2.1 CM2.1 CM2.1 Curator meeting, GFDL, Sep 20

19 Future Development  Bring DB terms to conventional terminology.  Set up model metadata schema standards and create tables in “curator” DB following this schema.  Fill these tables with real metadata extracted from models of GFDL, CCSM, MIT and from ESMF Component Database.  Implement tables for observation data metadata.  Implement DODS aggregated data support.  Build XML bridge for XML transcoding DB input/output Curator meeting, GFDL, Sep 20

20 END ENDQuestions? Suggestions? Suggestions? Objections? Objections?Thanks! Curator meeting, GFDL, Sep 20