Presentation is loading. Please wait.

Presentation is loading. Please wait.

- EGU 2010 ESSI5 - 07 May 2010 - 1 Building on the CMIP5 effort to prepare next steps : integrate community related effort in the every day workflow to.

Published byGilbert Norman Modified over 8 years ago

Similar presentations

Presentation on theme: "- EGU 2010 ESSI5 - 07 May 2010 - 1 Building on the CMIP5 effort to prepare next steps : integrate community related effort in the every day workflow to."— Presentation transcript:

1 - EGU 2010 ESSI5 - 07 May 2010 - 1 Building on the CMIP5 effort to prepare next steps : integrate community related effort in the every day workflow to lower the data distribution and data management burden Sébastien Denvil, Mark Morgan, Ashish Bhardwaj, Martial Mancip, and Patrick Brockmann Climate Modeling Group, IPSL

2 - EGU 2010 ESSI5 - 07 May 2010 - 2 Context : coutdown of the IPCC report  2010  Mid 2011 : Climate simulations  End of 2010  ? : Data distribution  End of 2010  July 2012 : articles submission  September 2013 : IPCC AR5 WG1 plenary session  October 2014 : Nobel Prize?

3 - EGU 2010 ESSI5 - 07 May 2010 - 3 Management of data since years in many climate modeling groups Mainly centralized, store on a SAN OpenDap access on Supercomputing Centre Basic system of data retrieval Access to raw data Security/Authentication/Restriction to data access : not an issue No on demand post-processing No metadata integration No support for high level database query

4 - EGU 2010 ESSI5 - 07 May 2010 - 4 Emerging requirements for Data management Move the data a minimum, keep them close to supercomputing centres if possible  Data access protocol, strong links with computing centres When data needs to be moved do it quickly and with a minimum amount of human intervention  Management of storage resources, fast network Keep a track of what we got, particularly what is on deep storage  Metadata et data catalogues Exploiting a federation of sites  EarthSystemGrid software stack

5 - EGU 2010 ESSI5 - 07 May 2010 - 5 CMIP5 global data amount Raw Data amount lower bound  565 TB Raw Data amount higher bound  1000 TB CMIP5 Distribution (50%)  280-500 TB Global Storage (Raw+Distributed)  800-1500 TB LMDz 0.5° (50 Km)

6 - EGU 2010 ESSI5 - 07 May 2010 - 6 Tropospheric chemistry & aerosols (INCA) Carbon / CO 2 (ORCHIDEE, NEMO/PISCES) Stratospheric chemistry / ozone (REPROBUS) Emissions Land use Volcanoes Solar irradiance Physic – Transport Atmosphere (LMDZ) Surface (ORCHIDEE) Ocean (NEMO/OPA) Sea ice (NEMO/LIM2) Coupler (OASIS) IPSL Earth System Model (ESM) Global climate Regional climate Various kind of Model Impacts studies Dynamical Downscaling (RCM) Statistical Downscaling

7 - EGU 2010 ESSI5 - 07 May 2010 - 7 Earth System Grid Federation to support CMIP5

8 - EGU 2010 ESSI5 - 07 May 2010 - 8 National Level: many partners International Level: many partners

9 - EGU 2010 ESSI5 - 07 May 2010 - 9 Data Node Architecture

10 - EGU 2010 ESSI5 - 07 May 2010 - 10 S 1... S N S 3 S 2 Simulation Execution Environment Input.ini.netCDF.make Events 100=Start 101=Stop Output.ini.netCDF SIMULATION MACHINE

11 - EGU 2010 ESSI5 - 07 May 2010 - 11 S 1... S N S 3 S 2 Simulation Execution Environment Input.ini.netCDF.make Events 100=Start 101=Stop Output.ini.netCDF Prodiguer Simulation Services Python (Async) Message Queues (RabbitMQ) Event Monitor Event Publisher SIMULATION MACHINE

12 - EGU 2010 ESSI5 - 07 May 2010 - 12 S 1... S N S 3 S 2 Simulation Execution Environment Input.ini.netCDF.make Events 100=Start 101=Stop Output.ini.netCDF Prodiguer Simulation Services Python (Async) Message Queues (RabbitMQ) Event Monitor Event Publisher SIMULATION MACHINE PRODIGUER AGGREGATING DATA NODE FIREWALL Base64

13 - EGU 2010 ESSI5 - 07 May 2010 - 13 Meta-Data Publication FRENCH SCIENTIFIC PARTNER/COMPUTING CENTRES DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... (DN=Data Node)

14 - EGU 2010 ESSI5 - 07 May 2010 - 14 Meta-Data Publication FRENCH SCIENTIFIC PARTNER/COMPUTING CENTRES DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... CORE (CMIP5)

15 - EGU 2010 ESSI5 - 07 May 2010 - 15 Meta-Data Publication FRENCH SCIENTIFIC PARTNER/COMPUTING CENTRES DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... CORE (CMIP5)OPERATIONAL

16 - EGU 2010 ESSI5 - 07 May 2010 - 16 Meta-Data Publication FRENCH SCIENTIFIC PARTNER/COMPUTING CENTRES DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... WEB SERVICES (RESTful, AtomPub) CORE (CMIP5)OPERATIONAL

17 - EGU 2010 ESSI5 - 07 May 2010 - 17 Meta-Data Publication FRENCH SCIENTIFIC PARTNER/COMPUTING CENTRES DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY WEB SERVICES (RESTful, AtomPub) OPERATIONALCORE (CMIP5)

18 - EGU 2010 ESSI5 - 07 May 2010 - 18 Meta-Data Publication FRENCH SCIENTIFIC PARTNER/COMPUTING CENTRES DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres CORE (CMIP5)OPERATIONAL

19 - EGU 2010 ESSI5 - 07 May 2010 - 19 Meta-Data Publication FRENCH SCIENTIFIC PARTNER/COMPUTING CENTRES DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres, RDF METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres CORE (CMIP5)OPERATIONAL

20 - EGU 2010 ESSI5 - 07 May 2010 - 20 Meta-Data Publication FRENCH SCIENTIFIC PARTNER/COMPUTING CENTRES DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres, RDF METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres XML Base64 CORE (CMIP5)OPERATIONAL

21 - EGU 2010 ESSI5 - 07 May 2010 - 21 Meta-Data Publication FRENCH SCIENTIFIC PARTNER/COMPUTING CENTRES DN-1 (CCRT) DN-N (Meteo-France) DN-3 (CERFACS) DN-2 (IDRIS)... DATABASE(S) PostGres, RDF-Triple ESG – GATEWAY DATABASE(S) eXist, PostGres, RDF METAFOR / IS-ENES WEB SERVICES (RESTful, AtomPub) PRODIGUER DATABASE(S) PostGres XML HTTPS / X509 XML HTTPS / X509 XML Base64 CORE (CMIP5)OPERATIONAL

22 - EGU 2010 ESSI5 - 07 May 2010 - 22 Conclusions European response to climate simulation proliferation has been built in close collaboration with the ESG-CET American consortium. To come in support to CMIP5 require a work on software environment, data storage, their handling, distribution to users AND a work to describe simulations, their contexts, and their results. ESGF, IS-ENES and METAFOR has been built to support this. The every day workflow and then the every day simulation must benefit from the work done to achieve “CMIP5 like” exercise. The aggregating data node approach is the one we choose. It’s an integration activity, leveraging what’s been done to support CMIP5 like activity. Operational Data « CMIP5 like » Data

Download ppt "- EGU 2010 ESSI5 - 07 May 2010 - 1 Building on the CMIP5 effort to prepare next steps : integrate community related effort in the every day workflow to."

Similar presentations

ESGF and ES-DOC Documenting climate models and their simulations ES-DOC current and future plans Working with ESGF Eric Guilyardi, Balaji, Cecelia DeLuca,

ESGF and ES-DOC Documenting climate models and their simulations ES-DOC current and future plans Working with ESGF Eric Guilyardi, Balaji, Cecelia DeLuca,
Climate Analytics on Global Data Archives Aparna Radhakrishnan 1, Venkatramani Balaji 2 1 DRC/NOAA-GFDL, 2 Princeton University/NOAA-GFDL 2. Use-case 3.

Climate Analytics on Global Data Archives Aparna Radhakrishnan 1, Venkatramani Balaji 2 1 DRC/NOAA-GFDL, 2 Princeton University/NOAA-GFDL 2. Use-case 3.
CMIP5: Overview of the Coupled Model Intercomparison Project Phase 5

CMIP5: Overview of the Coupled Model Intercomparison Project Phase 5
Geophysical Fluid Dynamics Laboratory Review June 30 - July 2, 2009 Geophysical Fluid Dynamics Laboratory Review June 30 - July 2, 2009.

Geophysical Fluid Dynamics Laboratory Review June 30 - July 2, 2009 Geophysical Fluid Dynamics Laboratory Review June 30 - July 2, 2009.
Part 1a: Overview of the UM system

Part 1a: Overview of the UM system
NCAS Unified Model Introduction Part 1a: Overview of the UM system University of Reading, 3-5 December 2014.

NCAS Unified Model Introduction Part 1a: Overview of the UM system University of Reading, 3-5 December 2014.
Sergey Belov, LIT JINR 15 September, NEC’2011, Varna, Bulgaria.

Sergey Belov, LIT JINR 15 September, NEC’2011, Varna, Bulgaria.
Institut Pierre-Simon Laplace (IPSL) Bringing together key laboratories of the Paris area in the field of environmental studies.

Institut Pierre-Simon Laplace (IPSL) Bringing together key laboratories of the Paris area in the field of environmental studies.
CLIMATE SCIENTISTS’ BIG CHALLENGE: REPRODUCIBILITY USING BIG DATA Kyo Lee, Chris Mattmann, and RCMES team Jet Propulsion Laboratory (JPL), Caltech.

CLIMATE SCIENTISTS’ BIG CHALLENGE: REPRODUCIBILITY USING BIG DATA Kyo Lee, Chris Mattmann, and RCMES team Jet Propulsion Laboratory (JPL), Caltech.
Overview of Coupled Model Intercomparison Project (CMIP) and IPCC AR5 Activities Ronald J Stouffer Karl Taylor, Jerry Meehl and many others June 2009.

Overview of Coupled Model Intercomparison Project (CMIP) and IPCC AR5 Activities Ronald J Stouffer Karl Taylor, Jerry Meehl and many others June 2009.
TPAC Digital Library Talk Overview Presenter:Glenn Hyland Tasmanian Partnership for Advanced Computing & Australian Antarctic Division Outline: TPAC Overview.

TPAC Digital Library Talk Overview Presenter:Glenn Hyland Tasmanian Partnership for Advanced Computing & Australian Antarctic Division Outline: TPAC Overview.
Sergey Belov, Tatiana Goloskokova, Vladimir Korenkov, Nikolay Kutovskiy, Danila Oleynik, Artem Petrosyan, Roman Semenov, Alexander Uzhinskiy LIT JINR The.

Sergey Belov, Tatiana Goloskokova, Vladimir Korenkov, Nikolay Kutovskiy, Danila Oleynik, Artem Petrosyan, Roman Semenov, Alexander Uzhinskiy LIT JINR The.
Z EGU2015-8404 Integration of external metadata into the Earth System Grid Federation (ESGF) K. Berger 1, G. Levavasseur 2, M. Stockhause 1, and M. Lautenschlager.

Z EGU Integration of external metadata into the Earth System Grid Federation (ESGF) K. Berger 1, G. Levavasseur 2, M. Stockhause 1, and M. Lautenschlager.
Eric Guilyardi (LOCEAN/IPSL and Univ. Reading) and the Metafor team Common Metadata for Climate Modelling Digital Repositories IS-ENES kick-off meeting.

Eric Guilyardi (LOCEAN/IPSL and Univ. Reading) and the Metafor team Common Metadata for Climate Modelling Digital Repositories IS-ENES kick-off meeting.
Grid for Coupled Ensemble Prediction (GCEP) Keith Haines, William Connolley, Rowan Sutton, Alan Iwi University of Reading, British Antarctic Survey, CCLRC.

Grid for Coupled Ensemble Prediction (GCEP) Keith Haines, William Connolley, Rowan Sutton, Alan Iwi University of Reading, British Antarctic Survey, CCLRC.
CCSM Portal/ESG/ESGC Integration (a PY5 GIG project) Lan Zhao, Carol X. Song Rosen Center for Advanced Computing Purdue University With contributions by:

CCSM Portal/ESG/ESGC Integration (a PY5 GIG project) Lan Zhao, Carol X. Song Rosen Center for Advanced Computing Purdue University With contributions by:
1 Eric Guilyardi and the Metafor team Common Metadata for Climate Modelling Digital Repositories Metafor Dissemination Workshop Abingdon, 14 March 2011.

1 Eric Guilyardi and the Metafor team Common Metadata for Climate Modelling Digital Repositories Metafor Dissemination Workshop Abingdon, 14 March 2011.
IS-ENES [ees-enes] InfraStructure for the European Network for Earth System Modelling IS-ENES will develop a virtual Earth System Modelling Resource Centre.

IS-ENES [ees-enes] InfraStructure for the European Network for Earth System Modelling IS-ENES will develop a virtual Earth System Modelling Resource Centre.
IS-ENES Kick-off meeting Paris, March 2009 Overview of JRA2 European ESM: Performance Enhancement Graham Riley, University of Manchester IS-ENES Kick-off.

IS-ENES Kick-off meeting Paris, March 2009 Overview of JRA2 European ESM: Performance Enhancement Graham Riley, University of Manchester IS-ENES Kick-off.
CIM – The Common Information Model in Climate Research

CIM – The Common Information Model in Climate Research

Similar presentations

Ads by Google