1. ESP workshop, Sept 2003 the Earth System Grid data portal http://www.earthsystemgrid.org/ presented by Luca Cinquini (NCAR/SCD/VETS) Acknowledgments: ESG collaboration: ANL, ISI, LBNL, LLNL, NCAR, ORNL funding: DOE/SciDAC

2. ESP workshop, Sept 2003 Introduction ESG goal: build a grid for the geosciences that enables management, discovery, distributed access, processing, & analysis of distributed terascale climate research data Use Globus and Grid technologies The ESG challenges (with respect to CDP):  Federation of distinct DOE and NSF data repositories, each with different data access policies and procedures  Interoperate between diverse remote storages at distinct sites (HPSS at ORNL, NERSC and MSS at NCAR)  Explicitely deal with file replicas management  Use powerful but new technology The ESG advantages: data scope is more limited (initially only climate model data in netCDF format)

3. ESP workshop, Sept 2003 Overview of ESG web portal Currently under major revision  Moving to MVC (Tomcat/Struts) architecture  New UI under development  Developing/plugging new applications Goal: easy-to-use, grid-enabled, production-level web portal by January 2004 to serve CCSM IPCC data to wide community of climate scientists  will be instrumental in determining the success of ESG ESG web portal components:  Security  Metadata  Data access

4. ESP workshop, Sept 2003 ESG metadata architecture ESG or climate metadata  Collection-level description metadata  Hierarchical information (project  ensamble  simulation  dataset  logical datafile)  Stored in set of relational tables in OGSA-DAI MySQL database at ISI (RDBMS with OGSA-alpha GS interface)  Input and output of database is XML  ESG web portal allows several power user queries of the db  Base for simple free text search and discovery of data

5. ESP workshop, Sept 2003 ESG metadata architecture Location/replica metadata  Indicates the physical locations of the many copies of a single logical file (replicas): lfn  [1..n] pfn  Stored in a system of distributed RLS (Replica Location Services): cross-updating grid-enabled MySQL databases installed at each site  Each RLS is composed of two parts:  LRC (Location Replica Catalog): stores lfn  [1..n]pfn at site  RLI (Replica Location Index): stores all lfns at that site, notifies federated sites  Any RLI in the system can be used as starting point for obtaining all replicas (at any site) of a given lfn Data search and discovery example: “pcm”

6. ESP workshop, Sept 2003 ESG metadata topology RLI MSS HPSS RLI HPSS RLI DISK RLI DISK OGSA-DAI MySQL RDBMS ESG WEB PORTAL Tomcat/Struts cross-update query LBNL ISI LLNL NCAR ORNL LRC

7. ESP workshop, Sept 2003 Activity diagram: data search and discovery query string ESG userESG system OGSA-DAI DB at ISI [0-n] simulation ids OGSA-DAI DB at ISI [0-m] datasets ids OGSA-DAI DB at ISI [0-l] lfns RLI at NCAR [0-l] LRC [0-k] pfn LRC at NCAR, LBNL, LLNL, ORNL step 1 step 2 step 3

8. ESP workshop, Sept 2003 ESG web portal authentication Security is at the centre of Globus/Grid software (GSI – Grid Security Infrastructure) A blessing and a curse at the same time: security can be an undesired burden on the user ESG graduated security model:  Data publishing operations (updating databases, moving data between sites) will be performed from the shell and require certificate-based authentication/authorization  Data access operations (metadata search and browsing, data download, data analysis) will be performed through the web portal and require only encrypted username/password – certificates will be used in the background but never exposed to the user  Note that some operations (retrieval from MSS, HPSS) are subject to local policies ESG currently developing “Mailman-like” grid-enabled web-based registration/authentication package to be deployed on ESG web portal

9. ESP workshop, Sept 2003 Activity diagram: new user registration web browser ESG web portal 1. email username password role email client 2. confirmation request 3. confirmation response email client web browser 4. processing request ESG userESG administrator 5. accept, reject ESG CA 6. certificate request, response MyProxy server 7. store certificate

10. ESP workshop, Sept 2003 Activity diagram: new user login web browser ESG web portal 1. username, password grid restricted application 4. proxy certificate ESG user MyProxy server 2. authentication request 3. proxy certificate

11. ESP workshop, Sept 2003 Web-based data retrieval from online & remote storage HRM (Hierarchical Resource Manager)  Allows (massive) data movement between a site local disk cache and remote storage (HPSS, MSS and others are supported)  Allows transfer of data between disks at separate sites through gridFTP (fast, parallel, tunable transfer protocol)  Many features: reliability, cache management, monitoring etc.

12. ESP workshop, Sept 2003 ESG data retrieval MSS HRM HPSS HRM HPSS HRM DISK HRM DISK ESG WEB PORTAL Tomcat/Struts gridFTP MyProxy authenticate GRAM GATEKEEPER submit execute gridFTP SERVER LBNL LLNL NCAR ORNL

13. ESP workshop, Sept 2003 Future development of ESG web portal Goal: have friendly, easy, production-level web portal by January ’04 to serve IPCC data:  Iron out any remaining metadata details  Install new web-based registration system  Improve user experience to download the data  Shorten steps/time required to find the replicas  Allow more friendly hierarchical data browsing (may generate THREDDS catalogs from OGSA-DAI and RLS)  Possibly, develop more powerful query capabilities of ESG metadata database  Possibly, add visualization and analysis capabilities

14. ESP workshop, Sept 2003 ESG TOPOLOGY (Sept 2003) RLI MSS HRM HPSS HRM RLI HPSS HRM RLI DISK HRM RLI DISK OGSA-DAI MySQL RDBMS ESG WEB PORTAL Tomcat/Struts cross-update gridFTP query MyProxy authenticate GRAM GATEKEEPER submit execute gridFTP SERVER LAS SERVER visualize LBNL ISI LLNL NCAR ORNL CAS ANL LRC