Web Portal Design Workshop, Boulder (CO), Jan 2003 Luca Cinquini (NCAR, ESG) The ESG and NCAR Web Portals Luca Cinquini NCAR, ESG Outline: 1.ESG Data Services Strategy 2.ESG Prototype Web Portal demo 3.NCAR Community Data Portal 4.NCAR Prototype Search & Discovery engine 5.Final comments

Web Portal Design Workshop, Boulder (CO), Jan 2003 Luca Cinquini (NCAR, ESG) ORNL: Climate storage & computational resources ORNL: Climate storage & computational resources LANL: Next generation coupled models & computing LANL: Next generation coupled models & computing ANL: Computational grids, & grid-based applications ANL: Computational grids, & grid-based applications USC/ISI: Computational grids, & grid-based applications USC/ISI: Computational grids, & grid-based applications NCAR: Climate change predication and scenarios NCAR: Climate change predication and scenarios LBNL: Climate storage facility LBNL: Climate storage facility LLNL: Model diagnostics & inter-comparison LLNL: Model diagnostics & inter-comparison The Earth System Grid

Web Portal Design Workshop, Boulder (CO), Jan 2003 Luca Cinquini (NCAR, ESG) The Earth System Grid “Synergistic collaboration” among several US national labs and research centers (ANL, ISI, LBNL, LLNL, NCAR, ORNL) involved in atmospheric science and scientific computing 3 year project funded by the DOE Scientific Discovery through Advanced Computing (SciDAC) Goal: build the next generation computational and data management environment for the geosciences: “a system of geographically distributed data and computational centers which will allow seamless access for earth scientists to data repositories, analysis tools and computational resources” Strategy: application and extension of Grid technologies (and other state of the art IT) to the geosciences Initial focus on next generation climate modeled data (PCM/CCSM)

Web Portal Design Workshop, Boulder (CO), Jan 2003 Luca Cinquini (NCAR, ESG) ESG Goal Functionality Security (authentication/autherization mechanisms, management of security credentials) High speed, reliable file transfer between data centers (both to/from local and remote storage) Management and use of scientific metadata (generation, storage, update, annotation, query) Data search and discovery through metadata Virtual data management (aggregation/subsetting of physical data files, transfer and access of data containers) Data analysis and visualization Resources monitoring and discovery Activity/usage logging and monitoring

Web Portal Design Workshop, Boulder (CO), Jan 2003 Luca Cinquini (NCAR, ESG) ESG Technology Strategy Use a core library of state of the art, powerful technologies (Globus, Java, XML, servlets, web services, Python) Accomplish desired functionality within modular, extensible software components (or services) Integrate components in powerful, easy to use, end to end applications Provide both descrete (web/grid services) and centralized (web portal) access to the applications

Web Portal Design Workshop, Boulder (CO), Jan 2003 Luca Cinquini (NCAR, ESG) Core software library (Globus, Java packages, CDAT, NCL, etc.) Core software library (Globus, Java packages, CDAT, NCL, etc.) ESG Software Architecture Transport Security Metadata Analysis Visualization Monitoring ESG Data Services Web Portal Servlets/JSP Web Portal Servlets/JSP Web/Grid Services RMI servers Web client Web service client RMI client

Web Portal Design Workshop, Boulder (CO), Jan 2003 Luca Cinquini (NCAR, ESG) ESG Super Computing 2002 Demo Prototype of a unified web portal that integrates powerful, different technologies in a unified flow that constitutes and example of a scientific activity session Data Search & Discovery (RLS, MCS) Data Search & Discovery (RLS, MCS) Authentication (GSI) Authentication (GSI) Data Selection Data Transfer (HRM) Data Transfer (HRM) Data Visualization (LAS) Data Visualization (LAS) Data Subsetting Authorization (CAS) Data Transfer (openDAPg)

Web Portal Design Workshop, Boulder (CO), Jan 2003 Luca Cinquini (NCAR, ESG) ESG SC 2002 DEMO

Web Portal Design Workshop, Boulder (CO), Jan 2003 Luca Cinquini (NCAR, ESG) TOMCAT Servlet engine TOMCAT Servlet engine MCS Metadata Cataloguing Services MCS Metadata Cataloguing Services RLS Replica Location Services RLS Replica Location Services SOAP RMI MyProxy server MyProxy server MCS client RLS client MyProxy client GRAM gatekeeper GRAM gatekeeper CAS Community Authorization Services CAS Community Authorization Services CAS client disk MSS Mass Storage System HPSS High Performance Storage System disk HPSS High Performance Storage System disk SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management SRM Storage Resource Management gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server gridFTP server openDAPg server openDAPg server CAS-enabled Striped-gridFTP server CAS-enabled Striped-gridFTP server LBNL LLNL ISI NCAR ORNL ANL The Earth System Grid Striped gridFTP client Striped gridFTP client gridFTP openDAPg server openDAPg server CAS-enabled Striped-gridFTP server CAS-enabled Striped-gridFTP server gridFTP openDAPg server openDAPg server CAS-enabled Striped-gridFTP server CAS-enabled Striped-gridFTP server gridFTP LAS Live Access Server LAS Live Access Server

Web Portal Design Workshop, Boulder (CO), Jan 2003 Luca Cinquini (NCAR, ESG) The NCAR Community Data Portal Internal NCAR project (SCD, ATD, CGD, Unidata, JOSS, HAO, DLESE) aimed at providing a central web portal for all of UCAR/NCAR data assets Data scope much larger than ESG: topic (model data, experimental, measurement, satellite), format (netCDF, HDF, Grib, etc.), amount (Pb), requirements (for analysis, access, metadata, etc.), user community But goal is at least initially narrower: initial work has focused on:  Data Visualization  LAS, DODS servers connected to Ferret, NCL or CDAT engines allow visualization of selected datasets  Metadata  Experimenting with THREDDS schema for data cataloguing  Developing Scientific Metadata Services: a system of software components that would allow management and use of metadata (generation, storage, update, query, browsing)  Initial prototype: “Google”-like search and discovery of data

THREDDS CATALOG THREDDS CATALOG Dataset metadata (DC) Dataset metadata (NcML) Dataset metadata (other) XML Native Database (Apache Xindice) XML Native Database (Apache Xindice) Collection: /thredds (name=ID) Collection: /dc (name=ID) Collection: /nc (name=ID) Collection: /default (name=ID) HARVESTING WEB SERVICE Relational Database (Mysql) Relational Database (Mysql) Table: DATASET (ID, key) Table: TEXT ( key, name, value, ns, qualifiers) Table: FLOAT ( key, name, value, ns, qualifiers) Table: DATETIME ( key, name, value, ns, qualifiers) SCIENTIFIC METADATA SERVICES Query Architecture EXTRACTOR APPLICATION METADATA QUERY SERVICE Generic Search & Discovery Query (SQL) Generic Search & Discovery Query (SQL) Text query Numeric query (ex: geograpchic) Temporal query Specialized Query (Xpath, Xquery) Specialized Query (Xpath, Xquery) List of Dataset IDs List of Dataset IDs XML document XML document DC XSL NC XSL THREDDS XSL GENERIC XSL

Web Portal Design Workshop, Boulder (CO), Jan 2003 Luca Cinquini (NCAR, ESG) CDP SEARCH AND DISCOVERY DEMO

Web Portal Design Workshop, Boulder (CO), Jan 2003 Luca Cinquini (NCAR, ESG) A model for a collaborative architecture Use MVC software design model for web development:  Model: core software libraries/packages where the business logic takes place (e.g., HRM, CDAT, NCL, openDAP).  View: user interface, how data is presented to the user  Controller: mediates interaction between Model and View, i.e. parses input request from View, invokes the corresponding methods on the Model, returns output to the View Additionally, integrate MVC applications by sharing data beans associated with the user session

Web Portal Design Workshop, Boulder (CO), Jan 2003 Luca Cinquini (NCAR, ESG) RLS server RLS API RLS web client RLS web UI HRM server HRM API HRM web client HRM web UI data (Java beans) MODEL (any language, Java API) CONTROLLER (Java servlets, JSP taglibs, Java classes) VIEW (JSP) Integrated MVC architecture example