1. JIM BASNEY 1, STUART MARTIN 2, JP NAVARRO 2, MARLON PIERCE 3, TOM SCAVO 1, LEIF STRAND 4, TOM URAM 2,5, NANCY WILKINS-DIEHR 6, WENJUN WU 2, CHOONHAN YOUN 6 1 NATIONAL CENTER FOR SUPERCOMPUTING APPLICATIONS, UNIVERSITY OF ILLINOIS AT URBANA-CHAMPAIGN 2 ARGONNE NATIONAL LABORATORY 3 INDIANA UNIVERSITY 4 CALIFORNIA INSTITUTE OF TECHNOLOGY 5 UNIVERSITY OF CHICAGO 6 SAN DIEGO SUPERCOMPUTER CENTER, UNIVERSITY OF CALIFORNIA AT SAN DIEGO The Problem Solving Environments of TeraGrid, Science Gateways, and the Intersection of the Two

2. TeraGrid, what is it? A unique combination of fundamental CI components Navajo Technical College, September 25, 2008

3. Gateways, what are they? Problem Solving Environments for Science Portal or client-server interfaces to high end resources  Web developments, explosion of digital data lead to the increased importance of the internet and the web for science  Only 16 years since the availability of web browsers Developments in web technology From static html to cgi forms to the wikis and social web pages of today Full impact on science yet to be felt  Web usage model resonates with scientists But, need persistency if the Web is to have a profound impact on science (this is key for all PSEs) TeraGrid provides common infrastructure for gateway developers Navajo Technical College, September 25, 2008

4. TeraGrid’s Infrastructure for Gateways Problem  Local compute resources are typically not enough for Gateways Goal  Make it easy to use any TeraGrid site from a Gateway Approach  Provide a set of client APIs and command line tools for use in Gateways/portals  Maintain and deploy a set of common services on each site  Maintain and deploy some central services

5. Infrastructure Capabilities Information Discovery  Find deployed services  Get details about the compute resources Data Management  Move data to and from compute resources Execution Management  Submit and monitor remote computational jobs Security  Make sure secure access is in place with all services and tools

6. Security Based on Grid Security Infrastructure (GSI)  Uses X509 PKI  End entity certificates (e.g. issued to a person or host)  User proxy certificates (valid for a limited period of time) Enables single sign-on to all TG resources Enables delegation  Users/clients can disconnect and let services perform actions securely on their behalf Integrated in grid middleware services  User Portal, MyProxy, GSISSH, GridFTP, GRAM, MDS, RFT, etc

7. GT4 Server Globus Web Service Java WS Container Gridmap GSI in Action GT4 Client Globus WS Client grid-proxy- init end entity credential Key proxy credential Key X.509 proxy certificate

8. Single Sign-On

9. Gateway Workflow with GSISSH GSISSH Service Scheduler (e.g., PBS) Compute Nodes GSISSH Service Scheduler (e.g., LSF) Compute Nodes Local Jobs Resource AResource B gateway Jobs GSISSH PBSLSF Client does: myproxy-logon (once) Move files with gsiscp Submit job with gsissh and lrm commands

10. Remote Execution Management Grid Resource Allocation and Management (GRAM) Provide an abstraction layer on top of various local resource managers (PBS, Condor, LSF, SGE, …)  Defines a common job description language  Client API and command line tools to asynchronously access remote LRMs  Fault tolerant  GSI Security  “job” Workflow  File staging before and after job execution  Lastly, File cleanup  File staging requires delegation

11. Traditional LRM Interaction Local Jobs Resource A Scheduler (e.g., PBS) Compute Nodes Satisfies many users and use cases TACC’s Ranger (62976 cores!) is the Costco of HTC ;-), one stop shopping, why do we need more?

12. Local Jobs Resource A GRAM4 Service Scheduler (e.g., PBS) Compute Nodes remote GRAM4 Jobs gramJob API Adds remote execution capability  Enable clients/devices to manage jobs from off of the cluster (Gateways!) GRAM Benefit

13. GRAM4 Service Scheduler (e.g., PBS) Compute Nodes GRAM4 Service Scheduler (e.g., LSF) Compute Nodes Local Jobs Resource AResource B GRAM4 Jobs gramJob API Provides scheduler abstraction

14. Gateway Perspective GRAM4 jobs Scalable job management Interoperability GRAM4 Sched Compute Nodes gramJob API GRAM4 Sched Compute Nodes GRAM4 Sched Compute Nodes GRAM4 Sched Compute Nodes GRAM4 Sched Compute Nodes GRAM4 Sched Compute Nodes GRAM4 Sched Compute Nodes GRAM4 Sched Compute Nodes GRAM4 Sched Compute Nodes GRAM4 Sched Compute Nodes GRAM4 Sched Compute Nodes GRAM4 Sched Compute Nodes GRAM4 Sched Compute Nodes GRAM4 Sched Compute Nodes GRAM4 Sched Compute Nodes GRAM4 Sched Compute Nodes GRAM4 Sched Compute Nodes GRAM4 Sched Compute Nodes

15. Data Management - GridFTP GridFTP  High-performance, secure, reliable data transfer protocol optimized for high-bandwidth wide-area  GSI Security  Third-party transfers  Parallel Transfers  Striping  Lots of small files (LOSF)  Can outperform other file transfer methods like scp  Limited in that it does not queue and throttle requests  Needs a reliable higher-level service, hence RFT

16. Data Management - RFT Reliable File Transfer  Adds reliability on top of GridFTP  GSI Security  Throttles requests  Retries non-fatal transfer errors  Resumes transfers from the last known position  Requires delegation in order to contact GridFTP servers on user’s behalf

17. Web Authn Resource ProviderScience Gateway WS GRAM Client WS GRAM Service proxy credential proxy certificate Key Java WS Container Webapp Web Interface Web Browser community credential Key community account Science Gateway with Community Credential

18. GT4 ServerGT4 Client Globus WS Client GridShib SAML PIP proxy certificate GridShib SAML Tools end entity credential Key SAML Globus Web Service PolicyLogs Java WS Container (with GridShib for GT) Security Context proxy credential SAML Key GridShib-enabled GSI

19. Web Authn Resource ProviderScience Gateway WS GRAM Client GridShib SAML PIP proxy certificate GridShib SAML Tools community credential Key SAML WS GRAM Service PolicyLogs Java WS Container (with GridShib for GT) Security Context Webapp attributes Web Interface Web Browser username proxy credential SAML Key GridShib-enabled Science Gateway

20. Information Management TeraGrid’s Integrated Information Services are a network of web services responsible for aggregating the availability of TeraGrid capability kits, software, and services across all the infrastructure providers  Where are the job submission, file-transfer, and login services needed by Gateways?  What is the queue status and estimated delay for each resource?  What are the available testbeds (non-production / experimental software)?  What are the Gateways (problem solving environments) available to users?

21. Clients High-Level Components Cache WS/REST HTTP GET WS/SOAP WS MDS4 Tomcat WebMDS Apache 2.0 TeraGrid Wide Information Services WS/SOAP WS MDS4 Service Provider Information Services TeraGrid Wide Information TeraGrid Wide Information Service Provider Information Service Provider Information

22. High-Availability Design … info.dyn.teragrid.org info.teragrid.org TeraGrid Dynamic DNS Server failover propagates globally in 15 minutes Clients Dynamic paths Static paths Service Provider Information Services TeraGrid Wide Information Services

23. Today, there are approximately 29 gateways using the TeraGrid NSF Program Officers, September 10, 2008

24. Selected Highlights from the PSE08 paper The Social Informatics Data (SID) Grid The Geosciences Network (GEON) QuakeSim Computational Infrastructure for Geodynamics (CIG) Conclusions

25. Social Informatics Data Grid Heavy use of “multimodal” data.  Subject might be viewing a video, while a researcher collects heart rate and eye movement data. Events must be synchronized for analysis, large datasets result Extensive analysis capabilities are not something that each researcher should have to create for themselves. NSF Program Officers, September 10, 2008 http://www.ci.uchicago.edu/res earch/files/sidgrid.mov

26. How does SIDGrid use the TeraGrid? Computationally intensive tasks  Speech, gesture, facial expression, and physiological measurements  Media transcoding for pitch analysis of audio tracks  Once stored in raw form, data streams converted to formats compatible with software for annotation, coding, integration, analysis  fMRI image analysis Workflows for massive job submissions and data transfers using Virtual Data System (VDS) Worflows converted to concrete execution plan via Pegasus Grid planner  TeraGrid information service (MDS)  Replica location service (RLS)  DAGMAN and Condor-G/GRAM

27. The goal of GEON is  to advance the field of geoinformatics and  to prepare and train current and future generations of geoscience researchers, educators, and practitioners in the use of cyberinfrastructure to further their research, education, and professional goals. GEON is providing several key features  data access, computational simulations, personal work spaces and analyses environments  identifying best practices with the objective of dramatically advancing geoscience research and education.

28. How does GEON use the TeraGrid? Computationally intensive tasks  Ability to speedily construct earth models, access observed earthquake recordings and simulate them to understand the subsurface structure and characteristics of seismic wave propagation in an efficient manner  SYNSEIS (SYNthetic SEISmogram generation tool), provides access to seismic waveform data and simulate seismic records using 2D and 3D models.  Conduct advanced calculations for simulating seismic waveforms of either earthquakes or explosions at regional distances (< 1000 km). GSI (security), GAMA (account management), GridFTP (data transfer), GRAM (job submission), MyWorkspace (job monitoring) Account management for classroom use, MyProjects collaboration tool and tagging also serve students

29. QuakeSim - Some Design Choices Build portals out of portlets (Java Standard) Reuse capabilities from our Open Grid Computing Environments (OGCE) project, the REASoN GPS Explorer project, and many TeraGrid Science Gateways. Decorate with Google Maps, Yahoo UI gadgets, etc. Use Java Server Faces to build individual component portlets. Build standalone tools, then convert to portlets at the very end. Use simple Web Services for accessing codes and data. Keep It Stateless … Use Condor-G and Globus job and file management services for interacting with high performance computers. TeraGrid Favor Google Maps and Google Earth for their simplicity, interactivity and open APIs. Generate KML and GeoRSS Use Apache Maven based build and compile system, SVN on SourceForge

30. Portlets + Client Stubs DB Service JDBC DB Job Sub/Mon And File Services Job Sub/Mon And File Services Operating and Queuing Systems Operating and Queuing Systems WSDL Browser Interface WSDL Visualization Or Map Service Visualization Or Map Service DB WSDL Host 1 (Quaketables) Host 2 (Grid)Host 3 (G Maps) SOAP/HTTP HTTP(S)

31. Two Approaches to the Middle Tier Grid Service Backend Resource Backend Resource Web Service Portal Comp. Grid Client Backend Resource Backend Resource Fat ClientThin Client Grid Protocol (SOAP) Grid Client HTTP + SOAP Grid Protocol (SOAP)

32. Daily RDAHMM Updates Daily analysis and event classification of GPS data from REASoN’s GRWS. Daily analysis and event classification of GPS data from REASoN’s GRWS.

33. Disloc output converted to KML and plotted. Disloc output converted to KML and plotted.

34. GeoFEST Finite Element Modeling portlet and plotting tools GeoFEST Finite Element Modeling portlet and plotting tools

35. “SWARM: Scheduling Large-scale Jobs over the Loosely-Coupled HPC Clusters,” S. L. Pallickara and M. E. Pierce, Friday, December 12, 2 p.m. to 2:30 p.m. http://escience2008.iu.edu/sessions/SWARM.shtml Standard Web Service Interface Request Manager Resource Ranking Manager DataModel Manager QBET Web Service Fault Manager Job Execution Manager Condor G with Birdbath User A’s Job Board User A’s Resource Pool Tokens for resource X,Y,Z User A’s Job Queue Job Distributor RDMBS MyProxy Server MyProxy Server High Performance Computing Clusters: Grid style clusters and condor computing nodes Desktop Users, Web Portal and Gateway style application Hosted by TeraGrid Project Hosted by UCSB

36. Membership-governed organization  40 institutional member, 9 foreign affiliates Supports and promotes Earth science by developing and maintaining software for computational geophysics NSF Program Officers, September 10, 2008

37. How does CIG use the TeraGrid? Seismograms allow scientists to understand the ground motion Computationally-intensive simulations run on TeraGrid using an assortment of 3D and 1D earth models produce synthetic seismograms  Necessary input datasets provided via the portal  Daemon (Python, Pyre) constantly polls the web site looking for work to do  GSI-OpenSSH and MyProxy credentials to submit jobs, monitors jobs, transfers output back to portal  status updates to the web site using HTTP POST  Users can download results in ASCII and Seismic Analysis Code (SAC) format  Visualizations include "beachball" graphics depicting the earthquake's source mechanism, and maps showing the locations of the earthquake and the seismic stations using GMT (http://gmt.soest.hawaii.edu/)http://gmt.soest.hawaii.edu/ Researchers quickly receive results and can concentrate on the scientific aspects of the output rather than on the details of running the analysis on a supercomputer Future Directions  Parameter explorations  Custom earth models for users

38. Conclusions Technical requirements of some PSEs dictate seamless access to high-end compute and data resources  A robust, flexible and scalable infrastructure can provide a foundation for many PSEs PSEs themselves must be treated as sustainable infrastructure  Researchers will not truly rely on PSEs for their work unless they have confidence that the PSE will remain operational for the long term and provide reliable services