Presentation is loading. Please wait.

Presentation is loading. Please wait.

NSF TeraGrid Review January 10, 2006

Published byPatricia Fisher Modified over 6 years ago

Similar presentations

Presentation on theme: "NSF TeraGrid Review January 10, 2006"— Presentation transcript:

1 NSF TeraGrid Review January 10, 2006 TeraGrid Information Services: Architecture, Current Capabilities, and Plans John-Paul “J-P” Navarro TeraGrid Grid Infrastructure Group “GIG” Area Co-Director for Software Integration and Information Services University of Chicago, Argonne National Laboratory SC07 November 14, 2007 Good afternoon, I’m JP Navarro I’m a part of the TeraGrid’s Grid Infrastructure Group where I focus on Software Integration and Information Services activities This talk will initroduce the TG’s IS: motivation, goals, architecture, current capabilities, and plans Charlie Catlett

2 What is the TeraGrid? 1 NSF funded facility with
NSF TeraGrid Review What is the TeraGrid? January 10, 2006 1 NSF funded facility with 11 resource providers and several other partners Grid Infrastructure Group (UChicago) UW UC/ANL PSC NCAR PU NCSA UNC/RENCI Caltech IU ORNL Tennessee USC/ISI SDSC LSU TACC Resource Provider (RP) Software Integration Partner Network Hub November 14, 2007 SC 07 Charlie Catlett

3 Grid Infrastructure Group (UChicago)
NSF TeraGrid Review What is the TeraGrid? January 10, 2006 Operating a coordinated high-performance compute, network, storage, and visualization infrastructure Grid Infrastructure Group (UChicago) UW UC/ANL PSC NCAR PU NCSA UNC/RENCI Caltech IU ORNL Tennessee USC/ISI SDSC LSU TACC Resource Provider (RP) Software Integration Partner Network Hub November 14, 2007 SC 07 Charlie Catlett

4 Open Scientific Discover
NSF TeraGrid Review What is the TeraGrid? January 10, 2006 To enable Open Scientific Discover Grid Infrastructure Group (UChicago) UW UC/ANL PSC NCAR PU NCSA UNC/RENCI Caltech IU ORNL Tennessee USC/ISI SDSC LSU TACC Resource Provider (RP) Software Integration Partner Network Hub November 14, 2007 SC 07 Charlie Catlett

5 (Tera)Grid in what sense?
NSF TeraGrid Review (Tera)Grid in what sense? January 10, 2006 Example Standard People Interfaces Request allocations & accounts Identify available resources Learn how to use resources Find resource status Ask for help Interact with the community Events Helpdesk What does the TeraGrid provide as a Grid that is different from what independent facilities would offer 1st were providing standard interfaces intended for PEOPLE POPS: single interface to request allocations on any or all resources “roaming” User Portal: standard user aware interface User Documentation: public documentation and information User Documentation & Knowledge Base Proposal System “POPS” User Portal November 14, 2007 SC 07 Charlie Catlett

6 (Tera)Grid in what sense?
NSF TeraGrid Review (Tera)Grid in what sense? January 10, 2006 Coordinated/Standard (User) Software Interactions Coordinated software Coordinated development & runtime Unix environment Standardized (Grid) service interfaces Grid Services Coordinated Software Remote Login Grid clients & tools Data Movement Development tools & languages Unix Environment Data Management What else does the TeraGrid provide as a Grid that is different from what independent facilities would offer The 2nd major thing we provide providing coordinated/standard software interactions Communication, Data, and Math tools & libraries Remote Execution Information November 14, 2007 SC 07 Charlie Catlett

7 Motivation for Information Services
NSF TeraGrid Review Motivation for Information Services January 10, 2006 Significant growth in size, diversity, and complexity of infrastructure Initially Today Resource Providers 4 11 Compute Resources 4 clusters 19 clusters, and SMPs, MPPs Resource Types compute, viz, storage plus instruments, gateways,.. Interconnects+MPI 3 types many types Processors Types 2 (IA-64, IA-32) 7 types (w/ non-Intel) Operating Systems Linux SLES8 ~12 Unix OSs Coordinated Software CTSS v1 CTSS v4 Our motivating vision major improvements to how TeraGrid Service Providers communicate information about their service offerings to the User Community November 14, 2007 SC 07 Charlie Catlett

8 How can users keep it all straight?
NSF TeraGrid Review Challenges January 10, 2006 Integrating many more types of infrastructure resources, Including more services providers, Providing more types of services, With more resources, providers, and services diversity, Supporting specialization. Our motivating vision major improvements to how TeraGrid Service Providers communicate information about their service offerings to the User Community How can users keep it all straight? November 14, 2007 SC 07 Charlie Catlett

9 Meeting the Challenge Re-architect CTSS
NSF TeraGrid Review Meeting the Challenge Re-architect CTSS January 10, 2006 Took Coordinated TeraGrid Software and Services “CTSS” v3: Monolithic coordinated capabilities In production since mid 2006 Re-architected into CTSS v4 “capability kits”: Small required core integration kit with minimum needed to integrate resources Plus ~10 optional user capability kits (at least 1 required) CTSS v3 (with exceptions) CTSS v4 TeraGrid Core Integration Remote Login Data Movement Data Management Our motivating vision major improvements to how TeraGrid Service Providers communicate information about their service offerings to the User Community Application Development & Runtime Science Workflow Remote Compute Wide Area GPFS Parallel Application (MPI) Visualization November 14, 2007 SC 07 Charlie Catlett

10 Meeting the Challenge Information Services
NSF TeraGrid Review Meeting the Challenge Information Services January 10, 2006 Information Services Vision: Create a coordinated way for TeraGrid participants to publish about the services they offer, Create a way for the TeraGrid to aggregate and index the information from TeraGrid participants, and to publish this information to the public in a form that can easily be used by user software, user interfaces, and TeraGrid service providers themselves to discover capabilities and how to access them Our motivating vision major improvements to how TeraGrid Service Providers communicate information about their service offerings to the User Community November 14, 2007 SC 07 Charlie Catlett

11 TG Information Services IS [NOT]
NSF TeraGrid Review TG Information Services IS [NOT] January 10, 2006 IS NOT IS A central database (Data Warehouse) A central index/aggregation (Google) A new user interface A way software (user interfaces) can access information A single implementation/tool Evolving set of software tools A single software interface Several useful interfaces (small set) A specific set of information Phased growing collection of information Changed data ownership Ownership maintained as appropriate Way to manage scientific information Way to manage Grid meta-data A data management system (database) An information publishing system When we launched this initiative there were a lot of different information types and repositories on the TeraGrid This slide explains how Information Services is positioned, and how it may be different from other information repositories iin a grid A coordinated way to index and publish public [Tera]Grid information thru software interfaces. November 14, 2007 SC 07 Charlie Catlett

12 High-Level Architecture
NSF TeraGrid Review High-Level Architecture January 10, 2006 TeraGrid Wide Information Services Apache 2.0 WS/REST HTTP GET Clients Cache Tomcat WebMDS TeraGrid Wide Respositories WS/SOAP Clients WS MDS4 Service Provider Information Services WS/SOAP WS MDS4 Clients Adapter Local Info November 14, 2007 SC 07 Charlie Catlett

13 Service Provider vs TG Wide Services
NSF TeraGrid Review Service Provider vs TG Wide Services January 10, 2006 Content: Locally owned and maintained information Originates anywhere the service provider wishes Services: 1 general purpose MDS service 2 scheduling MDS services: authenticated and public (merging) November 14, 2007 SC 07 Charlie Catlett

14 Service Provider vs TG Wide Services
NSF TeraGrid Review Service Provider vs TG Wide Services January 10, 2006 Content: Aggregate/index service provider information Additional central information (TGCDB, GIG operated services, …) Cached (service providers services can be down) Authenticated registrations Services: Several redundant servers (99.5% plus availability) Information caching (persistence) Several MDS4 services (WS/SOAP) WebMDS/Tomcat, Apache 2.0, … services (WS/REST) Content published in: HTML, XHTML/XML, XML, Atom, RSS, … November 14, 2007 SC 07 Charlie Catlett

15 Information Services Tooling
NSF TeraGrid Review Information Services Tooling January 10, 2006 WS/* (Tomcat 5.0, Apache 2.0) Benefits Very common web services platform Supports several web service interfaces (including simple) Supports multiple styles like REST, Web 2.0 Can be highly scalable Content Many formats: HTML, XHTML/XML, XML, RSS/Atom, … WebMDS (Globus 4.0.5/VDT 1.7.1) Live MDS4 content access XPath support XSLT transforms Many formats: HTML, XHTML/XML, XML, RSS/Atom WS/SOAP (Globus 4.0.5/VDT MDS4) Indexing, Trigger Registration, Publish, Subscribe Security/Authorization Robust WSRF interface XML November 14, 2007 SC 07 Charlie Catlett

16 High-Availability Design
NSF TeraGrid Review High-Availability Design January 10, 2006 Clients TG wide information services info.teragrid.org Service Provider Information Services info.dyn.teragrid.org TeraGrid Dynamic DNS Information services need to be available for live access by other Grid components, both TeraGrid services and user applications We believe reliability, availability, and scalability are essential Information Services administrators select servers Changes propagate globally with a 15 minute TTL … Doesn’t Change Dynamically Changes November 14, 2007 SC 07 Charlie Catlett

17 Information Services Users
NSF TeraGrid Review Information Services Users January 10, 2006 User Documentation User Portal Inca Testing Harness Gateways Peer Grids We our eating our own dogfood, publishing for internal consumption Users and developers like yourself are our also our target User Applications info.teragrid.org November 14, 2007 SC 07 Charlie Catlett

18 What’s in Production? Services Content (since when)
NSF TeraGrid Review What’s in Production? January 10, 2006 Services TeraGrid Resource Provider Information Services TeraGrid Wide Aggregating/Indexing Information Services Content (since when) Scheduling information for User Portal (Spring) Scheduler load, Queue contents (restricted) CTSS 4 capabilities kits (August) Which capability kits are available on each resource What software is available in each kit on each resource What services are available from each kit on each resource TeraGrid Central Database (tgcdb) keys and descriptions (October) November 14, 2007 SC 07 Charlie Catlett

19 Queue Contents in User Portal
NSF TeraGrid Review Queue Contents in User Portal January 10, 2006 Charlie Catlett

20 CTSS 4 Capability Kits For each capability kit on each resource
NSF TeraGrid Review CTSS 4 Capability Kits January 10, 2006 For each capability kit on each resource Current support level, and target support level Development, Testing, Production Support organization and contact Inca status URL Multiple version of a kit with different support levels November 14, 2007 SC 07 Charlie Catlett

21 CTSS 4 Capability Kit Software
NSF TeraGrid Review CTSS 4 Capability Kit Software January 10, 2006 For each kit software component on each resource Name, version, how to access it Multiple versions of a single component November 14, 2007 SC 07 Charlie Catlett

22 CTSS 4 Capability Kit Services
NSF TeraGrid Review CTSS 4 Capability Kit Services January 10, 2006 For each kit service on each resource Name, type, version, and Endpoint (contact location) GSI OpenSSH, GridFTP, SRB servers, PreWS & WS GRAM, MDS4 Multiple services of the same type November 14, 2007 SC 07 Charlie Catlett

23 CTSS Capability Kit Availability
NSF TeraGrid Review CTSS Capability Kit Availability January 10, 2006 Charlie Catlett

24 Where are the GridFTP services?
NSF TeraGrid Review Where are the GridFTP services? January 10, 2006 Charlie Catlett

25 What’s in Development? Expanded content Core Extension
NSF TeraGrid Review What’s in Development? January 10, 2006 Expanded content Local HPC Software Extended GridFTP service information (Meta)Scheduling support information Core Extension Information Services Metadata (registration w/o aggregation) Information Access tginfo, universal command line query tool WS/REST, Web 2.0 style information access Multiple formats: CSV TEXT, RSS/Atom, XML, … GLUE 2.0 Community publishing Community supported capabilities Community information services registration Coordinated software and services -> CTSS Local (uncoordinated) HPC software November 14, 2007 SC 07 Charlie Catlett

26 [Not so] Farfetched Possibilities
NSF TeraGrid Review [Not so] Farfetched Possibilities January 10, 2006 Gateways Publish or register to TeraGrid Wide Information Services Data collections Data collections register to TeraGrid Wide Information Services Access method, service Endpoint, paths Community software areas Which resources have each CSA What software is available in each CSA, how to access it Service Provider Planned and unplanned outage information Policies Peer grids/interoperability Resources, services available on peer grids (OSG, EGEE, …) ……. Software Integration is staffed to support expanding information services content November 14, 2007 SC 07 Charlie Catlett

27 More Information Information Services Documentation:
NSF TeraGrid Review More Information January 10, 2006 Information Services Documentation: (links to content and documentation) View Information Services Content: User Portal (scheduler load & queue contents): User Documentation (CTSS 4 kits, software, services): Information Service Main Page: November 14, 2007 SC 07 Charlie Catlett

Download ppt "NSF TeraGrid Review January 10, 2006"

Similar presentations

TeraGrid Community Software Areas (CSA) JP (John-Paul) Navarro TeraGrid Grid Infrastructure Group Software Integration University of Chicago and Argonne.

TeraGrid Community Software Areas (CSA) JP (John-Paul) Navarro TeraGrid Grid Infrastructure Group Software Integration University of Chicago and Argonne.
TeraGrid Deployment Test of Grid Software JP Navarro TeraGrid Software Integration University of Chicago OGF 21 October 19, 2007.

TeraGrid Deployment Test of Grid Software JP Navarro TeraGrid Software Integration University of Chicago OGF 21 October 19, 2007.
1 US activities and strategy :NSF Ron Perrott. 2 TeraGrid An instrument that delivers high-end IT resources/services –a computational facility – over.

1 US activities and strategy :NSF Ron Perrott. 2 TeraGrid An instrument that delivers high-end IT resources/services –a computational facility – over.
MTA SZTAKI Hungarian Academy of Sciences Grid Computing Course Porto, 22-24 January 2007. Introduction to Grid portals Gergely Sipos

MTA SZTAKI Hungarian Academy of Sciences Grid Computing Course Porto, January Introduction to Grid portals Gergely Sipos
Seminar Grid Computing ‘05 Hui Li Sep 19, 2005. Overview Brief Introduction Presentations Projects Remarks.

Seminar Grid Computing ‘05 Hui Li Sep 19, Overview Brief Introduction Presentations Projects Remarks.
Globus Toolkit 4 hands-on Gergely Sipos, Gábor Kecskeméti MTA SZTAKI

Globus Toolkit 4 hands-on Gergely Sipos, Gábor Kecskeméti MTA SZTAKI
Data Grids: Globus vs SRB. Maturity SRB  Older code base  Widely accepted across multiple communities  Core components are tightly integrated Globus.

Data Grids: Globus vs SRB. Maturity SRB  Older code base  Widely accepted across multiple communities  Core components are tightly integrated Globus.
Using Globus to Locate Services Case Study 1: A Distributed Information Service for TeraGrid John-Paul Navarro, Lee Liming.

Using Globus to Locate Services Case Study 1: A Distributed Information Service for TeraGrid John-Paul Navarro, Lee Liming.
Simo Niskala Teemu Pasanen

Simo Niskala Teemu Pasanen
Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.

Web-based Portal for Discovery, Retrieval and Visualization of Earth Science Datasets in Grid Environment Zhenping (Jane) Liu.
Kate Keahey Argonne National Laboratory University of Chicago Globus Toolkit® 4: from common Grid protocols to virtualization.

Kate Keahey Argonne National Laboratory University of Chicago Globus Toolkit® 4: from common Grid protocols to virtualization.
Network, Operations and Security Area Tony Rimovsky NOS Area Director

Network, Operations and Security Area Tony Rimovsky NOS Area Director
TeraGrid’s Integrated Information Service “IIS” Grid Computing Environments 2009 Lee Liming, JP Navarro, Eric Blau, Jason Brechin, Charlie Catlett, Maytal.

TeraGrid’s Integrated Information Service “IIS” Grid Computing Environments 2009 Lee Liming, JP Navarro, Eric Blau, Jason Brechin, Charlie Catlett, Maytal.
TeraGrid Gateway User Concept – Supporting Users V. E. Lynch, M. L. Chen, J. W. Cobb, J. A. Kohl, S. D. Miller, S. S. Vazhkudai Oak Ridge National Laboratory.

TeraGrid Gateway User Concept – Supporting Users V. E. Lynch, M. L. Chen, J. W. Cobb, J. A. Kohl, S. D. Miller, S. S. Vazhkudai Oak Ridge National Laboratory.
GIG Software Integration: Area Overview TeraGrid Annual Project Review April, 2008.

GIG Software Integration: Area Overview TeraGrid Annual Project Review April, 2008.
TeraGrid Information Services December 1, 2006 JP Navarro GIG Software Integration.

TeraGrid Information Services December 1, 2006 JP Navarro GIG Software Integration.
GIG Software Integration Project Plan, PY4-PY5 Lee Liming Mary McIlvain John-Paul Navarro.

GIG Software Integration Project Plan, PY4-PY5 Lee Liming Mary McIlvain John-Paul Navarro.
TeraGrid Information Services John-Paul “JP” Navarro TeraGrid Grid Infrastructure Group “GIG” Area Co-Director for Software Integration and Information.

TeraGrid Information Services John-Paul “JP” Navarro TeraGrid Grid Infrastructure Group “GIG” Area Co-Director for Software Integration and Information.
TeraGrid Information Services JP Navarro, Lee Liming University of Chicago TeraGrid Architecture Meeting September 20, 2007.

TeraGrid Information Services JP Navarro, Lee Liming University of Chicago TeraGrid Architecture Meeting September 20, 2007.
Advancing Scientific Discovery through TeraGrid Scott Lathrop TeraGrid Director of Education, Outreach and Training University of Chicago and Argonne National.

Advancing Scientific Discovery through TeraGrid Scott Lathrop TeraGrid Director of Education, Outreach and Training University of Chicago and Argonne National.

Similar presentations

Ads by Google