State-Wide Collaborative Grid Computing Course Barry Wilkinson Department of Computer Science University of North Carolina at Charlotte and Mark Holliday Department of Mathematics and Computer Science Western Carolina University Teaching and Learning with Technology Conference March 30th – April 1st, 2005, Raleigh, NC. Participating Sites

Talk Outline  Brief introduction to Grid Computing.  Software Tools.  Grid Computing Course.

Grid Computing Using geographically distributed interconnected computers for high performance computing and/or for resource sharing. The grid virtualizes heterogeneous geographically disperse resources From "Introduction to Grid Computing with Globus," IBM Redbooks

Virtual Organizations Grid computing offers potential of virtual organizations: – groups of people, both geographically and organizationally distributed, working together on a problem, sharing computers AND other resources such as databases and experimental equipment.  Crosses multiple administrative domains.

Shared Resources Can share much more than just computers:  Storage  Sensors for experiments at particular sites  Application Software  Databases  Network capacity, …

Interconnections and Protocols Focus now on:  using standard Internet protocols and technology, i.e. HTTP, SOAP, web services, etc.,  Grid computing standards around web services.

Applications  Originally e-Science applications – Computational intensive Not necessarily one big problem but a problem that has to be solved repeatedly with different parameters. – Data intensive. – Experimental collaborative projects  Now also e-Business applications to improve business models and practices.

History  Began in mid 1990’s with experiments using computers at geographically dispersed sites.  Seminal experiment – “I-way” experiment at 1995 Supercomputing conference (SC’95), using 17 sites across the US running: – 60+ applications. – Existing networks (10 networks).

Grid Networks and Projects  Numerous very high performance computing projects developed in late 1990’s and 2000’s.  Examples: USA TeraGrid, UK e-Science Grid, and many others.

TeraGrid

UK e-Science Grid

Software - Globus Project  Open source software toolkit developed for grid computing.  Roots in I-way experiment.  Work started in  Four versions developed to present time.  Reference implementations of grid computing standards.  Defacto standard for grid computing.

 GSI (Grid Security Infrastructure) – Grid security.  MDS (Monitoring and discovery Service) – Interface to system and service information.  GRAM (Grid Resource Allocation Manager) – Remote job submission and control.  GridFTP – Secure data transfer. Globus Key Components

From “Globus Toolkit 4 Tutorial,” MCNC Jan-Feb, 2005, Pawel Plaszczak and Bogdan Lobodzinski, Gridwise Technologies. 2. discover resource 3. submit job 4. transfer data 1.secure environment GSI GRAM MDS GridFTP

Globus Toolkit: Recent History  GT2 (2.4 released in 2002) – GRAM, MDS, GridFTP, GSI.  GT3 (3.2 released mid-2004): redesign – OGSA (Open Grid Service Architecture) - OGSI (Open Grid Services Infrastructure) based. – Introduced “Grid services” as an extension of web services. – OGSI now abandoned.  GT4 (release for April 2005): redesign – WSRF (Web service Resource Framework) based. – Grid standards merged with Web services.

Supercomputing 2003 Demonstration  We used Globus version 2.4 in a Supercomputing 2003 demo organized by the University of Melbourne.  21 countries involved, numerous sites.

A re-implementation based upon the Open Grid Service Architecture (OGSA) standard.  We used version 3.2 for the Fall 2004 course.  Underlying implementation of version 3.x used OGSI Open Grid Service Infrastructure), which was not embraced by the community. Version 3

Version 4  Currently under development to be released early  OGSA kept but OGSI abandoned in favor of new implementation standards based around web services. (Version 3 used “extended” web services)

Web Services-Based Grid Computing  Grid Computing is now strongly based upon web services.  Large number of newly proposed grid computing standards: – WS-Resource Framework – WS-Addressing – etc., etc. …..

Grid Computing Course (Fall 2004)  Originated from WCU on NCREN network. Broadcast to: – UNC-Wilmington – NC State University – UNC-Asheville – UNC-Greensboro – Appalachian State University – NC Central University – Cape Fear Community College – Elon University – UNC-Charlotte*  Instructors: – Barry Wilkinson and Clayton Ferner (UNC-Wilmington)  Several faculty helped at various sites  43 students (*canceled)

Participating Sites

Level and Prerequisites  Listed as an undergraduate course but can be taken for graduate credit. (Graduate students to do more work.)  Preferably have programming skills in Java on a Linux system.

Topics  Review of Internet technologies  Introduction to grid computing  Web services  Grid services (OGSA)  Security, Public Key Infrastructure  Globus 3.2  Condor-G  MPI and grid-enabled MPI  UNC-W workflow editor and other GUI tools  Grid computing applications 5 grid computing assignments. Software from NMI package.

Grid computingVirtual organizations, grid projects in the US and around the world, grid challenges. Internet IP addresses, HTTP, URL, HTTP, Technologies XML, Telnet, FTP, SSL. Web ServicesService-Oriented Architecture, service registry, XML documents, XML schema, namespaces, SOAP, XML/SOAP examples, WSDL, WSDL to/from code. Assignment 1 “Simple” Web service Java programming assignment. Tomcat environment, axis, JWS facility. Weeks 1 - 3

Weeks Grid ServiceConcepts, differences to Web services, stateful/ stateless/transient/non-transient, Open Grid Services Architecture (OGSA), OGSI, grid service factory, Web Services Resource Framework (WSRF) Assignment 2"Simple" grid service Java programming assignment. Globus 3.2 environment.Tools: ant.

Weeks SecuritySecure connection, authorization requirements, symmetric and asymmetric key cyptography, non- repudiation, digital signatures, X 509 certificates, certificate authorities. Globus: Basic structure, grid service container, service browser, Resource Allocation Manager, job submission with managed-job-globusrun, Grid Security Infrastructure, Globus certificates, simpleCA, proxies, creating a proxy, Assignment 3Submitting a Job to the Grid, GT3 mangaged-job-globusrun, job specified in RSL-2 (XML file).

Weeks ResourceMaster Managed Job Factory ManagementService, Resource Specification Language (RSL and RSL-2). Information Directory Services, LDAP, resource discovery. Schedulers andCondor, submit description resource brokers file, DAGMan, checkpointing, ClassAd, Condor-G. Assignment 4Submitting a Condor-G Job

Weeks High performance Grand challenge problems, computing (HPF)parallel computing, potential speed- up, types of parallel computers. Parallel Techniques suitable for a Grid, Programmingembarrassingly parallel computations, Monte Carlo, parameter studies, sample "big" problems, gravitational N-body problem Cluster ComputingBasic message passing techniques, programming models, synchronous message passing, asynchronous message passing, message tags, collective routines.

Weeks MPIProcess creation, communicators, unsafe message passing, MPI routines, running MPI program on a cluster. Grid-enabled MPIMPI-G2 internals, mpirun command, RSL script Assignment 5Running a simple MPI-G2 program (not done Fall 2004).

Grid Portal Design  “A web-based application server enhanced with the necessary software to communicate to grid services and resources”  “Provides application scientist a customized view of software and hardware resources from a web browser” Weeks 10 to 11

OGCEGrid

Workflow Technique  Functional decomposition - dividing problem into separate functions that take results from other functional units and pass on results onto functional units.  Interconnection patterns depends upon the problem.  Workflow - describes the flow of information between the units. Weeks 11 to 12

Workflow Example - Climate Modeling

GridNexus Workflow Editor  Developed by UNC-Wilmington.  Introduced in course.  Assignment set.

GridNexus Workflow using Grid Services

Guest Speakers  Professor Daniel A. Reed, Chancellor's Eminent Professor, Vice Chancellor for IT and CIO, UNC-Chapel Hill, Director of Institute for Renaissance Computing, University of North Carolina at Chapel Hill, Duke University, and NC State University: – “Grid computing: 21st Century Challenges.”  Dr. Wolfgang Gentzsch, Managing Director, MCNC Grid Computing and Networking Services: – “Grid Computing in the Industry”  Chuck Kesler, Director, Grid Deployment and Data Center Services, MCNC: – “Security Policy, Legal, and Regulatory Challenges in Grid Computing Environments”  Professor Ian Foster: Ian Foster, Argonne National Laboratory and University of Chicago: – “The Grid: Beyond the Hype” Taped presentation (originally given at Duke University, Sept. 14th, 2004). Weeks 14 to 15

Course Home Page WebCT also used for quizzes.

Fall 2005 Grid Computing Course  To originate from UNC-Charlotte.  Again in collaboration with UNC-Wilmington.  Can be for undergraduate or graduate credit.

Acknowledgements Partial support for this work was provided by the National Science Foundation and University of North Carolina, Office of the President. 1.National Science Foundation, “Introducing Grid Computing into the Undergraduate Curricula,” ref. DUE , PI: A. B. Wilkinson, $100,000, University of North Carolina Office of President, “A Consortium to Promote Computational Science and High Performance Computing,” PI: B. Kurtz (Appalachian State University) co-PI B. Wilkinson and others at various universities, total $650,000, University of North Carolina Office of President, “Fostering Undergraduate Research Partnerships through a Graphical User Environment for the North Carolina Computing Grid,” PI: R. Vetter (UNC-Wilmington), co-PI B. Wilkinson and others at various universities, total $557,634,

Papers Since Fall 2004 B. Wilkinson, M. Holliday, and C. Ferner, “Experiences in Teaching a Geographically Distributed Undergraduate Grid Computing Course,” Workshop on Collaborative and Learning Applications of Grid Technology and Grid Education, IEEE International Symposium on Cluster Computing and the Grid (CCGrid2005), Cardiff, UK, May , 2005, accepted. M. A. Holliday, B. Wilkinson, J. House, S. Daoud, and C. Ferner, “A Geographically-Distributed, Assignment- Structured Undergraduate Grid Computing Course,” SIGCSE 2005 Technical Symposium on Computer Science Education, St. Louis, Missouri, February , 2005.

Questions?