Large Scale Distributed Computing B. Ramamurthy

Evolution of Internet Computing scale Parallel HPC Semantic discovery ?????? Automate (discovery) Discover (intelligence) Transact Integrate Interact Inform Publish time B. Ramamurthy 2/22/2019

Computing Models High performance computing; Ex: grid computing Parallel computing Cloud computing Internet scale computing B. Ramamurthy 2/22/2019

Beginnings of The Grid Beginnings of the grid in Search for Extra Terrestrial Intelligence (seti@home project) http://planetary.org/html/UPDATES/seti/index.html The Wow signal: http://www.planetary.org/explore/topics/seti/seti_history_10.html B. Ramamurthy 2/22/2019

IT Issues Modernizing legacy systems Web-enabling business services Automating business processes Integration within organization Data and application integration Collaboration with outside organizations Security, accountability, availability, reliability, … B. Ramamurthy 2/22/2019

Grid Organizations Global Grid Forum (GGF): The Globus Alliance: www.globalgridforum.org a community-initiated forum of thousands of individuals from industry and research leading the global standardization effort for grid computing.  The Globus Alliance: www.globus.org conducts research and development to create fundamental technologies behind the "Grid," which lets people share computing power, databases, and other on-line tools securely across corporate, institutional, and geographic boundaries without sacrificing local autonomy.  Enterprise Grid Alliance (EGA): http://www.gridalliance.org/en/index.asp EGA working groups have been proposed to address: (1) Reference model; (2) Component provisioning; (3) Data provisioning; (4) Utility accounting (5) Grid security. GGF + EGA  OGF Open Science Grid (OSG): ? B. Ramamurthy 2/22/2019

Grid Technology Enabling technology. Natural evolution of distributed systems and the Internet. Middleware supporting network of systems to facilitate sharing, standardization and openness. Infrastructure and application model dealing with sharing of compute cycles, data, storage and other resources. Promoted by NSF through its Network Middleware Initiative (NMI). Publicized by prominent industries as on-demand computing, utility computing, etc. Move towards delivering “computing” to masses similar to other utilities (electricity and voice communication). B. Ramamurthy 2/22/2019

Types of Grid Batch-oriented High performance applications High Throughput applications Example: Condor Service-Oriented View all the resources and functions as services. Build application models around services. Anatomy of the grid Physiology of the grid It is this genre of grid that will move the grid technology towards business applications. Example: Globus B. Ramamurthy 2/22/2019

Service-oriented Standards Open Grid Services Architecture (OGSA) Open Grid Services Infrastructure (OGSI) Globus Toolkit (GT3) is a reference implementation In GT4, OGSI has been replaced by WSRF (Web services resource framework) B. Ramamurthy 2/22/2019

OGSA, OGSI and WS From tutorial: Satomayor’s GT3 Tutorial http://www.casa-sotomayor.net/gt3-tutorial-working/ B. Ramamurthy 2/22/2019

Features of Grid Service Logger object; Levels of logging: Info, .. Warn, Error, Fatal Filtering and redirecting to file, console Provides Standard security Provides notification of events … Stores service properties and States; for discovery, monitoring, negotiations, etc. Services with ability to migrate Permanent services such as naming service that get activated and terminated with the container B. Ramamurthy 2/22/2019

Current Status Information/ Application Servers Clients/Consumers Internet Information/ Application Servers Clients/Consumers Internet Application Application B. Ramamurthy 2/22/2019

Where are we heading? Web-enabling information Information/ Application Servers Clients/Consumers Web-enabling information Web-enabling applications/forms HTML Business to Consumers (B2C) Business to Business (B2B) Application to application Web Services, XML Standards for specifying operation in SOAP (Simple Object Access Protocol) B. Ramamurthy 2/22/2019

Financial: Build Portfolio Environment: Plan Forestation Beyond Search Engines: Enabling Information Technology and Scientific Applications Simple Search (stateless) Financial: Build Portfolio Medicine: Find Cure Environment: Plan Forestation Travel: Plan a Trip Complex multi-business applications B. Ramamurthy 2/22/2019

Web Services Standard A common operation on the Internet is search, the results of which is consumed by humans. We want to develop complex multi-business applications that are beyond the current search-type applications. Webservices (WS) is a standard that has been introduced by W3 consortium to address this important transition. Grid takes the web services to the next level: a grid service (GS) is a web service. GS = WS + state + standard features for security, reliability, integration, … Grid specifies a standard architecture, infrastructure, protocols and application program interface (API) for an open enterprise system. B. Ramamurthy 2/22/2019

Technology Pipeline Web 3.0 Web 2.0 ...... Internet Grid/GS …… ...... Internet Technology Pipeline …… Web/WS Grid/GS Technology Pipeline B. Ramamurthy 2/22/2019

Virtualization Encapsulating service operations behind a common message-oriented service interface is called service virtualization. Isolates users from details of service implementation and location. Assumes support of a standard architecture. Webservices (WS) can do this, however life cycle management, fault handling and other features we have seen in the Grid are not available with WS. Grid specification addresses these issues using a core set of standard services. B. Ramamurthy 2/22/2019

On to data-intensive applications Applicable to a certain class of applications MapReduce + GFS from Google MapReduce + HDFS Amazon’s suite: EC2, S3, mechanical turk, etc. B. Ramamurthy 2/22/2019

MapReduce Split 0 Split 1 Split 2 Split n input map() reduce() part1 partm map tasks reduce tasks output …. B. Ramamurthy 2/22/2019