Distributed Virtual Computer (DVC): Simplifying the Development of High-Performance Grid Applications Nut Taesombut and Andrew A. Chien Department of Computer Science and Engineering University of California, San Diego Workshop on Grids and Advanced Networks (GAN’04) Chicago, Illinois April 22, 2004

6/3/20152Outline Background and Motivation Distributed Virtual Computer (DVC) Example Application Related Work Summary and Future Work

6/3/20153 Emerging Opportunity of Lambda Grids Network Advances and Trends –DWDM optical paths (or lambdas) enable –Dedicated High Bandwidth –Dynamic Configuration Lambda Grid –Distributed, shared resources interconnected by plentiful lambdas –Configurable Connections and Capacity –Deterministic Communication Performance –Novel Communication Capabilities (e.g. optical multicast)

6/3/20154 OptIPuter Project OptIPuter – Large-scale Research Project on Impact of Lambdas on System Software and next-generation E-science –International Testbed for Experimentation (UCSD, UIC, UCI, Amsterdam, etc.) –Leading E-science Drivers (Neuroscience, Geophysical/Earth Sciences) –3-D Data Analysis, Visualization and Collaboration Applications –Data-intensive and Real-time, Distributed data sources/sinks –Wealth of Innovative System Software Research (protocols, DVC, storage, etc.) Smarr, Papadopoulos, Ellisman, Orcutt, Chien – UCSD DeFanti, Leigh - UIC

6/3/20155Motivation Building Grid Applications is Difficult! –Applications must deal with complexity of resource environment –Resource Heterogeneity, Performance, Communication –Multi-Organization Security, Resource Management –Shared and Untrusted Resource Environments –Low-Quality Networks Adding Low-level Management of Network Complicates the Task –No Uniform Interfaces (routers, switches, end nodes) –Wildly Different Semantic Level (BELOW IP!) –Novel Capabilities (e.g. multicast, RDMA, etc.) Key Requirements –A new abstraction which simplifies Grid environments –A view which integrates communication as first class

6/3/20156 Example of Grid Complexity Access to Resources Across Multiple Namespaces User-Controlled Configuration of Dynamic Lambda Network Heterogeneous Communication Internet

6/3/20157 Distributed Virtual Computer (DVC) A Simple Execution Environment for Grid Applications –Set of LambdaGrid resources (connections, resources) –Naming, access, and management services –Transparently shared amongst Applications Simplify Use of Network and Grid Capabilities –Automate compute/data resource binding –Automate dynamic λ-configuration; expose novel λ-capabilities –Leverage existing Grid Technologies (Globus, NWS etc.) DVC

6/3/20158 DVC Design Principles Separate Resource Config/Mgmt and Application Programming –Resource Environments Configured to Spec –Applications simply use them Aggregate and Bind Grid Resources; Present as Workgroup –Central resource control –Single namespace –Unified resource access mechanisms –Trusted and secure environment –Controllable performance Enable Collective Resource Views –Unified naming structures (e.g. collective names) –Collective properties (e.g. group communication, trust, access control)

6/3/20159 Example: Locally Simplified Grid Programming Single Control Domain Unified Naming Mechanism Uniform Use of Different Communication Mechanisms (e.g. protocols) DVC Domain comp3 str1 comp2 comp1 Simple View of DVC Internet

6/3/ How DVC’s Simplify Application Grid Programming Automate Resource Binding and Configuration –Reduce user interaction through resource broker and manager Unify Resource Naming and Access Mechanisms –Hide heterogeneity through simple abstractions Transparently Enable Security Protection –Implement cryptographic operations at the middleware layer Monitor and Control Resource and Communication Performance –Detect asynchronous events and notify application based on subscription

6/3/ Realizing Simplified Application Grid Programming DVC Manager –Single master controller –Resource selector/negotiator/scheduler –Trust mediator and security authority –Synchronizer of global state information Ghost Managers –Slave managers, each running at each bound resource –Job process controller at remote resource –Communication mediator –Resource status monitor and reporter DVC DVC Manager Ghost Manager Control Flow Data Flow

6/3/ Example: Dynamic Configuration of Lambda Grid Example Application: –BIRN (Biomedical Information Research Network) UCI NCMIR/UCSD SDSC Harvard UCLA UNC Duke BIRN DVC DVC Advantages –On-demand creation of a private Grid resource workgroup –Transparent use of novel communication capabilities –high-speed multi-point communication –SAN-like storage access across geographically distributed resources

6/3/ Example: Dynamic Configuration of Lambda Grid Sequence to Create a BIRN DVC 1.Create a resource configuration specification and send a request to bind resources 2.Create resource groups (i.e. for collective data source and sink) 3.Create multipoint-to-point and point-to-multipoint communication sessions 4.Define the properties of communication sessions (e.g. security and communication mechanisms) UCI NCMIR/UCSD SDSC Harvard UCLA UNC Duke ucsd harvard duke sdsc uci unc GTP + enc + auth TCP + Optical Multicast Physical-Level View of BIRN DVC Application-Level View of BIRN DVC Grp1 Grp2

6/3/ Related Work Abstractions of Distributed Resources –PVM [Geist94] Grid Middleware –Globus System Grid Programming Tools –GridRPC [Nakada02], MPICH-G2 [Karonis03], Condor-G [Frey01] –GrADS [Berman01], GridLab [Allen03], Federation of Resources –Virtual Organization (VO) [Foster01] Distributed Virtual Computer –Provide an Application-Focused Dynamic Resource Container –Dynamic resource configuration and sharing policies

6/3/ Summary and Future Work Summary –DVC’s provide simple computing environments for Grid applications –Locally simplified resource workgroup –DVC’s allow on-demand instantiation and dynamic configuration of Lambda-Grid –DVC’s enable simple use of novel communication capabilities Future Work –Develop the full implementation of the DVC Prototype –Implement as Web Services (i.e. WS-RF specification) –Deploy the prototype on the OptIPuter Testbed –Demonstrate with OptIPuter applications (Bioinformatics and Geophysical) –Explore other system technologies that can be integrated into the DVC framework –Real-time System –High-Performance Distributed File System

6/3/ Thank You Questions and Remarks? Contact Information: –Nut Taesombut OptIPuter Website: –