Grid’5000 Introduction to Grid' Grid’5000 A Nation Wide Experimental Grid

Grid’5000 Introduction to Grid' Grid raises a lot of research issues: Security, Performance, Fault tolerance, Load Balancing, Fairness, Coordination, Message passing, Data storage, Programming, Communication protocols and architecture, Deployment, etc. Theoretical models and simulators cannot capture real life conditions Production platforms have strong difficulties to reproduce experimental conditions How to test and compare? Fault tolerance protocols Security mechanisms Deployment tools etc. Grid: Distributed System Problematic renewal

Grid’5000 Introduction to Grid' log(cost) log(realism) mathsimulation emulation live systems Models: Sys, apps, Platforms, conditions Real systems Real applications Real platforms Real conditions Tools for Distributed System Studies To investigate Distributed System issues, we need: 1) Tools (model, simulators, emulators, experi. Platforms) 2) Strong interaction between these research tools Tools for Large Scale Distributed Systems Real systems Real applications “In-lab” platforms Synthetic conditions Key system mecas. Algo, app. kernels Virtual platforms Synthetic conditions

Grid’5000 Introduction to Grid' log(cost) log(realism) mathsimulation emulation live systems SimGrid MicroGrid Bricks NS, etc. Model Protocol proof Grid eXplorer WANinLab Emulab Grid’5000 TERAGrid PlanetLab Naregi Testbed We need a Grid experimental platform According to the current knowledge: There is no large scale testbed dedicated to Grid experiments  Grid’5000 as a live system  Grid eXplorer as a large scale emulator

Grid’5000 Introduction to Grid' )Remotely controllable Grid nodes installed in geographically distributed laboratories 2)A « Controllable » and « Monitorable » Network between the Grid nodes 3)A middleware infrastructure connecting the nodes (security) 4)A playground to prepare experiments 5)A toolkit to deploy, manage, run experiments and collect results What do we need for Grid experiments ?

Grid’5000 Introduction to Grid' )Building a nation wide experimental platform for Grid researches (like a particle accelerator for the computer scientists) 10/11 geographically distributed sites every site hosts a cluster (from 256 CPUs to 1K CPUs) All sites are connected by RENATER (French Academ. Network) RENATER hosts probes to trace network condition load Design and develop a system/middleware environment for safely test and repeat experiments 2) Use the platform for Grid experiments Address critical issues of Grid system/middleware: Programming, Scalability, Fault Tolerance, Scheduling Address critical issues of Grid Networking High performance transport protocols, Qos Port and test applications Investigate original mechanisms P2P resources discovery, Desktop Grids The Grid’5000 Project

Grid’5000 Introduction to Grid' Lab’s Network LAB/Firewall Router Test Cluster Control Master Site 1 Site 2 Site 3 Users (ssh loggin + password) Firewall/nat Control Slave Test Cluster Front end Control Slave Control site Grid’5000 Big Picture Gateway +VPN (192. For all nodes) One machine Can be seen as a Virtual Grid Gateway

Grid’5000 Introduction to Grid' Thierry Priol (ACI Grid Director) -Brigitte Plateau (President of ACI Grid SC) -Dani Vandrome (Director of Renater) -Frédéric Desprez (Lyon) -Michel Daydé (Toulouse) -Yvon Jégou (Rennes) -Stéphane Lantéri (Sophia) -Raymond Namyst (Bordeaux) -Pascale Primet (Lyon) -Olivier Richard (Grenoble) Steering Committee: (organizer: Franck Cappello, Orsay) Technical Committee: -David Gueldrech (Sophia) -Jean Claude Barbet (Orsay) -Franck Bonnassieux (UREC) -Julien le duc (Grenoble) -Fred Desprez (Lyon) -Yvon Jégou (Rennes) -Olivier Coulaud (Bordeaux) -Frédéric Barbaresco (Toulouse) Forums: Deployment/exploitation: Franck Cappello (AS1, RTP8) Programming models: Raymond Namyst (AS2, RTP8) Grid’5000 Committees

Grid’5000 Introduction to Grid' Grid’5000 Schedule Grid’5000 Hardware Call for proposals Sept03 Selection of 7 sites Nov03 ACI GRID Funding Jan04 Call for Expression Of Interest March04 Vendor selection Jun/July 04 Instal. First tests Spt 04 Final review Oct 04 Fisrt Demo (SC04) Nov 04 Grid’5000 System/middleware Forum Security Prototypes Control Prototypes Grid’5000 Programming Forum Grid’5000 Builder Community Grid’5000 Experiments

Grid’5000 Introduction to Grid' Grid’5000 Funding (ACI + Local District/Prefecture) Grid’ ,6M€ ~0,4€ ~0,35€ ~0,5€ ~0,35€ ~0,3?€ ~0,5€ ~3M€ for hardware only

Grid’5000 Introduction to Grid' Grid’5000 in September’2004 Grid 5000 nodes (soon 4 ) 3

Grid’5000 Introduction to Grid' Summary of Grid5000 XPs Networking –End Host Communication layer –High performance long distance protocols –High Speed Network Emulation –Grid Networking Layer Middleware / OS –Grid’5000 control/access/experiment automation –Scheduling / data distribution in Grid –Fault tolerance in Grid –Resource management –Computational Steering –Grid SSI OS and Grid I/O –Desktop Grid/P2P systems Programming –Component programming for the Grid (Java, Corba) –GRID-RPC –GRID-MPI –Code Coupling Applications –Multi-parametric applications (Climate modeling/Functional Genomic) –Large scale experimentation of distributed applications (Electromagnetism, multi-material fluid mechanics, parallel optimization algorithms, CFD, astrophysics –Medical images, Collaborating tools in virtual 3D environment

Grid’5000 Introduction to Grid' Middleware1(XP)Grid5000 Grid’5000 control - Computing Environment deployment (Ka-tools) - Experiment automation (security and control) - VGrid « mapping a virtual Grid on a real testbed » - Monitoring, benchmarking, performance characterization and analysis Grid Scheduling / data distribution - Scheduling : Data transfers, global communications, work stealing,... - Data re-distribution in Grid - Task distribution and load balancing in heterogeneous Grid - Mixed Parallelism (task and data parallelism) - Mixing data management and task scheduling - Hierarchical and Distributed Scheduling Fault tolerance in Grid - Fault tolerant Grid-RPC (RPC-V) - Hierarchical Fault tolerant MPI (MPICH-V) - Fault tolerant in data-flow approach (Athapascan) Grid SSI OS and Grid I/O - Grid file system (NFSG) - Grid-aware OS (Kerrighed) - Coupling Computational Grid with Reality Center XP: eXPeriments on

Grid’5000 Introduction to Grid' Middleware2(XP)Grid5000 Grid Management -AROMA tool : resources management over a Grid of clusters with different classes of services -Mobile agents for open Grid management -Management of Grids and hosted services (security, QoS, monitoring & control, dynamic configuration, …) -Optimization for wide area distributed query processing -Tools to support the development, administration and usage of heterogeneous resources over the Grid -Virtualization of data storage on Grids -Automatic Deployment of GridRPC middle tier. - Multiclusters and lightweights Grid resource management (OAR/CIGRI) Global Computing/P2P Middleware - Executing Web Services on Desktop Grid Workers (XtremWeb) - Distributing the Coordination in Desktop Grids (XtremWeb) - Harnessing Clusters as parallel Workers - Probabilistic certification in peer-to-peer systems - Large Scale Data Sharing Service based on JXTA (JuxMem) - Management services for textual document in P2P systems

Grid’5000 Introduction to Grid' End Host Communication layer - Communication libraries: Madeleine, MPICH/Madeleine - Intelligent Usage of NICs for local and wide area communications - Direct file access over Myrinet : ORFA/NFS and ORFA/LUSTRE High performance long distance protocols - Alternative Transport for very high speed networks (backpressure) - Differentiated transport with delay control on WAN - Reliable active and non active Multicast - Network Bandwidth optimization in Grid (VTHD++, Paco++). - High performance communication across heterogeneous networks - Fast forwarding and Multiplexing of data on gateway nodes High Speed Network Emulation - Automatic Deployment of emulated high speed domains - Experiment design for grid flow interactions studies Grid Networking Layer - Network Resource and QoS on demand - Grid Overlay and Programmable Routers - Measurement Services for network aware middleware Network(XP)Grid5000

Grid’5000 Introduction to Grid' Component programming on the grid - ProActive : a JAVA library (parallel, distributed, concurrent computing with security and mobility) - Assessment of scalability, deployment, security and fault tolerance issues - Hierarchical components architecture - PadicoTM/Paco++ combining parallel and distributed computing RPC Environment - Large scale experimentation of the DIET platform (Distributed Interactive Engineering Toolbox) - Client/Agent/Server model following the GridRPC standard with distributed scheduling agents MPI Environment - Time sharing Grid resources - Migration over Clusters with heterogeneous high speed networks Code Coupling - Application coupling with Athapascan - Communication / method invocation rescheduling into ORB (HOMA) - Fluid transfer simulation and geological code with PadicoTM/Paco++ Programming(XP)Grid5000

Grid’5000 Introduction to Grid' Applications1(XP)Grid5000 Multi-parametric applications - ACI GRID-TLSE Project : expertise site for sparse linear algebra - Climate modeling and Global Change -DataGène Project : Functional genomic Large scale experimentation of distributed applications –MECAGRID (ACI GRID project, Smash project-team) Massively parallel computations in multi-material fluid mechanics Study of numerical algorithms for heterogeneous computing platforms –Grid computing for medical applications (Epidaure project-team) Interoperable medical image registration grid service –Optimal design of complex systems (Coprin project-team) Evaluation of parallel optimization algorithms based on interval analysis techniques Study of load balancing strategies on heterogeneous resources -Fluid mechanics, molecular dynamics and host-parasite systems in population dynamics, etc. + CFD, astrophysics,… applications + Collaborating tools in virtual 3D environment.

Grid’5000 Introduction to Grid' Applications2(XP)Grid5000 Steering –JECS: a JAVA Environment for Computational Steering Distributed computing and interactive visualization of 3D numerical simulations (Caiman and Oasis project-teams) Collaborative environment Computational Electromagnetism application (JEM3D) –Steering of numerical simulations (ACI GRID-EPSN Project) Parallel on-line visualization / monitoring Data Redistribution Computational Steering by direct image manipulation