2. Denis Caromel1 Institut universitaire de France (IUF) OASIS Team INRIA -- CNRS - I3S -- Univ. of Nice Sophia-Antipolis REUNA, Santiago, May 2004 GRID and Components for Scientific Computing 1. Grid principles 2. Distributed Objects and Components: ObjectWeb ProActive LGPL environment 3. Application: 3D Electromagnetism 4. Towards Peer-To-Peer (P2P)

3. Denis Caromel2 The GRID PCs : 1 Milliard in 2002 (25 years) Forecast: 2 Milliards in 2008

4. Denis Caromel3 The Grid idea GRID = Electric Network Computer Power (CPU cycles) Electricity Can hardly be stored, if not used --> Lost But CPU cycles much harder to share than electricity: Production cannot be adjusted! Cannot really be delivered where needed! Not yet interoperability: Multiple Administrative Domains 2 important aspects : Computational + Data Grid Global management, Mutual sharing of the resource

5. Denis Caromel4 Example 50 Machines, 1.5 year of computation 5000 Machines, with only 50 % Efficiency ==> 10 days Applications: Research for an urgent vaccine Forecast a bush-fire path Forecast in real time a flooding consequences... etc. Challenge: Scale up

6. Denis Caromel5 Ubiquitous: some numbers PCs in my lab (INRIA Sophia) : ~ 1500 French Riviera : 1.3 Millions France : 25 Millions Europe : 108 Millions USA : 400 Millions Chile: ?? Millions South America: ?? Millions World : 1 Milliard in 2002 (25 ans) Forecast: 2 Milliards in 2008 France : 36 Millions of cellular phones 2.2 Millions of laptops 630 Thousand PDA (sources: ITU, Gartner Dataquest, IDC, 02-03, )

7. Denis Caromel6 The multiple GRIDs Scientific Grids : Parallel machines, Clusters Large equipments: Telescopes, Particle accelerators, etc. Enterprise Grids : Data, Integration: Web Services Remote connection, Security Intranet and Internet Grids, (miscalled P2P grid): Desktop office PCs: Desktop Intranet Grid Home PC: Internet Grid (e.g. SETI@HOME)

8. Denis Caromel7 Enterprise Grids Internet EJBServlets Apache Databases

9. Denis Caromel8 Scientific Grids Internet Clusters Parallel Machine Large Equipment

10. Denis Caromel9 Internet Grids Internet Job management for embarrassingly parallel application (e.g. SETI)

11. Denis Caromel10 The multiple GRIDs Scientific Grids Enterprise Grids Intranet and Internet Grids Strong convergence in process! At least at the infrastructure level, i.e. WS

12. Denis Caromel11 Grid: from enterprise... to regional Very hard deployment problems … right from the beginning

13. Denis Caromel12 Grid: from regional... to worldwide Communication France - Santiago: 70 ms Light Speed Challenge: Hide the latency ! Define adequate programming model

14. Denis Caromel13 Distributed Objects and Components ProActive

15. Denis Caromel14 Model: Remote Objects Asynchronous Communications, with automatic Futures Group Communications, Migration (computation mobility) Environment: XML Deployment, dynamic class-loading Various protocols: rsh, ssh, LSF, Globus, BPS,... Graphical Visualization and monitoring: IC2D ProActive: A Java API + Tools for the GRID Parallel, Distributed, Mobile, Activities, across the world ! SMPClustersLAN Desktop

16. Denis Caromel15 A Creating AO and Groups Typed Group Java or Active Object A ag = newActiveGroup (“A”, […], VirtualNode) V v = ag.foo(param);... v.bar(); //Wait-by-necessity V Group and Asynchrony are crucial for the GRID

17. Denis Caromel16 Mobility Same semantics guaranteed (RDV, FIFO order point to point, asynchronous) Safe migration (no agent in the air!) Local references if possible when arriving within a VM Tensionning (removal of forwarder)

18. Denis Caromel17 Mobility Same semantics guaranteed (RDV, FIFO order point to point, asynchronous) Safe migration (no agent in the air!) Local references if possible when arriving within a VM Tensionning (removal of forwarder)

19. Denis Caromel18 Mobility Same semantics guaranteed (RDV, FIFO order point to point, asynchronous) Safe migration (no agent in the air!) Local references if possible when arriving within a VM Tensionning (removal of forwarder) direct

20. Denis Caromel19 Mobility Same semantics guaranteed (RDV, FIFO order point to point, asynchronous) Safe migration (no agent in the air!) Local references if possible when arriving within a VM Tensionning (removal of forwarder) direct

21. Denis Caromel20 Mobility Same semantics guaranteed (RDV, FIFO order point to point, asynchronous) Safe migration (no agent in the air!) Local references if possible when arriving within a VM Tensionning (removal of forwarder) direct forwarder

22. Denis Caromel21 Mobility Same semantics guaranteed (RDV, FIFO order point to point, asynchronous) Safe migration (no agent in the air!) Local references if possible when arriving within a VM Tensionning (removal of forwarder) direct forwarder

23. Denis Caromel22 Mobility Same semantics guaranteed (RDV, FIFO order point to point, asynchronous) Safe migration (no agent in the air!) Local references if possible when arriving within a VM Tensionning (removal of forwarder) direct forwarder

24. Denis Caromel23 Mobility Same semantics guaranteed (RDV, FIFO order point to point, asynchronous) Safe migration (no agent in the air!) Local references if possible when arriving within a VM Tensionning (removal of forwarder) direct forwarder

25. Denis Caromel24 Parallel, Distributed, Hierarchical Components for the Grid

26. Denis Caromel25 A CORBA Component My Business Component Component interface Facets Event sources Event sinks Attributes Receptacles OFFERED REQUIRED Courtesy of Philippe Merle, Lille, OpenCCM platform

27. Denis Caromel26 Building CCM Applications = Assembling CORBA Component Instances Provide + Use, but flat assembly

28. Denis Caromel27 Content Controller The Fractal model: Hierarchical Component Defined by E. Bruneton, T. Coupaye, J.B. Stefani, INRIA & FT

29. Denis Caromel28 Content Controller Interface = access point

30. Denis Caromel29 Content Controller Hierarchical model : composites encapsulate primitives, which encapsulates Java code

31. Denis Caromel30 Content Controller Binding = interaction

32. Denis Caromel31 Binding = interaction Content Controller

33. Denis Caromel32 Controllers : non-functional properties Component Identity Binding Controller LifeCycle Controller Content Controller Content Controller Component = runtime entity

34. Denis Caromel33 3. Parallel and composite component 1. Primitive component Java + Legacy 2. Composite component ProActive Components for the GRID An activity, a process, … potentially in its own JVM C D Composite: Hierarchical, and Distributed over machines Parallel: Composite + Broadcast (group)

35. Denis Caromel34 A A B C P Group proxy A B C D Groups in Components Broadcast at binding, on client interface At composition, on composite inner server interface A parallel component!

36. Denis Caromel35 Migration Capability of composites Migrate sets of components, including composites

37. Denis Caromel36 Migration Capability of composites Migrate sets of components, including composites

38. Denis Caromel37 Co-allocation, Re-distribution e.g. upon communication intensive phase

39. Denis Caromel38 Co-allocation, Re-distribution e.g. upon communication intensive phase

40. Denis Caromel39 Co-allocation, Re-distribution e.g. upon communication intensive phase

41. Denis Caromel40 Environments

42. Denis Caromel41 How to deploy on the Various Kind of Grids ? Internet EJBServlets Apache Databases Internet Clusters Parallel Machine Large Equipment Internet Job management for embarrassingly parallel application (e.g. SETI)

43. Denis Caromel42 Abstract Deployment Model Problem: Difficulties and lack of flexibility in deployment Avoid scripting for: configuration, getting nodes, connecting, etc. A key principle: Virtual Node (VN) + XML deployment file Abstract Away from source code: Machines Creation Protocols Lookup and Registry Protocols Protocols and infrastructures: Globus, ssh, rsh, LSF, PBS, … Web Services, WSRF,...

44. Denis Caromel43 Mapping Virtual Nodes: example (1) <sshProcess class="org.objectweb.proactive.core.process.ssh.SSHJVMProcess" hostname="sea.inria.fr"> Infrastructure informations JVM on the current HostJVM started using SSH

45. Denis Caromel44 <bsubProcess class="org.objectweb.proactive.core.process.lsf.LSFBSubProcess" hostname=”cluster.inria.fr"> 12 Mapping Virtual Nodes: example (2) Infrastructure informations Definition of LSF deployment, … Globus

46. Denis Caromel45 C AB VNaVNb C AB VNc = VN(a,b) XML Deployment (Not in source) Separate or Co-allocation

47. Denis Caromel46 IC2D: Interactive Control and Debugging of Distribution

48. Denis Caromel47 Monitoring of RMI, Globus, Jini, LSF cluster Nice -- Baltimore ProActive IC2D: Width of links proportional to the number of com- munications

49. Denis Caromel48 Application

50. Denis Caromel49 A Parallel Object-Oriented Application for 3D Electromagnetism Visualize the radar reflection of a plane, medicine on head, etc. Pre-existing Fortran MPI version: EM3D Jem3D: Sequential object-oriented design, modular and extensible (in Java) Sequential version can be smoothly distributed: --> keeping structuring and object abstractions Efficient distributed version, large domains, Grid environment

51. Denis Caromel50 The interface

52. Denis Caromel51 Some Results Seq. Java/Fortran: 2 Comparison: Jem3D over - ProActive/RMI Sun - ProActive/RMI Ibis Em3D in - Fortran/MPI On 16 machines: Fortran: 13.8 ProActive/Ibis: 12 ProActive/RMI: 8.8 Grid experiment on 5 clusters (DAS 2): Speed up of 100 on 150 machines

53. Denis Caromel52 Perspective: P2P

54. Denis Caromel53 Architectures: Server to Peer-To-Peer (P2P) Internet EJBServlets Apache Databases SOA: Service Oriented Architectures

55. Denis Caromel54 Pure P2P: Definition Only PEERs, no above everything, top level, server(s) Every peer is, somehow, also a server No master … No slave ! System get organized dynamically, without static configuration Coherent, desired behavior, dynamically emerges

56. Denis Caromel55 P2P Examples (1)

57. Denis Caromel56 P2P can be difficult: need to be fault-tolerant, self healing

58. Denis Caromel57 Not a P2P system

59. Denis Caromel58 Neither a P2P system

60. Denis Caromel59 P2P Examples (2)

61. Denis Caromel60 P2P Examples (2bis)

62. Denis Caromel61 A P2P system at work Credit: from the movie Atlantis, Luc Besson

63. Denis Caromel62 Conclusion GRIDs: - Scientific - Enterprise - Internet Strong convergence in process (infrastructure): WS, WSRF Challenge: adequate programming models Asynchronous distributed objects, Mobility, Components High-level of abstraction, still Open and flexible Modern languages: Java, or others not Fortran, MPI, C++, … not the answer Perspectives: Real P2P Interactive, Graphical, Deployment and Control: Content http://ProActive.ObjectWeb.org

64. Denis Caromel63 Interactive Composition in IC2D Content Composition View Instead of XML ADL <sshProcess class="org.objectweb.proactive.core.process.ssh.SSHJVMProcess" hostname="sea.inria.fr">

65. Denis Caromel64 Merging Component + Activity / JVM Views: Component Monitoring Content Dynamic View

66. Denis Caromel65 Content Dynamic View Merging Component + Activity / JVM Views: Component Monitoring

67. Denis Caromel66 Conclusion (2) Infrastructure and Deployment Virtual Nodes and XML Protocols: ssh, Globus, LSF, PBS, … Transport: RMI, Ibis (Amsterdam) Web Services: XML, WSDL, SOAP (ongoing) OSGi (future work) Available in LGPL with ProActive http://ProActive.ObjectWeb.org in The freedom your applications deserve! Currently: a single application on 300 machines Goal: towards a few 1000s !

68. Denis Caromel67 C3D: distributed-//-collaborative