1. LISHEP 2004 Iosif Legrand Iosif Legrand California Institute of Technology DISTRIBUTED SERVICES

2. LISHEP 2004 Iosif Legrand Distributed Dynamic Services Architecture   Hierarchical structure of loosely coupled services which are independent & autonomous entities able to cooperate using a dynamic set of proxies or self describing protocols.   They need a dynamic registration and discovery & subscription mechanism   For an effective use of distributed resources, these services should provide adaptability and self-organization (aggregation and hierarchical orchestration)   Reliable on a large scale network distributed environment   Avoid single points of failure   Automatic re-activation of components and services   Scalable & Flexible for adding dynamically new services and automatically replicate existing ones to cope with time dependent load

3. LISHEP 2004 Iosif Legrand Distributed Object Systems CORBA, RMI, DCOM Lookup Service Stub Lookup Service Skeleton CLIENT Server “Traditional” Distributed Object Models (CORBA, DCOM) “IDL” Compiler The Stub is linked to the Client. The Client must know about the service from the beginning and needs the right stub for it The Server and the client code must be created together !!

4. LISHEP 2004 Iosif Legrand Web Services WSDL/SOAP Lookup Service WSDL CLIENT Server Lookup Service Interface SOAP The client can dynamically generate the data structures and the interfaces for using remote objects based on WSDL Platform independent

5. LISHEP 2004 Iosif Legrand Distributed Services Act as a true dynamic service and provide the necessary functionally to be used by any other services that require such information (Jini, interface to WSDL / SOAP)   mechanism to dynamically discover all the “Service Units"   remote event notification for changes in the any system   lease mechanism for each registered unit Dynamic Code Loading Lookup Service Proxy CLIENT Lookup Service Proxy Service Services can be used dynamically   Remote Services Proxy == RMI Stub   Mobile Agents Proxy == Entire Service   “Smart Proxies” Proxy adjusts to the client Any well suited protocol for the application

6. LISHEP 2004 Iosif Legrand JINI – Network Services A Service Registers with at least one Lookup Service using the same ID. It provides information about its functionality and the URL addressed from where interested clients may get the dynamic code to use it. The Service must ask each Lookup Service for a lease and periodically renew it. If a Service fails to renew the lease, it is removed form the Lookup Service Directory. When problems are solved, it can re-register. The lease mechanism allows the Lookup Service to keep an up to date directory of services and correctly handle network problems. Servic e Lookup Service Lookup Service Lookup Service Lookup Service CLIENT Register Service ID Register with ID Ask for a lease Get a lease for  T jar Web Server Publish the “Interface” jar jar Web Server Publish the “Interface” jar

7. LISHEP 2004 Iosif Legrand Mobile Agents u Mobile Agents are automatons entities able to migrate between AgentStations. u Each AgentStation offers the runtime environment, priority and controls the access permission for the hosted agents. u Mobile Agents interact with the AgentStations network to get access to the necessary information to migrate and to perform a certain task. u The agent migration is done as a weak migration ( e.g. the agent prepare itself before it is moved and save its state into structured which a serialized) u Mobile Agents should remain small and simple entities. To achieve this, and provide complex functionality they must use and access “static” services hosted by the AgentStation network.

8. LISHEP 2004 Iosif Legrand The Framework for Mobile Agents

9. LISHEP 2004 Iosif Legrand The Components of the Agent Station u AgentStation provides the platform on which agent services can be hosted and allowed to execute. Agents are authenticated by the station using the security mechanism of the framework before they are allowed to execute. Once an agent has been authenticated, it is allocated a new thread for execution.

10. LISHEP 2004 Iosif Legrand Effective File Replication u Identify the efficient way to replicate a data base file to all the interested Regional Centers. Find the right pass and order based on currently available bandwidth and additional constrains of the producer. u Use Remote Events for Global Notification u Allow interested parties to “cooperate” and improve the resource utilization and the global performance and the response time. u e.g. Avoid sending the same file twice over the same transatlantic internet connection if two Regional Centers in U.S. want the same file. u Based on currently available resources to find the optimal way to distribute the file from a global perspective.

11. LISHEP 2004 Iosif Legrand Effective File Replication using Mobile Agents Tier1 A Tier0 Tier2 A Tier2 B Tier1 B Remote Event Notification Java SpaceExample: u Remote Notification for creating a new file u Interested Centers deploy an agent in a time window. u Agents can get from the home Station Sever connectivity parameters u The agent objective is to bring the file home cooperating with other agents u Interested parties may join later and get the information from a JavaSpace

12. LISHEP 2004 Iosif Legrand

13. Monitoring: Data Collection Farm Monitor WEB Server Dynamic Thread Pool SNMP get & walk rsh | ssh remote scripts End-To-End measurements PULL Trap Agent (ucd – snmp) perl Trap Listener PUSH snmp trap Dynamic loading of modules or agents Configuration Control Other tools (Ganglia, MRT…)

14. LISHEP 2004 Iosif Legrand The Muti-Threaded Execution Architecture Each request is done in an independent thread A slow agent / busy node does not perturb the measurements of an entire system Ex: Monitor 300 nodes @ 30 seconds interval  10-15 Threads are running in parallel

15. LISHEP 2004 Iosif Legrand Data Handling Data Model   Configuration Farm, Function (Cluster), Node, Module   Monitored Values {Parameter, Value}   (Automatic) Mapping of the Data Model in : XML, SQL, SOAP, …   Configuration & Results objects are store in a DB (dynamically configurable for InstanDB, Postgres, MySQl, Oracle … )   Subscription to results objects matching a template / predicate   Clients can load filter objects into the Data Cache service and generate any derived (or aggregate) data structures and register to receive them.   Monitored parameters have a life time TIME

16. LISHEP 2004 Iosif Legrand Lookup Service Registration / Discovery / Remote Notification MonALISA Service Lookup Service Client (other service) DiscoveryRegistration MonALISA Service MonALISA Service Services Proxy Multiplexer Services Proxy Multiplexer Client (other service) Data Data Filters & Agents Filters & Agents

17. LISHEP 2004 Iosif Legrand Lookup Service Service Monitor UNIT & Data Handling Farm Monitor Data Cache Service & DB Configuration Control (SSL) Configuration Control (SSL) Lookup Service Predicates & Agents Monitor Data Stores WEB Service WSDL SOAP Client (other service) Java Discovery Registration Client (other service) Web client data McKoi DB MySQL MDS UDP MySQL Other tools User defined loadable Modules to write /sent data Predicates & Agents

18. LISHEP 2004 Iosif Legrand download download Secure – Automatic Update Mechanism for Services and Clients Update Signal SSL Admin Client Discovery MonaLisa Service Lookup Service Lookup Service MonaLisa Service Key store Key Update Signal SSL Web Server Sign Distribution download download Key store   All running services are update using the discovery mechanism   At startup each service check if it an update is done at a set of Web Servers   Clients use the Web Start mechanism

19. LISHEP 2004 Iosif Legrand Global Client for Farms and Network Connectivity DataTAG - - @ CALTECH

20. LISHEP 2004 Iosif Legrand Real-time Data for Large Systems “lxshare” cluster at cern ~ 600 ndoes

21. LISHEP 2004 Iosif Legrand Access to historical and real-time values Past values are presented and the GUI remains a registered listener and the new vales are added Real Time Histograms for various parameters

22. LISHEP 2004 Iosif Legrand Mobile Agents and Filters Simple “Global Load” filter agent Maximum Flow Data Replication Path Agent Deployed to each RC and evaluates the best path for real-time data replication From FNAL to all From CERN to all

23. LISHEP 2004 Iosif Legrand Pseudo – Clients & Dedicated Repositories Filter Agents / Data Pseudo Client Discovery MonaLisa Service MySQL IDB Lookup Service Lookup Service MonaLisa Service MySQL IDB MySQL TOMCAT JSP/servelts Filter Agents / Data WAP WEB

24. LISHEP 2004 Iosif Legrand MonALISA repositories

25. LISHEP 2004 Iosif Legrand PDA 3 D MonALISA Client Developed by Research Lab

26. LISHEP 2004 Iosif Legrand VRVS Architecture vrvs us vrvs eu pub funet star- light sinica kek cor- nell triumf cal- tech usf usp inet 2 vrvs 5 Reflectors are hosts that interconnect users by permanent IP tunnels. The active IP tunnels must be selected so that there is no cycle formed. Tree The selection is made according to the assumed network links performance.

27. LISHEP 2004 Iosif Legrand Input: A weighted connected graph G = (V,E) with n vertices and m edges Output: A minimum- spanning tree T Minimum Spanning Tree Algorithms Finding a tree T that contains all the vertices spanning tree of a graph G spanning tree and has the least total weight over all minimum-spanning tree (MST) such trees minimum-spanning tree (MST)

28. LISHEP 2004 Iosif Legrand Kruskal‘s Algorithm 1. 1.Each vertex is in its own cluster 2. Take the edge e with the smallest weight - if e connects two vertices in different clusters, then e is added to the MST and the two clusters, which are connected by e, are merged into a single cluster - if e connects two vertices, which are already in the same cluster, ignore it 3. Continue until n-1 edges were selected

29. LISHEP 2004 Iosif Legrand C FE AB D 5 64 3 4 2 12 3 2 Kruskal‘s Algorithm

30. LISHEP 2004 Iosif Legrand C FE AB D 5 64 3 4 2 12 3 2 Kruskal‘s Algorithm

31. LISHEP 2004 Iosif Legrand C FE AB D 5 64 3 4 2 12 3 2 Kruskal‘s Algorithm

32. LISHEP 2004 Iosif Legrand C FE AB D 5 64 3 4 2 12 3 2 Kruskal‘s Algorithm

33. LISHEP 2004 Iosif Legrand C FE AB D 5 64 3 4 2 12 3 2 Kruskal‘s Algorithm

34. LISHEP 2004 Iosif Legrand C FE AB D 5 64 3 4 2 12 3 2 cycle!! Kruskal‘s Algorithm

35. LISHEP 2004 Iosif Legrand C FE AB D 5 64 3 4 2 12 3 2 Kruskal‘s Algorithm

36. LISHEP 2004 Iosif Legrand C FE AB D 5 64 3 4 2 12 3 2 Kruskal‘s Algorithm

37. LISHEP 2004 Iosif Legrand C FE AB D 3 2 12 2 minimum- spanning tree Kruskal‘s Algorithm

38. LISHEP 2004 Iosif Legrand Barůvka‘s Algorithm 1. 1.For all vertices search the edge with the smallest weight of this vertex and mark these edges 2. 2.Search connected vertices (clusters) and replace them by a “new“ vertex (cluster) 3. 3.Remove the cycles and, if two vertices are connected by more than one edge, delete all edges except the “cheapest“

39. LISHEP 2004 Iosif Legrand C FE AB D 5 64 3 4 2 12 3 2 Barůvka‘s Algorithm Barůvka‘s Algorithm

40. LISHEP 2004 Iosif Legrand C FE AB D 5 64 3 4 2 12 3 2 Barůvka‘s Algorithm Barůvka‘s Algorithm

41. LISHEP 2004 Iosif Legrand C FE AB D 5 64 3 4 2 12 3 2 Barůvka‘s Algorithm Barůvka‘s Algorithm

42. LISHEP 2004 Iosif Legrand Monitoring VRVS Reflectors

43. LISHEP 2004 Iosif Legrand Monitoring VRVS Reflectors

44. LISHEP 2004 Iosif Legrand Global Client / Dynamic Discovery

45. LISHEP 2004 Iosif Legrand SUMMARY u MonALISA is able to dynamically discover all the “Service Units" used by a community and through the remote event notification mechanism keeps an update state for the entire system u Automatic & secure code update (services and clients). u Dynamic configuration for services. Secure Admin interface. u Access to aggregate farm values and all the details for each node u Selected real time / historical data for any subscribed listeners u Active filter agents to process the data and provided dedicated / customized information to other services or clients. u Mobile Agents for decision support and global optimization. u Dynamic proxies and WSDL & WAP pages for services. u Embedded SNMP support and interfaces with other tools ( LSF, PBS, Ganglia, Hawkeye, IEPM-BW…) u Dedicate pseudo-clients for repository, WAP access or decision making units u It proved to be a stable and reliable distributed service system. It is currently running at ~ 100 sites http://monalisa.cacr.caltech.edu