1. 1 Iosif Legrand, Harvey Newman, Ramiro Voicu, Costin Grigoras, Catalin Cirstoiu, Ciprian Dobre An Agent Based, Dynamic Service System to Monitor, Control and Optimize Distributed Systems Control and Optimize Distributed Systems ACAT - November 2008 ERICE

2. Iosif Legrand November 2008 2 2 The MonALISA Framework  MonALISA is a Dynamic, Distributed Service System capable to collect any type of information from different systems, to analyze it in near real time and to provide support for automated control decisions and global optimization of workflows in complex grid systems.  The MonALISA system is designed as an ensemble of autonomous multi-threaded, self-describing agent-based subsystems which are registered as dynamic services, and are able to collaborate and cooperate in performing a wide range of monitoring tasks. These agents can analyze and process the information, in a distributed way, and to provide optimization decisions in large scale distributed applications.

3. Distributed Object Systems CORBA, DCOM Lookup Service Stub Lookup Service Skeleton CLIENT Server “Traditional” Distributed Object Models (CORBA, DCOM) “IDL” Compiler The Stub is linked to the Client. The Client must know about the service from the beginning and needs the right stub for it The Server and the client code must be created together !!

4. Distributed Object Systems Web Services WSDL/SOAP Lookup Service WSDL CLIENT Server Lookup Service Interface SOAP The client can dynamically generate the data structures and the interfaces for using remote objects based on WSDL Platform independent

5. Iosif Legrand November 2008 5 Mobile Code and Distributed Services Act as a true dynamic service and provide the necessary functionally to be used by any other services that require such information (Jini, interface to WSDL / SOAP)  mechanism to dynamically discover all the “Service Units"  remote event notification for changes in the any system  lease mechanism for each registered unit Dynamic Code Loading Lookup Service Proxy CLIENT Lookup Service Proxy Service Services can be used dynamically  Remote Services Proxy == RMI Stub  Mobile Agents Proxy == Entire Service  “Smart Proxies” Proxy adjusts to the client Any well suited protocol for the application

6. Iosif Legrand November 2008 6 MonALISA Service & Data Handling 6 Data Store Data Cache Service & DB Configuration Control (SSL) Predicates & Agents Data (via ML Proxy) Applications Clients or Higher Level Services WS Clients and service Web Service WSDL SOAP Lookup Service Lookup Service Registration Discovery Postgres AGENTS FILTERS / TRIGGERS Monitoring Modules Collects any type of information Dynamic Loading Push and Pull

7. Iosif Legrand November 2008 7 The MonALISA Architecture 7 Regional or Global High Level Services, Repositories & Clients Secure and reliable communication Dynamic load balancing Scalability & Replication AAA for Clients Distributed Dynamic Registration and Discovery- based on a lease mechanism and remote events JINI-Lookup Services Secure & Public MonALISA services Proxies HL services Agents Network of Distributed System for gathering and analyzing information based on mobile agents: Customized aggregation, Triggers, Actions Fully Distributed System with no Single Point of Failure

8. Iosif Legrand November 2008 8 Lookup Service Registration / Discovery Admin Access and AAA for Clients MonALISA Service Lookup Service Client (other service) Discovery Registration (signed certificate) MonALISA Service MonALISA Service Services Proxy Multiplexer Services Proxy Multiplexer Client (other service) Admin SSL connection Trust keystore AAA services Client authentication Data Data Filters & Agents Filters & Agents Trust keystore Application Applications

9. Iosif Legrand November 2008 9 Monitoring Grid sites, Running Jobs, Network Traffic, and Connectivity 9 TOPOLOGY JOBS ACCOUNTING Running Jobs

10. Iosif Legrand November 2008 10 Monitoring architecture in ALICE 10 Long History DB LCG Tools MonALISA @Site ApMon AliEn Job Agent ApMon AliEn Job Agent ApMon AliEn Job Agent MonALISA @CERN MonALISA LCG Site ApMon AliEn CE ApMon AliEn SE ApMon Cluster Monitor ApMon AliEn TQ ApMon AliEn Job Agent ApMon AliEn Job Agent ApMon AliEn Job Agent ApMon AliEn CE ApMon AliEn SE ApMon Cluster Monitor ApMon AliEn IS ApMon AliEn Optimizers ApMon AliEn Brokers ApMon MySQL Servers ApMon CastorGrid Scripts ApMon API Services MonaLisaRepository Aggregated Data rss vsz cpu time run time job slots free space nr. of files open files Queued JobAgents cpu ksi2k job status disk used processes load net In/out jobs status sockets migrated mbytes active sessions MyProxy status Alerts Actions

11. Iosif Legrand November 2008 11 http://pcalimonitor.cern.ch ALICE : Global Views, Status & Jobs

12. Iosif Legrand November 2008 12 ALICE: Job status – history plots

13. Iosif Legrand November 2008 13 ALICE: Resource Usage monitoring   Cumulative parameters   CPU Time & CPU KSI2K   Wall time & Wall KSI2K   Read & written files   Input & output traffic (xrootd)   Running parameters   Resident memory   Virtual memory   Open files   Workdir size   Disk usage   CPU usage   Aggregated per site

14. Iosif Legrand November 2008 14 ALICE: Job agents monitoring   From Job Agent itself   Requesting job   Installing packages   Running job   Done   Error statuses   From Computing Element   Available job slots   Queued Job Agents   Running Job Agents

15. Iosif Legrand November 2008 15 Monitoring the Execution of Jobs and the Time Evolution 15 SPLIT JOBS LIFELINES for JOBS Job Job1 Job2 Job3 Job 31 Job 32 Summit a Job DAG

16. Iosif Legrand November 2008 16 Two levels of decisions: local (autonomous), global (correlations). Actions triggered by: values above/below given thresholds, absence/presence of values, correlations between any values. Action types: alerts (emails/instant msg/atom feeds), running an external command, automatic charts annotations in the repository, running custom code, like securely ordering a ML service to (re)start a site service. ML Service Actions based on global information Actions based on local information Traffic Jobs Hosts Apps Temperature Humidity A/C Power … Sensors Local decisions Global decisions Local and Global Decision Framework Global ML Services

17. Iosif Legrand November 2008 17 ALICE: Automatic job submission Restarting Services 17 MySQL daemon is automatically restarted when it runs out of memory Trigger: threshold on VSZ memory usage ALICE Production jobs queue is kept full by the automatic submission Trigger: threshold on the number of aliprod waiting jobs Administrators are kept up-to-date on the services’ status Trigger: presence/absence of monitored information

18. Iosif Legrand November 2008 18 ALICE is using the monitoring information to automatically: resubmit error jobs until a target completion percentage is reached, submit new jobs when necessary (watching the task queue size for each service account)‏ production jobs, RAW data reconstruction jobs, for each pass, restart site services, whenever tests of VoBox services fail but the central services are OK, send email notifications / add chart annotations when a problem was not solved by a restart, dynamically modify the DNS aliases of central services for an efficient load-balancing. Most of the actions are defined by few lines configuration files. Automatic actions in ALICE

19. Iosif Legrand November 2008 19 Monitoring USLHCnet  Operations & management assisted by agent-based software  Used on the new CIENA equipment used for network managment

20. Iosif Legrand November 2008 20 USLHCnet: Precise measurements for the Operational Status on the WAN Link  Operations & management assisted by agent-based software  Used on the new CIENA equipment used for network managment

21. Iosif Legrand November 2008 21 USLHCnet: Traffic on different segments

22. Iosif Legrand November 2008 22 USLHCnet: Accounting for Integrated Traffic

23. Iosif Legrand November 2008 23 The UltraLight Network BNL ESnet IN /OUT

24. Iosif Legrand November 2008 24 Available Bandwidth Measurements Embedded Pathload module. 24

25. Iosif Legrand November 2008 25 Monitoring Network Topology, Latency, Routers NETWORKS AS ROUTERS Real Time Topology Discovery & Display

26. Iosif Legrand November 2008 26 EVO : Real-Time monitoring for Reflectors and the quality of all possible connections

27. Iosif Legrand November 2008 27 EVO: Creating a Dynamic, Global, Minimum Spanning Tree to optimize the connectivity A weighted connected graph G = (V,E) with n vertices and m edges. The quality of connectivity between any two reflectors is measured every second. Building in near real time a minimum- spanning tree with addition constrains

28. Iosif Legrand November 2008 28 Dynamic MST to optimize the Connectivity for Reflectors Frequent measurements of RTT, jitter, traffic and lost packages The MST is recreated in ~ 1 S case on communication problems.

29. Iosif Legrand November 2008 29 EVO: Optimize how clients connect to the system for best performance and load balancing

30. Iosif Legrand November 2008 30 FDT – Fast Data Transfer  FDT is an application for efficient data transfers.  Easy to use. Written in java and runs on all major platforms.  It is based on an asynchronous, multithreaded system which is using the NIO library and is able to:  stream continuously a list of files  use independent threads to read and write on each physical device  transfer data in parallel on multiple TCP streams, when necessary  use appropriate size of buffers for disk IO and networking  resume a file transfer session

31. Iosif Legrand November 2008 31 FDT – Fast Data Transfer Pool of buffers Kernel Space Pool of buffers Kernel Space Data Transfer Sockets / Channels Independent threads per device Restore the files from buffers Control connection / authorization

32. Iosif Legrand November 2008 32 FDT features April 2007 Iosif Legrand 32  The FDT architecture allows to "plug-in" external security APIs and to use them for client authentication and authorization. Supports several security schemes : IP filtering IP filtering SSH SSH GSI-SSH GSI-SSH Globus-GSI Globus-GSI SSL SSL  User defined loadable modules for Pre and Post Processing to provide support for dedicated MS system, compression …  FDT can be monitored and controlled dynamically by MonALISA services

33. Iosif Legrand November 2008 33 October 2006 Iosif Legrand 33 FDT – Memory to Memory Tests in WAN CPUs Dual Core Intel Xenon @ 3.00 GHz, 4 GB RAM, 4 x 320 GB SATA Disks Connected with 10Gb/s Myricom ~9.0 Gb/s ~9.4 Gb/s

34. Iosif Legrand November 2008 34 Disk -to- Disk transfers in WAN NEW YORK  GENEVA Reads and writes on 4 SATA disks in parallel on each server Mean traffic ~ 210 MB/s ~ 0.75 TB per hour MB/s CERN  CALTECH Reads and writes on two 12-port RAID Controllers in parallel on each server Mean traffic ~ 545 MB/s ~ 2 TB per hour 1U Nodes with 4 Disks 4U Disk Servers with 24 Disks October 2007 Iosif Legrand u Lustre read/ write ~ 320 MB/s between Florida and Caltech u Works with xrootd u Interface to dCache using the dcap protocol

35. Iosif Legrand November 2008 35 Dynamic restoration of lightpath if a segment has problems Monitoring Optical Switches

36. Iosif Legrand November 2008 36 Monitoring the Topology and Optical Power on Fibers for Optical Circuits Port power monitoring Controlling Glimmerglass Switch Example

37. Iosif Legrand November 2008 37 “On-Demand”, End to End Optical Path Allocation 37 Internet A >FDT A/fileX B/path/ OS path available Configuring interfaces Starting Data Transfer Monitor Control TL1 Optical Switch MonALISA Service MonALISA Distributed Service System B OS Agent Active light path Regular IP path Real time monitoring APPLICATION LISA AGENTLISA sets up - Network Interfaces - TCP stack - Kernel parameters - RoutesLISA  APPLICATION “use eth1.2, …” LISA Agent Agent DATA CREATES AN END TO END PATH < 1s Detects errors and automatically recreate the path in less than the TCP timeout path in less than the TCP timeout

38. Iosif Legrand November 2008 38 CERN Geneva CALTECH Pasadena Starlight Manlan USLHCnet Internet2 Controlling Optical Planes Automatic Path Recovery “Fiber cut” simulations The traffic moves from one transatlantic line to the other one FDT transfer (CERN – CALTECH) continues uninterrupted TCP fully recovers in ~ 20s 1 2 3 4 FDT Transfer 4 Fiber cuts simulations 200+ MBytes/sec From a 1U Node 4 fiber cut emulations

39. Iosif Legrand November 2008 39 End to End Path Provisioning on different layers Layer 3 Layer 2 Layer 1 Default IP route VCAT and VLAN channels Optical path Site A Site B Monitor layout / Setup circuit Monitor host & end-to-end paths / Setup end-host parameters Control transfers and bandwidth reservations Monitor interfaces traffic

40. Iosif Legrand November 2008 40 APPLICATION >FDT A/fileX B/path/ path or channel allocation Configuring interfaces Starting Data Transfer Regular IP path Local VLANs Recommended to use two NICs -one for management /one for data -- bonding two NICs to the same IP MAP Local VLANs to WAN channels or light paths “On-Demand”, L2 Dynamic Channel and Path Allocation

41. Iosif Legrand November 2008 41 The Need for Planning and Scheduling for Large Data Transfers In Parallel Sequential 2.5 X Faster to perform the two reading tasks sequentially

42. Iosif Legrand November 2008 42 User Scheduling Control Monitoring End Host Agents Realtime Feedback Request  Channel allocation based on VO/Priority, [ + Wait time, etc.]  Create on demand a End-to-end path or Channel & configure end-hosts  Automatic recovery (rerouting) in case of errors  Dynamic reallocation of throughputs per channel: to manage priorities, control time to completion, where needed  Reallocate resources requested but not used Dynamic Path Provisioning Queueing and Scheduling

43. Iosif Legrand November 2008 43 Dynamic priority for FDT Transfers on common segments Priority 4 Priority 2 Priority 8

44. Iosif Legrand November 2008 44 Bandwidth Challenge at SC2005 151 Gbs ~ 500 TB Total in 4h

45. Iosif Legrand November 2008 45 FDT & MonLISA Used at SC 2006 April 2007 Iosif Legrand 17.7 Gb/s Disk to Disk on 10 Gb/s link used in on 10 Gb/s link used in Both directions from Florida to Caltech

46. Iosif Legrand November 2008 46 Official BWC Hyper BWC SC2006 April 2007 Iosif Legrand

47. Iosif Legrand November 2008 47 SC2007 +80 Gb/s disk to disk

48. Iosif Legrand November 2008 48 Communities using MonALISA 48 Major Communities  ALICE  CMS  ATLAS  EVO  LGC RUSSIA  UNAM Grid (Mx)  ITU  USLHCNET  ULTRALIGHT  GLORIAD  ABILENE  RoEduNET  Enlightened - - VRVS ALICE USLHCnet EVO OSG MonALISA Today Running 24 X 7 at ~340 Sites  Collecting ~ 1 000 000 parameters in near real-time  Update rate of 20,000 parameter updates per second  Monitoring  12,000 computers  > 100 WAN Links  Thousands of Grid jobs running concurrently http://monalisa.caltech.edu