1. Local Monitoring Module (LMM) Author: Anna Bekkerman abekkerm@ecs.umass.edu

2. Managing LMM’s Setup When LMM is started the following components are created: – LocalServer and Sender Functionality: Communicate with RAPIDS server – CurrentSetup Functionality: Contains parameters of the current monitoring setup – Metrics, processes, update rate etc. – DBManager Functionality: Concurrent access to the monitoring setup LocalServer LMM Sender CurrentSetup DBManager Sends signals, events Sets up parameters Requests current parameters signals, events commands

3. Managing LMM’s Setup Requirement: provide control over the level of monitoring intrusiveness Solution: dynamic modification of the monitoring setup Design: – During the experiment user sends setup commands – In the beginning of each collection session LMM requests current setup parameters – DBManager handles setup modification commands as well as LMM’s requests Dynamic setup modification has not been implemented in the current version of RAPIDS

4. Start-up Procedure 1. Start network control application (if needed) 2. Initialize RAPIDS Message Queue (RMQ) 3. Start heartbeat application 4. Launch processes

5. Network Control Requirement: simulate packet drop rates and transit delays on links between radar nodes and the SOCC Solution: use iptables to forward packets to user-land application that will delay or drop them SOCC1SOCC2 Radar node Radar node Radar node SOCC

6. Network Control: Implementation Configure iptables to use the QUEUE target SOCC1SOCC2 Radar node Radar node Radar node SOCC … -A INPUT -s 128.119.245.36 -j QUEUE … /etc/sysconfig/iptables on emmy5.casa.umass.edu : emmy5 emmy2 emmy2.casa.umass.edu All packets coming from emmy2 will be queued!

7. Network Control: Implementation Start iptables Load the ip_queue module that forwards packages to the user space > /sbin/service iptables start Applying iptables firewall rules: [ OK ] > /sbin/modprobe ip_queue > /sbin/lsmod Module Size Used by ip_queue 14553 0 … These commands should be executed under root login! In order to unload the ip_queue module do: > /sbin/modprobe –r ip_queue In order to stop iptables do: > /sbin/service iptables stop

8. Network Control: Implementation When LMM is started it will launch the simuwan_usr application that processes forwarded packages Problem: simuwan_usr must be started by root Solution: use sudo utility to start the application – The utility is used to run commands with the root user's privileges

9. How to Set Up the sudo Utility X On the machine X edit the sudoers file (as root ): Specify commands that should be executed under root login: X, Now, while running on X under someLogin, LMM should be able to start/stop simuwan_usr > /usr/sbin/visudo someLogin X = NOPASSWD : /usr/share/rapids/bin/ simuwan_usr, /usr/bin/kill

10. More on simuwan_usr Application Two types of action can be applied to the packets – Delay for either constant or variable amount of time – Drop according to the specified drop rate Packets that are not dropped can be delayed still Uses Glib ’s event loop to process forwarded packets – Each packet is an event – Each event should be assigned a verdict: ACCEPT or REJECT – Each event can be assigned a timeout before it is dispatched

11. RAPIDS Message Queue (RMQ) RMQ employs Unix message queues to store Messaging and Application events Events are generated: – Through wrapped library function calls – Using RAPIDS API RMQManager : – Creates/removes RMQ – Periodically retrieves events and prepares them for sending to the RAPIDS server Application 1 Function call RMQ LMM RMQManager To server

12. Network Monitoring Requirement: monitor status of links between SOCC and radar nodes Solution: send “I’m alive” messages from radars to the SOCC Drawback: false alarms

13. Network Monitoring: Implementation heartbeat_socc application is started on the SOCC node – If there is more than one node in the SOCC, the first one specified in the configuration file is chosen heartbeat_sensor applications are started on the rest of the nodes SOCC periodically pings nodes SOCC1SOCC2 Radar node Radar node SOCC heartbeat_sensor heartbeat_socc

14. Network Monitoring: Implementation If node X replays, SOCC generates Variable event: “X=true” If node X does not replay, SOCC generates Variable event: “X=false” When RAPIDS server receives false event for node X, it reports failure for connection SOCC ↔ X SOCC1SOCC2 Radar node Radar node SOCC heartbeat_sensor heartbeat_socc RMQ LMM RMQManager To server “Variable” event

15. Launching Processes User provides commands to start/stop processes in the configuration file RAPIDS server sends these commands to LMMs while setting them up LMM writes commands to a script – Scripts are created in the home directory and deleted at the end of the experiment – Script name is start_commands / stop_commands followed by the sequence number of the node where the script is executed SOCC nodes have 1-digit sequence numbers: 1, 2, 3 … Sensor nodes have 3-digit sequence numbers: 100, 101, 102 … Starting script is executed in the beginning of the experiment Stopping script is executed when LMM receives stop signal

16. Collection Session: Class Diagram DBManagerSyncBuffer RMQManager CollectionSession void start() CommandProvider vector commands(Bucket *s, Bucket *p, Bucket *e) CommandExecutor void execute() vector commands; BucketProcessBucket SystemBucket EventBucket Sender void send(Bucket *b) executes starts creates

17. Collection Session: Algorithm Create three Bucket s for storing system metrics, process metrics and events Generate commands using CommandProvider – CommandProvider requests current set of monitored metrics from DBManager – Depending on the current setup different set of commands will be generated Bucket virtual int sizeInBytes() = 0 virtual int writeContent(unsigned char *buf, int offset) const = 0 virtual void addMetric()

18. Collection Session: Algorithm Run CommandExecutor – System/process metrics: Each command reads current value of a certain metric – For example: CPU utilization, workload etc. Command writes metric values to a bucket – Events: RMQManager inserts events into a SyncBuffer Special EventCatcher command retrieves events from the buffer and puts them into a bucket Send events/metrics to the RAPIDS server using Sender

19. Commands: Class Diagram Command void store() void start() Bucket *bucket; CPUUsageEventsCatcher MemoryUsage Workload Ps

20. Commands: Implementation CPUUsage – Reads values from /proc/stat MemoryUsage – Reads values from /proc/meminfo Workload – Reads values from /proc/loadavg Ps – Looks through all process subdirectories in /proc – Reads the filename of the process from /proc/[pid]/stat – Stores information about processes whose names were provided in RAPIDS configuration file