1. Department of Information Engineering The Chinese University of Hong Kong A Framework for Monitoring and Measuring a Large-Scale Distributed System in Real Time By Lei ZHAN A Framework for Monitoring and Measuring a Large-Scale Distributed System in Real Time By Lei ZHAN Aug 16 th, 2013

2. Outline Department of Information Engineering The Chinese University of Hong Kong A Framework for Monitoring and Measuring a Large-Scale Distributed System in Real Time By Lei ZHAN Background Framework Design Case Study Demonstration Future Works 2

3. Background Department of Information Engineering The Chinese University of Hong Kong A Framework for Monitoring and Measuring a Large-Scale Distributed System in Real Time By Lei ZHAN Internet Services on Distributed Infrastructure Content Delivery Network P2P Systems Data Centers Cloud Computing Services Monitoring Framework to guarantee reliable services and high quality of user experience monitor and manage the deployed systems. 3

4. Objectives Department of Information Engineering The Chinese University of Hong Kong A Framework for Monitoring and Measuring a Large-Scale Distributed System in Real Time By Lei ZHAN Accuracy Real-time Visualization Scalability 4

5. Framework Design Department of Information Engineering The Chinese University of Hong Kong A Framework for Monitoring and Measuring a Large-Scale Distributed System in Real Time By Lei ZHAN 5

6. Components – End Hosts Department of Information Engineering The Chinese University of Hong Kong A Framework for Monitoring and Measuring a Large-Scale Distributed System in Real Time By Lei ZHAN Refer to peer in P2P system, processing unit in Cloud, data center in CDN, etc. Deployed in a large-scale and distributed manner Measurement Data Resources unique id for each End Host generate feedback message periodically 6

7. Components – Coordinator Department of Information Engineering The Chinese University of Hong Kong A Framework for Monitoring and Measuring a Large-Scale Distributed System in Real Time By Lei ZHAN Locates between End Host and Feedback Server Responsible for collecting feedback messages from End Hosts forwarding them to Feedback Servers Why Coordinator? unique target for all End Hosts making Feedback Server more flexible 7

8. Components – Feedback Server Department of Information Engineering The Chinese University of Hong Kong A Framework for Monitoring and Measuring a Large-Scale Distributed System in Real Time By Lei ZHAN Locates between Coordinator & Monitoring Platform Responsible for aggregating feedback messages from Coordinator responding data requests from Monitoring Platform 8

9. Components – Monitoring Platform Department of Information Engineering The Chinese University of Hong Kong A Framework for Monitoring and Measuring a Large-Scale Distributed System in Real Time By Lei ZHAN Provides measurement data processing and analysis visualization views of data statistic for administrator Operates in real-time mode: communicate with Feedback Server static mode: read data from local log files 9

10. Framework Design Department of Information Engineering The Chinese University of Hong Kong A Framework for Monitoring and Measuring a Large-Scale Distributed System in Real Time By Lei ZHAN 10 Feedback Messages Request Aggregated Log Files

11. Case Study Department of Information Engineering The Chinese University of Hong Kong A Framework for Monitoring and Measuring a Large-Scale Distributed System in Real Time By Lei ZHAN 2012 London Olympic Games live broadcast through the Internet within HK P2P Video Streaming System developed by ASTRI * adopted by i-Cable ** 11 * The Hong Kong Applied Science and Technology Research Institute (ASTRI) was founded by the Government of Hong Kong SAR in 2000 with a mission to enhance Hong Kong’s competitiveness in technology-based industries through applied research. ** i-Cable is an internet Service Provider in Hong Kong, and is now one of Hong Kong's leading integrated communications companies.

12. Real-time Monitoring Department of Information Engineering The Chinese University of Hong Kong A Framework for Monitoring and Measuring a Large-Scale Distributed System in Real Time By Lei ZHAN Whole Period 17 days (July 27 th – Aug 12 th ) Key Metrics system statistics number of new peers, total number of peers average peer upload rate, average peer download rate average peer contribution ratio system performance peer startup delay, peer continuity quality of experience 12

13. Monitoring Platform Department of Information Engineering The Chinese University of Hong Kong A Framework for Monitoring and Measuring a Large-Scale Distributed System in Real Time By Lei ZHAN Playback in 2 Modes Visualization 4 different views Map View District View Histogram View Timeline View filtering & control More in the Demonstration 13

14. Measurement Results Department of Information Engineering The Chinese University of Hong Kong A Framework for Monitoring and Measuring a Large-Scale Distributed System in Real Time By Lei ZHAN 14

15. Demonstration Department of Information Engineering The Chinese University of Hong Kong A Framework for Monitoring and Measuring a Large-Scale Distributed System in Real Time By Lei ZHAN 15 Monitoring Platform operates in static mode the data of Aug 2 nd, 2012 4 visualization views

16. Discussion (I) Department of Information Engineering The Chinese University of Hong Kong A Framework for Monitoring and Measuring a Large-Scale Distributed System in Real Time By Lei ZHAN Measurement Result window based statistics identify End Host by its id update records upon new feedback message consider latest state inside the window as current state time window moving average method for analysis window size >= feedback message period data synchronized at Feedback Servers avoid synchronization problem of feedback messages 16

17. Discussion (II) Department of Information Engineering The Chinese University of Hong Kong A Framework for Monitoring and Measuring a Large-Scale Distributed System in Real Time By Lei ZHAN Real-time Delay more Feedback Servers log files at Feedback Servers generate more frequently compress before sending to Monitoring Platform Scalability multiple Coordinators more Feedback Servers sampling on feedback messages 17

18. Future Works Department of Information Engineering The Chinese University of Hong Kong A Framework for Monitoring and Measuring a Large-Scale Distributed System in Real Time By Lei ZHAN Generalize for other Systems IP Geo-location Map View & District View IP -> Physical Address wired IP Geo-location 18

19. Q&A Department of Information Engineering The Chinese University of Hong Kong A Framework for Monitoring and Measuring a Large-Scale Distributed System in Real Time By Lei ZHAN 19