1. Grid Technology CERN IT Department CH-1211 Geneva 23 Switzerland www.cern.ch/i t DBCF GT IT Monitoring WG Update on the tool hunt & MonALISA monitoring 12 December 2011

2. Grid Technology Agenda MonALISA monitoring Tool hunt –Storage/Analysis –Real Time visualization –Data Mining tools Update on the tool hunt - 2

3. Grid Technology MonALISA monitoring (by Pablo) Slides: http://cern.ch/go/zM67http://cern.ch/go/zM67 Rates –4 proxies (brokers) –6K values per second, peak rate: 20K per second UDP, Lost rate < 5*10 -6 Security –Shared password, firewall-like config on services Flexible with adding new sensors Data storage: –RDBMS (MySQL, PostgreSQL, memory) –Hadoop implementation (results not so good) Update on the tool hunt - 3

4. Grid Technology Storage/Analysis RDBMS –Oracle, MySQL NoSQL –See slides from last meeting: http://cern.ch/go/7Tzghttp://cern.ch/go/7Tzg Columnar Databases –Infobright Community Edition –Vertica –Teradata Update on the tool hunt - 4

5. Grid Technology Display tools Development tools: –Python, Django –GoogleChart, GraphTool (Matplotlib) –AJAX/jQuery –Java (BE) Real-time Monitoring –OpenTSDB –Graphite –Feedzai Data Mining –Pentaho –JasperSoft Update on the tool hunt - 5

6. Grid Technology Real-Time Monitoring Update on the tool hunt - 6

7. Grid Technology OpenTSDB http://opentsdb.net/ Distributed, scalable Time Series Database Written on top of HBase Scales to: –Thousands of machines –Many billions of data points –At http://www.stumbleupon.com/: 150M Datapoints/Day (We need more) Slides and talk: –http://www.youtube.com/watch?v=WlsyqhrhRZAhttp://www.youtube.com/watch?v=WlsyqhrhRZA –http://opentsdb.net/misc/opentsdb-strata.pdfhttp://opentsdb.net/misc/opentsdb-strata.pdf Update on the tool hunt - 7

8. Grid Technology Graphite http://graphite.wikidot.com/ Designed to handled numeric time-series data At Orbitz.com: 230M distinct metrics per day –Scales horizontally on both the frontend and the backend Written in Python, Webapp in Django Update on the tool hunt - 8

9. Grid Technology Feedzai Pulse http://www.feedzai.com/ Computing Seminar@CERN (Feb 2011): –http://indico.cern.ch/conferenceDisplay.py?confId=12380 2http://indico.cern.ch/conferenceDisplay.py?confId=12380 2 –Castor Monitoring Demo Millions of events in real-time Real-time data processing and analytics Predictive Analytics Update on the tool hunt - 9

10. Grid Technology Business Intelligence Tools Update on the tool hunt - 10

11. Grid Technology Pentaho http://www.pentaho.com/ Supports Big Data Platforms (Hadoop, NoSQL): –Webcast: http://cern.ch/go/6kq6http://cern.ch/go/6kq6 Open Source Community Edition: –http://community.pentaho.com/http://community.pentaho.com/ Enterprise Edition offers more functionality but is not for free… Update on the tool hunt - 11

12. Grid Technology Jaspersoft http://www.jaspersoft.com/ Supports Big Data Platforms (Hadoop, NoSQL): –http://www.jaspersoft.com/bigdatahttp://www.jaspersoft.com/bigdata Open Source Community Edition: –http://jasperforge.org/http://jasperforge.org/ Enterprise Edition offers more functionality but is not for free… Update on the tool hunt - 12

13. Grid Technology Summary Your input is welcome Next step: –Tool matching based on the use cases https://tomtools.cern.ch/confluence/display/MWG/U se+Cases Update on the tool hunt - 13