Monitoring Of XRootD Federation A. Petrosyan, D. Oleynik, J. Andreeva “Creating federated data stores for the LHC” workshop, 13.09.12, IN2P3, Lyon

"Creating federated data stores for the LHC", IN2P3, Lyon Historical overview XRootD federation monitoring architecture was presented at Lyon last November, during previous “Creating federated data stores for the LHC” workshop Motivation Metrics Requirements Technologies For details refer to https://indico.in2p3.fr/getFile.py/access?contribId=14&resId=1&materialId=slides&confId=5527 13.09.2012 "Creating federated data stores for the LHC", IN2P3, Lyon

Components and technologies Messages from XRootD servers are being collected via UCSD UDP collector and then being sent into AMQ After being read from AMQ, data is stored in Hbase storage Hadoop processing to prepare data summaries Web-services for data export Dashboard XBrowse transfer interface as UI 13.09.2012 "Creating federated data stores for the LHC", IN2P3, Lyon

"Creating federated data stores for the LHC", IN2P3, Lyon Why? Why UCSD UDP collector? Feds are ready to install this collector Provides all necessary functionality Supported Why AMQ? AMQ is a messaging system which is used for the monitoring purposes on the WLCG infrastructure Is used in production by various applications including SAM Message brokers supported by CERN IT Why Hadoop? Free Scalable Provides good data processing capabilities for log-like data Why XBrowse? Interface created to show transfers Current UI of ATLAS DDM monitoring: http://dashb-atlas-data.cern.ch/ddm2/ Extendable 13.09.2012 "Creating federated data stores for the LHC", IN2P3, Lyon

Data flow for the XRootD federation monitoring 13.09.2012 "Creating federated data stores for the LHC", IN2P3, Lyon

UDP messages collector Implemented by Matevz Tadel Can be installed anywhere, implemented as Linux daemon Sends complete transfer message to ActiveMQ Message includes: Domain from, host and ip address Domain to, host and address User info File, size Bytes read/written Time transfer started/ended Num of operations Ect. 13.09.2012 "Creating federated data stores for the LHC", IN2P3, Lyon

"Creating federated data stores for the LHC", IN2P3, Lyon AMQ2Hadoop collector Can be installed anywhere, implemented as Linux daemon Listens ActiveMQ queue Extracts messages Inserts into Hbase raw table 13.09.2012 "Creating federated data stores for the LHC", IN2P3, Lyon

"Creating federated data stores for the LHC", IN2P3, Lyon Hadoop processing Reads raw table Prepares data summary: 10 min stats as structure: From To Sum bytes read Sum bytes written Amount files read Amount files written Inserts summary data into summary table MapReduce: we use Java, we also working on enabling Pig routines 13.09.2012 "Creating federated data stores for the LHC", IN2P3, Lyon

"Creating federated data stores for the LHC", IN2P3, Lyon Storage2UI data export Web-service Extracts data from the storage Feeds Dashboard XBrowse UI 13.09.2012 "Creating federated data stores for the LHC", IN2P3, Lyon

"Creating federated data stores for the LHC", IN2P3, Lyon XBrowse UI Implemented by David Tuckett Standalone web application Can be installed everywhere Accepts data in JSON format Presents transfers data as site matrix 13.09.2012 "Creating federated data stores for the LHC", IN2P3, Lyon

"Creating federated data stores for the LHC", IN2P3, Lyon 13.09.2012 "Creating federated data stores for the LHC", IN2P3, Lyon

"Creating federated data stores for the LHC", IN2P3, Lyon Status Full prototype from UDP collecting to UI using simulated data is implemented in Dubna: http://xrdfedmon-dev.jinr.ru/ui/#date.from=201206210000&date.interval=0&date.to=201206220000&grouping.dst=(host)&grouping.src=(host) Basic functionality provides source/destination plots and matrix with a possibility to show local access/transfers versus remote, copying versus reading Now migrating to CERN and real ATLAS and CMS data Installation of Hadoop cluster at CERN (accomplished) Setting up AMQ collector and MapReduce processing with real ATLAS and CMS data (in progress) 13.09.2012 "Creating federated data stores for the LHC", IN2P3, Lyon

"Creating federated data stores for the LHC", IN2P3, Lyon Nearest plans Finish migration from JINR to CERN Move from prototype to production in data processing Start real data (ATLAS, CMS federations) consuming Further development will follow the ATLAS and CMS requirements, see presentation of Rob Gardner: https://twiki.cern.ch/twiki/pub/LCG/WLCGTransferMonitoring/Requirements.pdf 13.09.2012 "Creating federated data stores for the LHC", IN2P3, Lyon

"Creating federated data stores for the LHC", IN2P3, Lyon Thanks for attention 13.09.2012 "Creating federated data stores for the LHC", IN2P3, Lyon