1. Monitoring Of XRootD Federation
A. Petrosyan, D. Oleynik, J. Andreeva
“Creating federated data stores for the LHC” workshop, , IN2P3, Lyon

2. "Creating federated data stores for the LHC", IN2P3, Lyon
Historical overview
XRootD federation monitoring architecture was presented at Lyon last November, during previous “Creating federated data stores for the LHC” workshop
Motivation
Metrics
Requirements
Technologies
For details refer to
"Creating federated data stores for the LHC", IN2P3, Lyon

3. Components and technologies
Messages from XRootD servers are being collected via UCSD UDP collector and then being sent into AMQ
After being read from AMQ, data is stored in Hbase storage
Hadoop processing to prepare data summaries
Web-services for data export
Dashboard XBrowse transfer interface as UI
"Creating federated data stores for the LHC", IN2P3, Lyon

4. "Creating federated data stores for the LHC", IN2P3, Lyon
Why?
Why UCSD UDP collector?
Feds are ready to install this collector
Provides all necessary functionality
Supported
Why AMQ?
AMQ is a messaging system which is used for the monitoring purposes on the WLCG infrastructure
Is used in production by various applications including SAM
Message brokers supported by CERN IT
Why Hadoop?
Free
Scalable
Provides good data processing capabilities for log-like data
Why XBrowse?
Interface created to show transfers
Current UI of ATLAS DDM monitoring:
Extendable
"Creating federated data stores for the LHC", IN2P3, Lyon

5. Data flow for the XRootD federation monitoring
"Creating federated data stores for the LHC", IN2P3, Lyon

6. UDP messages collector
Implemented by Matevz Tadel
Can be installed anywhere, implemented as Linux daemon
Sends complete transfer message to ActiveMQ
Message includes:
Domain from, host and ip address
Domain to, host and address
User info
File, size
Bytes read/written
Time transfer started/ended
Num of operations
Ect.
"Creating federated data stores for the LHC", IN2P3, Lyon

7. "Creating federated data stores for the LHC", IN2P3, Lyon
AMQ2Hadoop collector
Can be installed anywhere, implemented as Linux daemon
Listens ActiveMQ queue
Extracts messages
Inserts into Hbase raw table
"Creating federated data stores for the LHC", IN2P3, Lyon

8. "Creating federated data stores for the LHC", IN2P3, Lyon
Hadoop processing
Reads raw table
Prepares data summary: 10 min stats as structure:
From To
Sum bytes read
Sum bytes written
Amount files read
Amount files written
Inserts summary data into summary table
MapReduce: we use Java, we also working on enabling Pig routines
"Creating federated data stores for the LHC", IN2P3, Lyon

9. "Creating federated data stores for the LHC", IN2P3, Lyon
Storage2UI data export
Web-service
Extracts data from the storage
Feeds Dashboard XBrowse UI
"Creating federated data stores for the LHC", IN2P3, Lyon

10. "Creating federated data stores for the LHC", IN2P3, Lyon
XBrowse UI
Implemented by David Tuckett
Standalone web application
Can be installed everywhere
Accepts data in JSON format
Presents transfers data as site matrix
"Creating federated data stores for the LHC", IN2P3, Lyon

11. "Creating federated data stores for the LHC", IN2P3, Lyon
"Creating federated data stores for the LHC", IN2P3, Lyon

12. "Creating federated data stores for the LHC", IN2P3, Lyon
Status
Full prototype from UDP collecting to UI using simulated data is implemented in Dubna:
Basic functionality provides source/destination plots and matrix with a possibility to show local access/transfers versus remote, copying versus reading
Now migrating to CERN and real ATLAS and CMS data
Installation of Hadoop cluster at CERN (accomplished)
Setting up AMQ collector and MapReduce processing with real ATLAS and CMS data (in progress)
"Creating federated data stores for the LHC", IN2P3, Lyon

13. "Creating federated data stores for the LHC", IN2P3, Lyon
Nearest plans
Finish migration from JINR to CERN
Move from prototype to production in data processing
Start real data (ATLAS, CMS federations) consuming
Further development will follow the ATLAS and CMS requirements, see presentation of Rob Gardner:
"Creating federated data stores for the LHC", IN2P3, Lyon

14. "Creating federated data stores for the LHC", IN2P3, Lyon
Thanks for attention
"Creating federated data stores for the LHC", IN2P3, Lyon