1. DDM FAX Dashboard status and future Luca Magnoni IT/SDC 2 nd June 2014

2. News  Alexandre left  Thanks for all the work!  New people take over:  Luca Magnoni  Marian Babik  Domenico Giordano (continues Data Popularity)  Cristovao Cordeiro  Collaboration from Dubna and Brunel university 2nd June 2014 DDM FAX dashboard / status update 2

3. The XRootD Dashboard project  Dashboard project as common ground for monitoring miscellaneous data transfer services  XRootD, FTS, etc.  Same architecture and code /different instances:  ATLAS FAX, CMS AAA, (LHCb) 2nd June 2014 DDM FAX dashboard / status update 3

4. Architecture (for XRootD ) 2nd June 2014 DDM FAX dashboard / status update 4

5. FAX dashboard  dashb-atlas-xrootd-transfers.cern.ch  Twiki:  https://twiki.cern.ch/twiki/bin/view/LCG/WLCGDataTransferMonitoring https://twiki.cern.ch/twiki/bin/view/LCG/WLCGDataTransferMonitoring  Jira:  https://its.cern.ch/jira/browse/DASHB/component/15024 https://its.cern.ch/jira/browse/DASHB/component/15024  Mailing List:  dashboard-support@cern.ch 2nd June 2014 DDM FAX dashboard / status update 5

6. Open Issue / Solution  Issue: Topology resolution  Incorrect accounting for sites on the same domain (e.g. in2p3.fr)  Solution: improve host/domain to site mapping and rely on server-provided site-name  Work already started (Cristovao)  Servers have to report site-name correctly 2nd June 2014 DDM FAX dashboard / status update 6

7. Open Issue/Solution II  Issue: EOS data missing  EOS data already collected  Statistics generation is the bottleneck  PL/SQL procedure latency proportional to raw data volume  Not matter which statistic method used  Solution: promising results on new storage/processing technology  HDFS/MapReduce  Scale horizontally  ready for other transfer services/protocols  In the framework of the WLCG Analytics project : https://twiki.cern.ch/twiki/bin/view/LCG/WLCGMonDataAnalytics https://twiki.cern.ch/twiki/bin/view/LCG/WLCGMonDataAnalytics  Temporary fix:  Try to optimize the PL/SQL procedure, if possible 2nd June 2014 DDM FAX dashboard / status update 7

8. Open issue/Solution III  Issue: Transfer accounting for for Multi- VO sites  Static mapping between XRootD server/Gled collector/broker topic  Solution: GLED-mix with multi VO filtering in autumn  Others: missing concepts of failures  from XRootD reporting 2nd June 2014 DDM FAX dashboard / status update 8

9. On data consistency  Comparison between Monalisa and DDM FAX dashboard was done for the last week ( thanks Igor, from Dubna )  very good overall data consistency  (< 1-5% diff)  few exceptions, with major discrepancies  Site not reporting on f-stream  detailed result herehere  Comparison can be easily automated  we are working on a comparison web tool for systematic validation 2nd June 2014 DDM FAX dashboard / status update 9

10. On the raw data access  No plan to support programmatic way to access raw data  Planning to offer snapshot for data mining  Files  DB dump 2nd June 2014 DDM FAX dashboard / status update 10

11. Now, Your turn!  What you would like to get from the tool?  Missing data?  If it is on the reporting side, we cannot do much!  If you experience issues, let us know 2nd June 2014 DDM FAX dashboard / status update 11