DDM FAX Dashboard status and future Luca Magnoni IT/SDC 2 nd June 2014

News  Alexandre left  Thanks for all the work!  New people take over:  Luca Magnoni  Marian Babik  Domenico Giordano (continues Data Popularity)  Cristovao Cordeiro  Collaboration from Dubna and Brunel university 2nd June 2014 DDM FAX dashboard / status update 2

The XRootD Dashboard project  Dashboard project as common ground for monitoring miscellaneous data transfer services  XRootD, FTS, etc.  Same architecture and code /different instances:  ATLAS FAX, CMS AAA, (LHCb) 2nd June 2014 DDM FAX dashboard / status update 3

Architecture (for XRootD ) 2nd June 2014 DDM FAX dashboard / status update 4

FAX dashboard  dashb-atlas-xrootd-transfers.cern.ch  Twiki:   Jira:   Mailing List:  2nd June 2014 DDM FAX dashboard / status update 5

Open Issue / Solution  Issue: Topology resolution  Incorrect accounting for sites on the same domain (e.g. in2p3.fr)  Solution: improve host/domain to site mapping and rely on server-provided site-name  Work already started (Cristovao)  Servers have to report site-name correctly 2nd June 2014 DDM FAX dashboard / status update 6

Open Issue/Solution II  Issue: EOS data missing  EOS data already collected  Statistics generation is the bottleneck  PL/SQL procedure latency proportional to raw data volume  Not matter which statistic method used  Solution: promising results on new storage/processing technology  HDFS/MapReduce  Scale horizontally  ready for other transfer services/protocols  In the framework of the WLCG Analytics project :  Temporary fix:  Try to optimize the PL/SQL procedure, if possible 2nd June 2014 DDM FAX dashboard / status update 7

Open issue/Solution III  Issue: Transfer accounting for for Multi- VO sites  Static mapping between XRootD server/Gled collector/broker topic  Solution: GLED-mix with multi VO filtering in autumn  Others: missing concepts of failures  from XRootD reporting 2nd June 2014 DDM FAX dashboard / status update 8

On data consistency  Comparison between Monalisa and DDM FAX dashboard was done for the last week ( thanks Igor, from Dubna )  very good overall data consistency  (< 1-5% diff)  few exceptions, with major discrepancies  Site not reporting on f-stream  detailed result herehere  Comparison can be easily automated  we are working on a comparison web tool for systematic validation 2nd June 2014 DDM FAX dashboard / status update 9

On the raw data access  No plan to support programmatic way to access raw data  Planning to offer snapshot for data mining  Files  DB dump 2nd June 2014 DDM FAX dashboard / status update 10

Now, Your turn!  What you would like to get from the tool?  Missing data?  If it is on the reporting side, we cannot do much!  If you experience issues, let us know 2nd June 2014 DDM FAX dashboard / status update 11