efi.uchicago.edu ci.uchicago.edu Sharing Network Resources Ilija Vukotic Computation and Enrico Fermi Institutes University of Chicago Federated Storage Workshop SLAC, April 10, 2014

efi.uchicago.edu ci.uchicago.edu 2 Content Monitoring systems in use by ATLAS Issues we will have to address Proposals – Not presenting solutions, just starting a conversion

efi.uchicago.edu ci.uchicago.edu 3 Glossary Monitoring – Systems testing, visualizing functionality of the federation elements (endpoints, redirectors) – Notification of problems Accounting – Systems storing detailed information on previous transfers – Data mining and visualization tools Network monitoring – Systems measuring, storing, predicting, delivering information on current/historical parameters describing connections between sites. – Cost matrix – for each source-destination pair gives a value best describing expected performance of a transfer.

efi.uchicago.edu ci.uchicago.edu 4 Glossary Production jobs (RECO, PILE) - high CPU, low IO Derivation framework - high CPU, low IO User analysis jobs (more reductions, cut flow, …) - low CPU, high IO Failover - when jobs fail to access input data locally and failover to reading from a remote location Overflow - when jobs are explicitly instructed to use input data from a remote location AGIS – ATLAS Grid Information System – Describes FAX topology – Controls what sites export through FAX – Toggle sites ON/OFF for failover – Sets limits on rates delivered-to / used-from FAX

efi.uchicago.edu ci.uchicago.edu 5 ATLAS Monitoring: Infrastructure Simple cron checking basic functionality of endpoints and redirectors (direct, upstream, downstream, privacy, monitoring) SSB - visualization, history, and reporting problems – a primary source of information on state of FAX infrastructure.

efi.uchicago.edu ci.uchicago.edu 6 ATLAS Monitoring: detailed & summary IO + various Many other automatic tools – monitor & collect endpoint information – Run diagnostics, troubleshooting

efi.uchicago.edu ci.uchicago.edu 7 ATLAS: Accounting Dashboard monitor – Described in detail by A. Beche – Our authoritative source Throughput Rate per file active transfersFinished transfers

efi.uchicago.edu ci.uchicago.edu 8 ATLAS: Accounting Use in Production & Analysis PanDA-specific FAX accounting – FAX failover jobs – FAX “overflow” (re-brokered) accounting (in progress)

efi.uchicago.edu ci.uchicago.edu 9 Network Monitoring Sources perfSONAR – Bandwidth – each 6 hours – pinger – 20 minutes – Latency and packet loss - continuous FTS – Collects rates of transfers for all the files. Source- destination combinations not used during one week are probed with test file transfers. – Averaged over file size and over 15 days o 1 GB FAX cost matrix – Estimate of expected rate (MB/s) between a grid worker node and FAX endpoints using single stream xrdcp – Memory-to-memory transfer – Sampling interval: 30 min

efi.uchicago.edu ci.uchicago.edu 10 FAX cost matrix Data collected between 42 ANALY queues (compute sites) and every FAX endpoint Jobs submitted by HammerCloud Results to ActiveMQ, consumed by SSB with network & throughput measurements (perfSONAR and FTS) HammerCloud FAX redirection REST SSB FAX cost matrix SSB FAX cost matrix FTS

efi.uchicago.edu ci.uchicago.edu 11 FAX cost matrix Results averaged over the last week N.B. – rate observed to the worker node (not site BW)

efi.uchicago.edu ci.uchicago.edu 12 Future More data will come our way, on top of current No big increase in storage space Faster networks will come All the experiments will lean on them: – to reduce number of data replicas – more efficiently use available CPUs – decrease failure rate (failover) More and more end users will use federations as locally (T3s) available storage become insufficient

efi.uchicago.edu ci.uchicago.edu 13 Issues Up to now experiments concentrated on getting infrastructure stable, and being able to scale to the network limits There remain challenges – We need to wisely use network resources o Make applications “network aware” – We need to assure we don’t saturate any site’s links – Come up with way to fair share among experiments? Most of the links are not dedicated and we’ll never be able to account for all the traffic on them, but the more we know better we will be able to use it CMS and ALICE evidently did not need to worry much about it and still made almost no accident. ATLAS has more data and higher IO requirements (for analysis jobs), so has to be more careful.

efi.uchicago.edu ci.uchicago.edu 14 Smart network usage Make applications “network aware” – Grid submitted jobs o Submission system (JEDI) has to know expected network performance for all the combinations of sources and destinations. o It has to internally enforce limits set by the sites. – Tier-3s queues and individual end-users o Can not expect an end-user to know or care about network o When working outside of a framework, make sure that they access the federation from place closest to them o Instrument frameworks to choose the source with highest available bandwidth to the client

efi.uchicago.edu ci.uchicago.edu 15 Fair sharing & avoiding link saturation ideas Limit bandwidth usage by endpoints – Hard way, but simple: limit access with proxy servers o Requires dedicated node(s) for each experiment, expensive for smaller endpoints o Very suboptimal network usage – if one of the VOs is not using the allocated bandwidth other VOs cannot use it. o Not fine grained – Easier way, but implies development: storage service accounts and throttles bandwidth usage by VO o Often traffic flows directly from data servers – Potentially expensive to do – Need VO names at the data servers

efi.uchicago.edu ci.uchicago.edu 16 US usage by VO and service XrootD FTS atlas alice cms

efi.uchicago.edu ci.uchicago.edu 17 Other possibilities Control bandwidth usage centrally – Needs unified site identification – Have sites set what is their allocation per experiment for both input and output – Have accounting from DDM dashboard – Inform VOs when using more than what site allocated – Inform both VOs crossing the limit and also the Site when bandwidth reaches 90% of total allocation – VOs individually decide how they use that information o Stop submissions to site o Move/copy data elsewhere o Warn users, kill jobs

efi.uchicago.edu ci.uchicago.edu 18 Not so urgent… Unify cost measurements, their storage, distribution Measuring between cloud providers and CE We need a way of communicating experiment’s needs Cost of transaction (stress on SE)