Ramping up the LCG Service The LCG Service Challenges: Ramping up the LCG Service Jamie Shiers, CERN-IT-GD-SC April 2005 LCG.

Ramping up the LCG Service The LCG Service Challenges: Ramping up the LCG Service Jamie Shiers, CERN-IT-GD-SC April 2005 LCG

LCG Service Challenges – Deploying the Service Agenda  Goals and Timelines of the LCG Service Challenges  Review of SC1 and SC2  Summary of LHC Experiments’ Computing Models  Outline of SC3 and SC4 FULL PRODUCTION SERVICE!  After that it’s the FULL PRODUCTION SERVICE!  Plans for involving Tier2 sites in the Service Challenges  Detailed SC3 planning

LCG Service Challenges – Deploying the Service Acknowledgements / Disclaimer  These slides are the work of many people  LCG Storage Management Workshop, Service Challenge Meetings etc  Plus relevant presentations I found with Google!  The work behind the slides is that of many more…  All those involved in the Service Challenges at the many sites…  I will use “LCG” in the most generic sense…  Because that’s what I know / understand… (?)

LCG Service Challenges – Deploying the Service Partial Contribution List  James Casey: SC1/2 review  Jean-Philippe Baud: DPM and LFC  Sophie Lemaitre: LFC deployment  Peter Kunszt: FiReMan  Michael Ernst: dCache  gLite FTS: Gavin McCance  ALICE: Latchezar Betev  ATLAS: Miguel Branco  CMS: Lassi Tuura  LHCb: atsareg@in2p3.fr

LCG Service Challenges – Deploying the Service LCG Service Challenges - Overview  LHC will enter production (physics) in April 2007  Will generate an enormous volume of data  Will require huge amount of processing power  LCG ‘solution’ is a world-wide Grid  Many components understood, deployed, tested..  But…  Unprecedented scale  Humungous challenge of getting large numbers of institutes and individuals, all with existing, sometimes conflicting commitments, to work together  LCG must be ready at full production capacity, functionality and reliability in less than 2 years from now  Issues include h/w acquisition, personnel hiring and training, vendor rollout schedules etc.  Should not limit ability of physicist to exploit performance of detectors nor LHC’s physics potential  Whilst being stable, reliable and easy to use

LCG Service Challenges – Deploying the Service LCG Deployment Schedule

LCG Service Challenges – Deploying the Service Service Challenges  Purpose  Understand what it takes to operate a real grid service – run for days/weeks at a time (outside of experiment Data Challenges)  Trigger/encourage the Tier1 & large Tier-2 planning – move towards real resource planning – based on realistic usage patterns  Get the essential grid services ramped up to target levels of reliability, availability, scalability, end-to-end performance  Set out milestones needed to achieve goals during the service challenges  NB: This is focussed on Tier 0 – Tier 1/large Tier 2  Data management, batch production and analysis  Short term goal – by end 2004 – have in place a robust and reliable data management service and support infrastructure and robust batch job submission Ian Bird – ian.bird@cern.ch From early proposal, May 2004

LCG Service Challenges – Deploying the Service Why Service Challenges? To test Tier-0  Tier-1  Tier-2 services  Network service  Sufficient bandwidth: ~10 Gbit/sec  Backup path  Quality of service: security, help desk, error reporting, bug fixing,..  Robust file transfer service  File servers  File Transfer Software (GridFTP)  Data Management software (SRM, dCache)  Archiving service: tapeservers,taperobots, tapes, tapedrives,..  Sustainability  Weeks in a row un-interrupted 24/7 operation  Manpower implications: ~7 fte/site  Quality of service: helpdesk, error reporting, bug fixing,..  Towards a stable production environment for experiments Kors Bos – Presentation to LHCC, March 7 2005

LCG Service Challenges – Deploying the Service Whither Service Challenges?  First discussions: GDB May - June 2004  May 18 - Lessons from Data Challenges and planning for the next steps (+ Discussion) (1h10') ( transparencies )transparencies  June 15 - Progress with the service plan team (10') ( document )document  Other discussions: PEB June 2004  June 8 - Service challenges - proposal (40') ( transparencies )transparencies  June 29 - Service challenges - status and further reactions (30') ( transparencies )transparencies  May 2004 HEPiX  LCG Service Challenges Slides from Ian Bird (CERN)LCG Service Challenges  My involvement: from January 2005  Current Milestones: http://lcg.web.cern.ch/LCG/PEB/Planning/deployment/Grid%20Deploy ment%20Schedule.htm http://lcg.web.cern.ch/LCG/PEB/Planning/deployment/Grid%20Deploy ment%20Schedule.htm

LCG Service Challenges – Deploying the Service Key Principles  Service challenges result in a series of services that exist in parallel with baseline production service  Rapidly and successively approach production needs of LHC  Initial focus: core (data management) services  Swiftly expand out to cover full spectrum of production and analysis chain  Must be as realistic as possible, including end-end testing of key experiment use-cases over extended periods with recovery from glitches and longer-term outages  Necessary resources and commitment pre-requisite to success!  Effort should not be under-estimated!

LCG Service Challenges – Deploying the Service SC1 Review  SC1 did not complete its goals successfully  Dec04 - Service Challenge I complete  mass store (disk) - mass store (disk)  3 T1s (Lyon, Amsterdam, Chicago) (others also participated…)  500 MB/sec (individually and aggregate)  2 weeks sustained  Software; GridFTP plus some scripts  We did not meet the milestone of 500MB/s for 2 weeks  We need to do these challenges to see what actually goes wrong  A lot of things do, and did, go wrong  We need better test plans for validating the infrastructure before the challenges (network throughput, disk speeds, etc…)  OK, so we’re off to a great start with the Service Challenges…

LCG Storage Workshop “Service Challenge 2 Review” James Casey, IT-GD, CERN CERN, 5th April 2005

LCG Service Challenges – Deploying the Service Overview  Reminder of  Targets for the Service Challenge  Plan and timescales  CERN Tier-0 Configuration  Tier-1 Configuration  Throughput phase of Service Challenge 2  Transfer Software  Monitoring  Outages and problems encountered  SC2 single site tests

LCG Service Challenges – Deploying the Service SC2 - Overview  “Service Challenge 2”  Throughput test from Tier-0 to Tier-1 sites  Started 14 th March  Set up Infrastructure to 7 Sites  NIKHEF/SARA, IN2P3, FNAL, BNL, FZK, INFN, RAL  100MB/s to each site  500MB/s combined to all sites at same time  500MB/s to a few sites individually  Goal : by end March, sustained 500 MB/s at CERN

LCG Service Challenges – Deploying the Service SC2 met its throughput targets  >600MB/s daily average for 10 days was achieved: Midday 23 rd March to Midday 2 nd April  Not without outages, but system showed it could recover rate again from outages  Load reasonable evenly divided over sites (give network bandwidth constraints of Tier-1 sites)

LCG Service Challenges – Deploying the Service Division of Data between sites SiteAverage throughput (MB/s)Data Moved (TB) BNL6151 FNAL6151 GridKA133109 IN2P39175 INFN8167 RAL7258 SARA10688 TOTAL600500

LCG Service Challenges – Deploying the Service CERN Tier-0 Configuration (1/3)  Service Challenge 1 was on an experimental hardware setup  10 HP IA64 nodes within the opencluster  Not a standard service  No LEMON monitoring  No operator coverage  No standard configuration  Plan was to move to a “standard” configuration  IA32 “worker nodes” running CASTOR SRM/gridftp  Still serving data from local disks  Also to move to “production” networking  The opencluster is a test and research facility  Again, non-monitored network hardware

LCG Service Challenges – Deploying the Service CERN Tier-0 Configuration (2/3)  This didn’t all go to plan…  First set of 20 IA32 worker nodes weren’t connected to right network segments  They were installed, configured and tested before this was discovered  Replacement nodes (20 IA32 disk servers) were supplied too late to get installed, configured and tested before SC2  Had to fallback to openlab IA64 nodes for SC2  These weren’t connected to the production network on their inward facing interface (towards CERN LAN)  Added 10 extra IA64 nodes to system – 16 in total were used for data transfer in SC2

LCG Service Challenges – Deploying the Service CERN Tier-0 Configuration (3/3)

LCG Service Challenges – Deploying the Service Tier-1 Storage Configuration  Small set of storage configurations  Most sites ran “vanilla” Globus gridftp servers  SARA, INFN, IN2P3, FZK  The rest of the sites ran dCache  FNAL, BNL, RAL  Most sites used local or system-attached disk  FZK used SAN via GPFS  FNAL used production CMS dCache, including tape  Load-balancing for most plain gridftp sites was done at the RADIANT layer  INFN deployed “n-1” DNS alias – highest loaded machine was replaced in alias every 10 minutes  Alleviated problems see with pure round-robin on gridftp servers

LCG Service Challenges – Deploying the Service Tier-1 Network Connectivity  Sites are in the middle of their upgrade path to LCG networking  10Gb will be the norm – now it’s more like 1Gb  All this is heavily tied to work done in the LCG T0-T1 networking group  Most sites were able to provided dedicated network links  Or at least links where we could get most of the supplied bandwidth  IN2P3, BNL still were on shared links with a bit more congestion  Needed to be dealt with differently  Upped the number of concurrent TCP streams per file transfer

LCG Service Challenges – Deploying the Service Tier-1 Network Topology

LCG Service Challenges – Deploying the Service Transfer Control Software (1/3)  LCG RADIANT Software used to control transfers  Prototype software interoperable with the gLite FTS  Plan is to move to gLite FTS for SC3  Initial promising results presented at Lyon SC Meeting in Feb.  More details in LCG Storage Workshop tomorrow  Run on a single node at CERN, controlling all Tier-0 to Tier-1 “channels”  Transfers done via 3rd-party gridftp  ‘radiant-load-generator’ was used to generate transfers  Configured for each channel to load-balance where appropriate  Specifies number of concurrent streams for a file transfer  This was normally =1 for a dedicated link  Ran from cron to keep transfer queues full

LCG Service Challenges – Deploying the Service Transfer Control Software (2/3)  Control of load on a channel via number of concurrent transfers  Final Configuration: #Chan : State : Last Active :Bwidth: Files: From : To FZK :Active :05/03/29 15:23:47 :10240 :5 :cern.ch :gridka.de IN2P35:Active :05/03/29 15:16:32 :204 :1 :cern.ch :ccxfer05.in2p3.fr IN2P33:Active :05/03/29 15:20:30 :204 :1 :cern.ch :ccxfer03.in2p3.fr INFN :Active :05/03/29 15:23:07 :1024 :8 :cern.ch :cr.cnaf.infn.it IN2P32:Active :05/03/29 15:21:46 :204 :1 :cern.ch :ccxfer02.in2p3.fr FNAL :Inactiv:Unknown :10240 :0 :cern.ch :fnal.gov IN2P3 :Active :05/03/29 15:23:40 :204 :1 :cern.ch :ccxfer01.in2p3.fr IN2P34:Active :05/03/29 15:18:00 :204 :1 :cern.ch :ccxfer04.in2p3.fr BNL :Inactiv:Unknown :622 :24 :cern.ch :bnl.gov NL :Active :05/03/29 15:22:54 :3072 :10 :cern.ch :tier1.sara.nl RAL :Active :05/03/29 15:23:09 :2048 :12 :cern.ch :gridpp.rl.ac.uk

LCG Service Challenges – Deploying the Service Transfer Control Software (3/3)  RADIANT controlled transfers were pure gridftp only  SRM components did not get developed in time  FNAL and BNL did SRM transfers by pulling from their end  FNAL used PhEDEx to pull data  BNL used simple driver scripts around srmCopy command  BNL started some radiant-controlled gridftp transfers too  Used the exact transfer nodes as the rest of the sites (radiantservice.cern.ch)  SRM interactions helped debug issues with dCache SRM and DNS load-balanced SRM/gridftp servers  Fixed java DNS alias problems seen at FNAL  Did not work smoothly for most of the challenge with FNAL mode of operation  Main problem solved and fix in place by end of throughtput phase  How to do srmCopy’s with 1000s of files is still not well understood

LCG Service Challenges – Deploying the Service Monitoring @ CERN  MRTG Graphs of network usage out of cluster  LEMON monitoring of cluster  CPU usage  Disk usage  Network usage  Gridftp logfile monitoring

LCG Service Challenges – Deploying the Service Tier-1 Monitoring  Gridftp Monitoring (http://challenge.matrix.sara.nl/SC/)  Main monitoring tool used during the SC2  Hourly throughput per site  Hourly throughput per host  Daily throughput per site  Daily throughput per host

LCG Service Challenges – Deploying the Service Service Outages (1/2)  Progress page kept in SC wiki of  All tunings made to the system  All outages noted during the running  Any actions needed to cleanup and recover service  No real 24x7 service in place  Manual monitoring of monitoring webpages  Best-effort restart of service  Also at Tier-1 sites – problems communicated to service challenge teams, but this was not a 24x7 coverage

LCG Service Challenges – Deploying the Service Service Outages (2/2)  Capacity in the cluster meant that we could recover from one site not being active  Other sites would up their load a bit automatically due to gridftp stream rates increasing  Only thing that killed transfers were CERN outages  We did not do any scheduled outages during the SC  No procedures for starting a schedule outage  If we had done one to move to managed network infrastructure, it would have removed some of the scheduled ones

LCG Service Challenges – Deploying the Service Outage Breakdown - CERN  Myproxy instability  Locks up under high load  Understood by developers of myproxy  Can be handled by watchdog job  Particular problem on restart after network outage  CERN LAN Network outages  Had 2 long-ish outages (~12 hours)  Issue with being on non-managed network hardware  Database quota limit hit  Tablespace was being monitored but not quota  Quota monitoring added  Database load problems  Caused intermittent drops in throughput due to new jobs not being scheduled  In-memory job queues in the transfer agents meant these we’re a big problem

LCG Service Challenges – Deploying the Service DNS load-balancing of gridftpd  An issue raised at Lyon SC Meeting  “load-balanced” gridftp servers  Not really load-balanced with round-robin – no awareness of state or behaviour of other nodes  Can end up with all transfers going to one host, with all others idle  CNAF put in place “worst one out” load-balancing  Heaviest loaded node is taken out of alias every 10 minutes  Smoothed out load

LCG Service Challenges – Deploying the Service Individual site tests  Overlapped with LCG Storage Management Workshop  Sites can pick days in next two weeks when they have the capacity  500MB/s to disk  60MB/s to tape  FNAL was running 500MB/s disk tests at the time…

LCG Service Challenges – Deploying the Service FNAL Failover it shows  that when the starlight link was cut, our transfers fell back to our ESNET connection,  when our network folks rerouted it to our 1 GE link  when starlight came back.

LCG Service Challenges – Deploying the Service SC2 Summary  SC2 met it’s throughput goals – and with more sites than originally planned!  A big improvement from SC1  But we still don’t have something we can call a service  Monitoring is better  We see outages when they happen, and we understood why they happen  First step towards operations guides  Some advances in infrastructure and software will happen before SC3  gLite transfer software  SRM service more widely deployed  We have to understand how to incorporate these elements

LCG Service Challenges – Deploying the Service SC1/2 - Conclusions  Setting up the infrastructure and achieving reliable transfers, even at much lower data rates than needed for LHC, is complex and requires a lot of technical work + coordination  Even within one site – people are working very hard & are stressed. Stressed people do not work at their best. Far from clear how this scales to SC3/SC4, let alone to LHC production phase  Compound this with the multi-site / multi-partner issue, together with time zones etc and you have a large “non-technical” component to an already tough problem (example of technical problem follows…)  But… the end point is fixed (time + functionality)  We should be careful not to over-complicate the problem or potential solutions  And not forget there is still a humungous amount to do…  (much much more than we’ve done…)

LCG Service Challenges – Deploying the Service Computing Model Summary - Goals  Present key features of LHC experiments’ Computing Models in a consistent manner  High-light the commonality  Emphasize the key differences  Define these ‘parameters’ in a central place (LCG web)  Update with change-log as required  Use these parameters as input to requirements for Service Challenges  To enable partners (T0/T1 sites, experiments, network providers) to have a clear understanding of what is required of them  Define precise terms and ‘factors’

LHC Computing Models Summary of Key Characteristics of LHC Computing Models

LCG Service Challenges – Deploying the Service Where do these numbers come from?  Obtained from LHC Computing Models as reviewed in January  Part of plan is to understand how sensitive overall model is to variations in key parameters  Iteration with experiments is on-going  i.e. I have tried to clarify any questions that I have had  Any mis-representation or mis-interpretation is entirely my responsibility  Sanity check: compare with numbers from MoU Task Force  (Actually the following LCG document now uses these numbers!) http://cern.ch/LCG/documents/LHC_Computing_Resources_report.pdf

LCG Service Challenges – Deploying the Service NominalThese are the raw figures produced by multiplying e.g. event size x trigger rate. HeadroomA factor of 1.5 that is applied to cater for peak rates. EfficiencyA factor of 2 to ensure networks run at less than 50% load. RecoveryA factor of 2 to ensure that backlogs can be cleared within 24 – 48 hours and to allow the load from a failed Tier1 to be switched over to others. Total Requirement A factor of 6 must be applied to the nominal values to obtain the bandwidth that must be provisioned. Arguably this is an over-estimate, as “Recovery” and “Peak load” conditions are presumably relatively infrequent, and can also be smoothed out using appropriately sized transfer buffers. But as there may be under-estimates elsewhere…

All numbers presented will be nominal unless explicitly specified

LCG Service Challenges – Deploying the Service LHC Parameters (Computing Models) Yearpp operationsHeavy Ion operations Beam time (seconds/year) Luminosity (cm -2 s -1 ) Beam time (seconds/year) Luminosity (cm -2 s -1 ) 20075 x 10 6 5 x 10 32 -- 2008(1.8 x) 10 7 2 x 10 33 (2.6 x) 10 6 5 x 10 26 200910 7 2 x 10 33 10 6 5 x 10 26 201010 7 10 34 10 6 5 x 10 26 (Real time given in brackets above)

LCG Service Challenges – Deploying the Service LHC Schedule – “Chamonix” workshop  First collisions: two months after first turn on in August 2007  32 weeks of operation, 16 weeks of shutdown, 4 weeks commissioning = 140 days physics / year (5 lunar months)

LCG Service Challenges – Deploying the Service Overview of pp running ExperimentSIMSIMESDRAWTriggerRECOAODTAG ALICE400KB40KB1MB100Hz200KB50KB10KB ATLAS2MB500KB1.6MB200Hz500KB100KB1KB CMS2MB400KB1.5MB150Hz250KB50KB10KB LHCb400KB25KB2KHz75KB25KB1KB

LCG Service Challenges – Deploying the Service pp questions / uncertainties  Trigger rates essentially independent of luminosity  Explicitly stated in both ATLAS and CMS CM docs  Uncertainty (at least in my mind) on issues such as zero suppression, compaction etc of raw data sizes  Discussion of these factors in CMS CM doc p22:  RAW data size ~300kB (Estimated from MC)  Multiplicative factors drawn from CDF experience  MC Underestimation factor 1.6  HLT Inflation of RAW Data, factor 1.25  Startup, thresholds, zero suppression,…. Factor 2.5  Real initial event size more like 1.5MB  Could be anywhere between 1 and 2 MB  Hard to deduce when the even size will fall and how that will be compensated by increasing Luminosity  i.e. total factor = 5 for CMS raw data  N.B. must consider not only Data Type (e.g. result of Reconstruction) but also how it is used  e.g. compare how Data Types are used in LHCb compared to CMS  All this must be plugged into the meta-model!

LCG Service Challenges – Deploying the Service Overview of Heavy Ion running ExperimentSIMSIMESDRAWTriggerRECOAODTAG ALICE300MB2.1MB12.5MB100Hz2.5MB250KB10KB ATLAS 5MB50Hz CMS7MB50Hz1MB200KBTBD LHCbN/A

LCG Service Challenges – Deploying the Service Heavy Ion Questions / Uncertainties  Heavy Ion computing models less well established than for pp running  I was concerned about model for 1 st /2 nd /3 rd pass reconstruction and data distribution  “We therefore require that these data (Pb-Pb) are reconstructed at the CERN T0 and exported over a four-month period after data taking. This should leave enough time for a second and third reconstruction pass at the Tier 1’s” (ALICE)  Heavy Ion model has major impact on those Tier1’s supporting these experiments  All bar LHCb!  These issues have since been clarified:  Raw data export will be spread over shutdown;  First pass reconstruction should complete at least 6 months prior to following year’s AA data taking (ALICE) or during shutdown (CMS);  2 nd pass (ALICE) will not involve the full data sample;  3 rd pass is a complete pass which should complete prior to following year’s AA run  Implies data out of CERN roughly constant; will vary per T1 (pp/AA/shutdown) depending on experiments supported

LCG Service Challenges – Deploying the Service Data Rates from MoU Task Force MB/SecRALFNALBNLFZKIN2P3CNAFPICT0 Total ATLAS106.870.00173.53106.87 707.87 CMS69.29 0.0069.29 415.71 ALICE0.00 135.21 0.00405.63 LHCb6.330.00 6.33 31.67 T1 Totals MB/sec182.4969.29173.53317.69 182.491560.87 T1 Totals Gb/sec1.460.551.392.54 1.4612.49 Estimated T1 Bandwidth Needed (Totals * 1.5(headroom))*2(capacity)4.381.664.167.62 4.3837.46 Assumed Bandwidth Provisioned10.00 70.00 http://cern.ch/LCG/MoU%20meeting%20March%2010/Report_to_the_MoU_Task_Force.doc

LCG Service Challenges – Deploying the Service Tier1 Sites CentreALICEATLASCMSLHCbTarget Data Rate MBytes/sec ASCCXX110 CNAFXXXX220 PICXXX200 CC-IN2P3XXXX220 FZKXXXX220 RALXXXX220 BNLX154 FNALX50 TRIUMFX65 NIKHEFXXX175 NORDIC DATA GRID FACILITYXXX90 Target data rate at CERN1,600 Target data rates calculated from raw computing model numbers during pp running  No (in)efficiency factors, no overhead, etc.  Assume each T1 takes equal fraction (except BNL)  Balanced by fraction of resources allocated per experiment?

LCG Service Challenges – Deploying the Service Heavy Ion Data Rates  ATLAS / CMS data rates limited by online system  LHCb does not participate in Heavy Ion programme  Current model is that data is distributed during shutdown, rather than inline with data taking

LCG Service Challenges – Deploying the Service pp / AA data rates (equal split) CentreALICEATLASCMSLHCbRate into T1Rate into T1 (AA) ASCC, Taipei0110118.728.2 CNAF, Italy1111205.097.2 PIC, Spain0111179.028.2 IN2P3, Lyon1111205.097.2 GridKA, Germany1111205.097.2 RAL, UK1111205.097.2 BNL, USA010072.211.3 FNAL, USA001046.516.9 TRIUMF, Canada010072.211.3 NIKHEF/SARA, Netherlands1101158.580.3 Nordic Centre110098.280.3 Totals61076

LCG Service Challenges – Deploying the Service Streaming  All experiments foresee RAW data streaming, but with different approaches:  CMS: O(50) streams based on trigger path  Classification is immutable, defined by L1+HLT  Atlas: 4 streams based on event types  Primary physics, Express line, Calibration, Debugging and diagnostic  LHCb: >4 streams based on trigger category  B-exclusive, Di-muon, D* Sample, B-inclusive  Streams are not created in the first pass, but during the “stripping” process  Not clear what is the best/right solution. Probably bound to evolve in time. Francesco Forti, Pisa

LCG Service Challenges – Deploying the Service Reprocessing  Data need to be reprocessed several times because of:  Improved software  More accurate calibration and alignment  Reprocessing mainly at T1 centers  LHCb is planning on using the T0 during the shutdown – not obvious it is available  Number of passes per year  But experience shows the reprocessing requires huge effort!  Use these numbers in the calculation but 2 / year will be good going! AliceAtlasCMSLHCb 3224 Francesco Forti, Pisa

LCG Service Challenges – Deploying the Service Base Requirements for T1s  Provisioned bandwidth comes in units of 10Gbits/sec although this is an evolving parameter  From Reply to Questions from Computing MoU Task Force…  Since then, some parameters of the Computing Models have changed  Given the above quantisation, relatively insensitive to small-ish changes  Important to understand implications of multiple-10Gbit links, particularly for sites with Heavy Ion programme  Spread of AA distribution during shutdown probably means 1 link sufficient…  For now, planning for 10Gbit links to all Tier1s

LCG Service Challenges – Deploying the Service Tier1 SUMMARY  High data rate transfer tests Karlsruhe to CERN already underway  10G available now from Bologna to CERN  Testing of 10G lambdas from Lyon and Amsterdam can commence from July 2005  Amsterdam (and Taipei) will use Netherlight link to CERN until GÉANT2 paths are available  Testing of Barcelona link at 10G from October 2005  Nordic distributed facility restricted to 1G until late-2006 when 10G available  RAL could operate between 2 and 4 x 1GE (subject to scheduling and NetherLight/CERN agreement) until late-2006 when 10G available. Interconnection of UKLight to GEANT2 might make an earlier transition to higher capacities possible. Tier0/Tier1/Tier2 NREN Connectivity Overview John Chevers, Project Manager, DANTE Amsterdam 8 th April 2005

LCG Service Challenges – Deploying the Service Data Rate / Site - Conclusions  It is clear that these are only estimates  Experiments will almost certainly split data into streams which will not be of equal size  The share of resources that each T1 provides to a given experiment will also influence the amount of data that is sent there  But it is impossible (and not relevant?) to do a precise calculation  Which in any case becomes further clouded when the reprocessing and analysis is folded in…

LCG Service Challenges – Deploying the Service Dedicated connections for SCs Tier1LocationNRENsStatus dedicated link ASCCTaipei, TaiwanASnet, SURFnet1 Gb via SURFnet, testing BNLUpton, NY, USAESnet, LHCnet622 Mbit shared CNAFBologna, ItalyGeant2, GARR1 Gb now, 10 Gb in Sept FNALBatavia, ILL, USAESnet, LHCnet10 Gb, tested IN2P3Lyon, FranceRenater1 Gb now, 10 Gb in Sept GridKaKarlsruhe, GermanyGeant2, DFN10 Gb, tested SARAAmsterdam, NLGeant2, SURFnet10 Gb, testing NorduGridScandinaviaGeant2, NordunetNot participating yet PICBarcelona, SpainRedIris, Geant2Not participating yet RALDidcot, UKGeant2, Ukerna2 x 1 Gb via SURFnet soon TriumfVancouver, CanadaCanet, LHCnet1 Gb via SURFnet, testing Kors Bos, LCG-LHCC Referees Meeting, March 2005

LCG Service Challenges – Deploying the Service T1/T2 Roles Tier1  Keep certain portions of RAW, ESD, sim ESD  Full copies of AOD + TAG, calibration data  Official physics group large scale data analysis  ALICE + LHCb:  also contribute to simulation Tier2  Keep certain portions of AOD and full copies of TAG for real + simulated data  LHCb: sim only at T2s  Selected ESD samples  Produce simulated data  General end-user analysis Based on “T1 Services for T2 Centres” document (Just type this into Google)

LCG Service Challenges – Deploying the Service MC Data UnitsALICEATLASCMSLHCb p-pPb-Pbp-p Time to reconstruct 1 eventkSI2k sec5.467515252.4 Time to simulate 1 eventkSI2k sec35150001004550 ParameterUnitALICEATLASCMSLHCb p-pPb-Pb Events/yearGiga10.121.520 Events SIM/yearGiga10.010.41.54 Ratio SIM/data% 100%10%20%100%20% Tier2 sites offer 10 – 1000 kSI2K years ATLAS: 16MSI2K years over ~30 sites in 2008 CMS: 20MSI2K years over ~20 sites in 2008

LCG Service Challenges – Deploying the Service GridPP Estimates of T2 Networking Number of T1s Number of T2s Total T2 CPU Total T2 Disk Average T2 CPU Average T2 DiskNetwork In Network Out KSI2KTBKSI2KTBGb/s ALICE6211370026006521240.0100.600 ATLAS10301620069005402300.1400.034 CMS6 to 10252072554508292181.0000.100 LHCb61476002354320.008 The CMS figure of 1Gb/s into a T2 comes from the following: Each T2 has ~10% of current RECO data and 1/2 AOD (real+MC sample) These data are refreshed every 3 weeks compatible with frequency of (possible) major selection pass at T1s See CMS Computing Model S-30 for more details

LCG Service Challenges – Deploying the Service ALICE Computing Needs From as posted 25 Feb. 2005 Table 2.6T0Sum T1sSum T2sTotal CPU (MSI2K) [Peak]7.513.813.735 Transient Storage (PB)0.447.62.510.54 Permanent storage (PB/year)2.37.509.8 Bandwidth in (Gbps)820.075 Bandwidth out (Gbps)61.50.27

Service Challenge 3 Goals and Timeline for Service Challenge 3

LCG Service Challenges – Deploying the Service Service Challenge 3 - Phases High level view:  Setup phase (includes Throughput Test)  2 weeks sustained in July 2005  “Obvious target” – GDB of July 20 th  Primary goals:  150MB/s disk – disk to Tier1s;  60MB/s disk (T0) – tape (T1s)  Secondary goals:  Include a few named T2 sites (T2 -> T1 transfers)  Encourage remaining T1s to start disk – disk transfers  Service phase  September – end 2005  Start with ALICE & CMS, add ATLAS and LHCb October/November  All offline use cases except for analysis  More components: WMS, VOMS, catalogs, experiment-specific solutions  Implies production setup (CE, SE, …)

LCG Service Challenges – Deploying the Service SC3 – Production Services  SC3 is a relatively small step wrt SC2 in terms of throughput!  We know we can do it technology-wise, but do we have a solution that will scale?  Let’s make it a priority for the coming months to streamline our operations  And not just throw resources at the problem…  which we don’t have…  Whilst not forgetting ‘real’ goals of SC3… i.e. services!

LCG Service Challenges – Deploying the Service SC3 – Service Phase “all offline Use Cases except for analysis”  It sounds easy: “all offline Use Cases except for analysis”  And it some senses it is: these are well understood and tested  So its clear what we have to do:  Work with the experiments to understand and agree on the experiment-specific solutions that need to be deployed  Agree on a realistic and achievable work-plan that is consistent with overall goals / constraints  Either that or send a ’droid looking for Obi-Wan Kenobi…

LCG Service Challenges – Deploying the Service Service Phase - Priorities  Experiments have repeatedly told us to focus on reliability and functionality  This we need to demonstrate as a first step…  But cannot lose sight of need to pump up data rates – whilst maintaining production service – to pretty impressive “DC” figures

LCG Service Challenges – Deploying the Service SC3 Preparation Workshop  This (proposed) workshop will focus on very detailed technical planning for the whole SC3 exercise.  It is intended to be as interactive as possible, i.e. not presentations to an audience largely in a different (wireless) world.  There will be sessions devoted to specific experiment issues, Tier1 issues, Tier2 issues as well as the general service infrastructure.  Planning for SC3 has already started and will continue prior to the workshop.  This is an opportunity to get together to iron out concerns and issues that cannot easily be solved by e-mail, phone conferences and/or other meetings prior to the workshop.

LCG Service Challenges – Deploying the Service SC3 Preparation W/S Agenda  4 x 1/2 days devoted to experiments  in B160 1-009, phone conferencing possible  1 day focussing on T1/T2 issues together with output of above  In 513 1-024, VRVS available  Dates are 13 – 15 June (Monday – Wednesday)  Even though conference room booked tentatively in February, little flexibility in dates even then!

LCG Service Challenges – Deploying the Service SC3 on  SC3 is significantly more complex than previous challenges  It includes experiments s/w, additional m/w, Tier2s etc  Proving we can transfer dummy files from A-B proves nothing  Obviously need to show that basic infrastructure works…  Preparation for SC3 includes:  Understanding experiments’ Computing Models  Agreeing involvement of experiments’ production teams  Visiting all (involved) Tier1s (multiple times)  Preparing for the involvement of 50-100 Tier2s  Short of resources at all levels:  “Managerial” – discussing with experiments and Tier1s (visiting)  “Organizational” – milestones, meetings, workshops, …  “Technical” – preparing challenges and running CERN end – 24 x 7 ???

LCG Service Challenges – Deploying the Service SC3 – Milestone Decomposition  File transfer goals:  Build up disk – disk transfer speeds to 150MB/s with 1GB/s out of CERN  SC2 was 100MB/s – agreed by site  Include tape – transfer speeds of 60MB/s with 300MB/s out of CERN  Tier1 goals:  Bring in additional Tier1 sites wrt SC2 (at least wrt the original plan…)  PIC and Nordic most likely added later: SC4?  Tier2 goals:  Start to bring Tier2 sites into challenge  Agree services T2s offer / require  On-going plan (more later) to address this via GridPP, INFN etc.  Experiment goals:  Address main offline use cases except those related to analysis  i.e. real data flow out of T0-T1-T2; simulation in from T2-T1  Service goals:  Include CPU (to generate files) and storage  Start to add additional components  Catalogs, VOs, experiment-specific solutions etc, 3D involvement, …  Choice of software components, validation, fallback, …

LCG Service Challenges – Deploying the Service SC3 – Experiment Goals  Meetings on-going to discuss goals of SC3 and experiment involvement  Focus on:  First demonstrate robust infrastructure;  Add ‘simulated’ experiment-specific usage patterns;  Add experiment-specific components;  Run experiments offline frameworks but don’t preserve data;  Exercise primary Use Cases except analysis (SC4)  Service phase: data is preserved…  Has significant implications on resources beyond file transfer services  Storage; CPU; Network… Both at CERN and participating sites (T1/T2)  May have different partners for experiment-specific tests (e.g. not all T1s)  In effect, experiments’ usage of SC during service phase = data challenge  Must be exceedingly clear on goals / responsibilities during each phase!

LCG Service Challenges – Deploying the Service SC3 – Experiment Involvement Cont.  Regular discussions with experiments have started  ATLAS: at DM meetings  ALICE+CMS: every ~2 weeks  LHCb: no regular slot yet, but discussions started…  Anticipate to start first with ALICE and CMS (exactly when TDB) ATLAS and LHCb around October  T2 sites being identified in common with these experiments  More later…  List of experiment-specific components and the sites where they need to be deployed being drawn up  Need this on April timeframe for adequate preparation & testing

ALICE Physics Data Challenge ’05 and LCG Service Challenge 3 Latchezar Betev / ALICE Geneva, 6 April 2005 LCG Storage Management Workshop

LCG Service Challenges – Deploying the Service Outline  Goals and structure of PDC’05  Basic parameters and participation plans  Services, processing layout  Timeline

LCG Service Challenges – Deploying the Service Goals and structure  The transfer framework can only be discussed as a part of the overall ALICE Grid strategy  PDC’05 : Test and validation of the remaining parts of the ALICE Offline computing model:  Quasi-online reconstruction of RAW data at CERN (T0), without calibration  ‘push’ of the data from CERN to T1’s  Second phase (delayed) reconstruction at T1’s with calibration and remote storage  Do all of the above again ENTIRELY on the GRID  Structure – logically divided in three phases:  Phase 1 – Slow production of events on the GRID, storage at CERN  Phase 2 (during SC3) – Pass 1 reconstruction at CERN, push data from CERN to T1’s, Pass 2 reconstruction at T1s with calibration and storage:  Phase 2 (throughput phase of SC3) – how fast we can push data out  Phase 3 – Analysis of data (batch) and interactive analysis with PROOF

LCG Service Challenges – Deploying the Service Parameters of the exercise  Data production:  Physics driven, list of physics signals to be produced is being collected now  The processed data has to be preserved – capacity for its storage has been pledged by the participating computing centres for ALICE  Potentially as large data volume as in PDC’04 (50-80TB) – ALICE detector and physics studies require new MC generated signals and improved statistics of the already produced ones  SC3 - ALICE participation currently being discussed:  Computing and storage capacities  Network capacity: reserved/shared and elements (PDC data transfer volume ~200 TB)  Availability of services (more later)  Monitoring: experiment specific and interaction with the SC3 tools

LCG Service Challenges – Deploying the Service Services availability  Transfer and storage services (part 1 of SC3):  GridFTP  gLite FTS  SRM  Experimental solution  Other  How will these become ‘stable services’ for the experiments and with what functionality  Be deployed on sufficiently large computing and storage capacities on the sites (more than on test-beds)  Other Grid services (part 2 of SC3):  We assume that EGEE middleware available on LCG-SC3

LCG Service Challenges – Deploying the Service ALICE PDC’05 layout AliEn CE/SE Central services: -Catalogue -Task queue -Job optimization -etc. AliEn CE/SE AliEn CE/SE LCG CE/SE LCG UI LCG RB LCG CE/SE LCG CE/SE Job submission File registration Phase 1: Event production and storage at CERN Phase 2: pass 1 reconstruction -> fast push to T1 - > pass 2 reconstruction/calibration -> store at T1

LCG Service Challenges – Deploying the Service PDC’05 timeline Apr05–SC2 Complete service AprMayJunJulAugSepOctNovDec 2005 Start event production: - at all available computing capacities (AliEn and LCG) SC3 Throughput test ALICE data ‘push’: - reserved/shared bandwidth - test of available services SC3 Service test: - add more complexity to the picture, data analysis

LCG Service Challenges – Deploying the Service PDC’05 continuation  Last step:  Analysis of the reconstructed data  That is at the end of SC3:  We have some more time to think about it

ATLAS File Transfer Framework and SC3

LCG Service Challenges – Deploying the Service Baseline Requirements  Baseline requirements: simplest system ATLAS can use and must absolutely be production-quality  Requirements for a File Transfer Service:  GridFTP  SRM get and put. SRM copy useful.  Delegate user credentials (GSI, MyProxy)  FIFO queue  All requirements already met with proposed interface  Now will check gLite FTS implementation  Other possibilities:  Globus RFT: SRM support?  CMS PhEDEX

LCG Service Challenges – Deploying the Service Future plans  gLite FTS integration tests:  Instead of DQ doing SRM/GridFTP calls, it will use gLite FTS as the ‘data transfer layer’  Simple use of FIFO queue: feed gLite FTS with SURL->SURL requests  Keep logic of handling requests and grid catalog interactions within ATLAS framework  Schedule:  Planned for July early-phase SC3  ATLAS will do its own testing of gLite FTS on private testbed prior before July: starting as soon as possible  ATLAS sees benefits on trying gLite FTS as soon as possible  To see ASAP whether it meets medium term requirements  Data transfer will require significant effort to ramp up!  Help debugging gLite FTS  Transfers between more sites  A real usage with test data  Uniform low-level file transfer layer?

CMS & Service Challenge 3 (Preliminary)

LCG Service Challenges – Deploying the Service Service Challenge 3 CMS transfers  PhEDEx will be used for SC3 CMS transfer tests  Available to help set up if others have interest  Not the only CMS service that needs setting up at the sites  Transfer features expected to be tested  Simultaneous data import, export and serving for local processing  Must be representative of real experiment data flow  Must use realistic files and realistic storage  This will become the next production service, right?  To/from tape at least on some Tier 1 sites  We are working on file size  Cannot afford to fail  Suggest testing a couple of different configurations according to region/site preferences: SRM-SRM transfers, GridFTP only, FTS  EGEE FTS not a high priority for CMS, may be for some sites?  Risk for using for all sites is too high, not clear why for e.g. U.S. ! !

LCG Service Challenges – Deploying the Service Service Challenge 3 Site services  Transfers: import and export  For CMS tests, using PhEDEx installation at site  Serving data to bulk data processing applications  Simulated and/or real applications  This requires several other services to be available  Computing element, job submission, output harvesting for transfer, software installation + publishing into the information system, bookkeeping / monitoring databases for production, file catalogue, PubDB or successor  The above are expected to be available concurrently  Throughput phase: concurrent import/export transfers only  Service phase: all, but top throughput not required

LCG Service Challenges – Deploying the Service Service Challenge 3 Schedule  CMS will participate in the “early phase” (cf. Jamie)  July: throughput phase  T0/T1/T2 simultaneous import/export  To and from tape at T1s — application level end-to-end transfers  Real files, real storage  August: setup phase  Agents work while we all enjoy our holidays?  September: service phase 1 — modest throughput  Demonstrate bulk data processing, simulation at T1, T2s  Requires software, job submission, output harvesting, monitoring, …  Not everything everywhere, something reasonable at each site  November: service phase 2 — modest throughput  Phase 1 + continuous data movement  Precursor for next production service

LCG Service Challenges – Deploying the Service Summary  Data transfers is a substantial topic  The interesting world is beyond “SRM-or-FTS?”…  CMS has a production data transfer system  Many major and smaller sites already involved  Handles large scale continuous transfers, mostly on background  Relatively low overhead  Planning next steps  Significant amount of work to ramp up everything  Major issues remain to be sorted out  That’s why we are here, doing service challenges  That’s why CMS visit sites directly for technical contact  We have to press on, service challenge or not  It’s an exciting time, but have to move swiftly :-)

LCG Service Challenges – Deploying the Service LHCb & Service Challenges  We would like to get access to the FTS system as early as possible  May ?  Our own evaluation of stability  Start with one central Request Store instance  Add more instances as necessary  Do bulk transfers for Stripping data distribution to Tier1 centers  September  T0-T1, T1-T1 transfers  Not part of LHCb Data Challenge

LCG Service Challenges – Deploying the Service SC3 – Milestone Decomposition  File transfer goals:  Build up disk – disk transfer speeds to 150MB/s  SC2 was 100MB/s – agreed by site  Include tape – transfer speeds of 60MB/s  Tier1 goals:  Bring in additional Tier1 sites wrt SC2  PIC and Nordic most likely added later: SC4?  Tier2 goals:  Start to bring Tier2 sites into challenge  Agree services T2s offer / require  On-going plan (more later) to address this via GridPP, INFN etc.  Experiment goals:  Address main offline use cases except those related to analysis  i.e. real data flow out of T0-T1-T2; simulation in from T2-T1  Service goals:  Include CPU (to generate files) and storage  Start to add additional components  Catalogs, VOs, experiment-specific solutions etc, 3D involvement, …  Choice of software components, validation, fallback, …

LCG Service Challenges – Deploying the Service A Simple T2 Model N.B. this may vary from region to region  Each T2 is configured to upload MC data to and download data via a given T1  In case the T1 is logical unavailable, wait and retry  MC production might eventually stall  For data download, retrieve via alternate route / T1  Which may well be at lower speed, but hopefully rare  Data residing at a T1 other than ‘preferred’ T1 is transparently delivered through appropriate network route  T1s are expected to have at least as good interconnectivity as to T0  Each Tier-2 is associated with a Tier-1 who is responsible for getting them set up  Services at T2 are managed storage and reliable file transfer  DB component at T1; user agent also at T2  1GBit network connectivity – shared (less will suffice to start with, more maybe needed!)

LCG Service Challenges – Deploying the Service Initial Tier2 Sites for SC3 SiteTier1Experiment Bari, ItalyCNAF, ItalyAlice, CMS Turin, ItalyCNAF, ItalyAlice DESY, GermanyFZK, GermanyATLAS, CMS Lancaster, UKRAL, UKATLAS London, UKRAL, UKCMS ScotGrid, UKRAL, UKLHCb US Tier2sBNL / FNALATLAS / CMS

LCG Service Challenges – Deploying the Service Prime Tier-2 sites  For SC3 we aim for  DESY   FZK (CMS + ATLAS)  Lancaster   RAL (ATLAS)  London   RAL (CMS)  Scotgrid   RAL (LHCb)  Torino   CNAF (ALICE)  US sites   FNAL (CMS)  Responsibility between T1 and T2 (+ experiments)  CERN’s role limited  Develop a manual “how to connect as a T2”  Provide relevant s/w + installation guides  Assist in workshops, training etc.  Other interested parties: Prague, Warsaw, Moscow,..  Also attacking larger scale problem through national / regional bodies  GridPP, INFN, HEPiX, US-ATLAS, US-CMS SiteTier1Experiment Bari, ItalyCNAF, ItalyCMS Turin, ItalyCNAF, ItalyAlice DESY, GermanyFZK, GermanyATLAS, CMS Lancaster, UKRAL, UKATLAS London, UKRAL, UKCMS ScotGrid, UKRAL, UKLHCb US Tier2sBNL / FNALATLAS / CMS More sites are appearing! For CMS, also Legnaro, Rome and Pisa For Atlas, sites will be discussed next week Plans for a workshop in Bari end-May advancing well. Tutorials also foreseen in the UK. Pilot also with FZK / DESY – see May HEPiX Also foreseen for Spain, France, … US sites through US-expt?

LCG Service Challenges – Deploying the Service Tier2 RegionCoordinating BodyComments ItalyINFNA workshop is foreseen for May during which hands-on training on the Disk Pool Manager and File Transfer components will be held. UKGridPPA coordinated effort to setup managed storage and File Transfer services is being managed through GridPP and monitored via the GridPP T2 deployment board. Asia-PacificASCC TaipeiThe services offered by and to Tier2 sites will be exposed, together with a basic model for Tier2 sites at the Service Challenge meeting held at ASCC in April 2005. EuropeHEPiXA similar activity will take place at HEPiX at FZK in May 2005, together with detailed technical presentations on the relevant software components. USUS-ATLAS and US-CMSTier2 activities in the US are being coordinated through the corresponding experiment bodies. CanadaTriumfA Tier2 workshop will be held around the time of the Service Challenge meeting to be held in Triumf in November 2005. Other sitesCERNOne or more workshops will be held to cover those Tier2 sites with no obvious regional or other coordinating body, most likely end 2005 / early 2006.

Tier2 and Base S/W Components 1)Disk Pool Manager (of some flavour…)  e.g. dCache, DPM, … 2)gLite FTS client (and T1 services) 3)Possibly also local catalog, e.g. LFC, FiReMan, … 4)Experiment-specific s/w and services ( ‘agents’ ) 1 – 3 will be bundled with LCG release. Experiment-specific s/w will not…

Tier2 Software Components Overview of dCache, DPM, gLite FTS + LFC and FiReMan

File Transfer Software SC3 Gavin McCance – JRA1 Data Management Cluster LCG Storage Management Workshop April 6, 2005, CERN

LCG Service Challenges – Deploying the Service Reliable File Transfer Requirements  LCG created a set of requirements based on the Robust Data Transfer Service Challenge  LCG and gLite teams translated this into a detailed architecture and design document for the software and the service  A prototype (radiant) was created to test out the architecture and was used in SC1 and SC2  Architecture and design have worked well for SC2  gLite FTS (“File Transfer Service”) is an instantiation of the same architecture and design, and is the candidate for use in SC3  Current version of FTS and SC2 radiant software are interoperable

LCG Service Challenges – Deploying the Service gLite FTS Summary  Core FTS software is in good shape  On its way to becoming software that can provide a manageable, reliable service  Stress-testing underway  Gaining operational experience with it early  Increasing as we head to SC3  We are now understanding how the experiment frameworks can plug onto it  Interested to discuss further and work with experiments  ‘Pilot’ to be setup at CERN in May  To allow experiments to start to evaluate / gain experience

LCG Service Challenges – Deploying the Service Disk Pool Manager (DPM) aims  Provide a solution for the small Tier-2s in LCG-2  This implies 1 to 10 Terabytes in 2005  Focus on manageability  Easy to install  Easy to configure  Low effort for ongoing maintenance  Easy to add/remove resources  Support for multiple physical partitions  On one or more disk server nodes  Support for different space types – volatile and permanent  Support for multiple replicas of a file within the disk pools

LCG Service Challenges – Deploying the Service DPM Manageability  Few daemons to install  Disk Pool Manager  Name Server  SRM  No central configuration files  Disk nodes request to add themselves to the DPM  Easy to remove disks and partitions  Allows simple reconfiguration of the Disk Pools  Administrator can temporarily remove file systems from the DPM if a disk has crashed and is being repaired  DPM automatically configures a file system as “unavailable” when it is not contactable

LCG Service Challenges – Deploying the Service DPM Features  DPM access via different interfaces  Direct Socket interface  SRM v1  SRM v2 Basic  Also offer a large part of SRM v2 Advanced  Global Space Reservation (next version)  Namespace operations  Permissions  Copy and Remote Get/Put (next version)  Data Access  Gridftp, rfio (ROOTD, XROOTD could be easily added)  DPM Catalog shares same code as LCG File Catalog  Possibility to act as a “Local Replica Catalog” in a distributed catalog

LCG Service Challenges – Deploying the Service DPM Status  DPNS, DPM, SRM v1 and SRM v2 (without Copy nor global space reservation) have been tested for 4 months  The secure version has been tested for 6 weeks  GsiFTP has been modified to interface to the DPM  RFIO interface is in final stage of development

LCG Service Challenges – Deploying the Service DPM - Proposal for SC3  Provide a possible solution for the small Tier-2s  This implies 1 to 10 Terabytes in 2005  Focus on manageability  Easy to install  Easy to configure  Low effort for ongoing maintenance  Easy to add/remove resources  Support for multiple physical partitions  On one or more disk server nodes  Replacement of ‘Classic SE’  Only metadata operations needed (data does not need to be copied)  Working with Tier2s regarding evaluation

CASTOR status and plans: proposals for SC3 5/4/2005

LCG Service Challenges – Deploying the Service Outline  CASTOR status & plans  SC3 proposal  CASTOR version  SRM version

LCG Service Challenges – Deploying the Service CASTOR status & plans  The CASTOR version running in production follows the original CASTOR design defined in 1999 – 2000 and fully deployed for production use in early 2002  Name space in Oracle, MySQL  good scalability (currently >33M files)  Known limitations:  The stager disk residence catalog is in memory  scales to O(200k) resident files but not beyond  No file access request throttling  No controlled resource sharing  Stager limitations has lead to a deployment with many stager instances  ~40 instances, each with each its own disk pool  Bad sharing and loadbalancing  Difficult to operate

LCG Service Challenges – Deploying the Service CASTOR status & plans  New stager (disk pool manager)  Disk file residence catalog in Oracle (later also MySQL)  good scalability  Externalized request scheduling  throttling and controlled resource sharing  Many features targeted for easy operation and automated management  New stager status:  Developments started in 2003  First version of complete new system was ready for testing in late 2004  Running in testbed since 4 months  Functionality OK  Performance:  Tier-0 requirements heavily tested with satisfactory results  Tier-1 requirements tests still ongoing. Request handling requires more tuning  New stager API not compatible with old stager  new SRM required

LCG Service Challenges – Deploying the Service New stager  Not going to present the architecture here. Please refer to CASTOR presentations presented at various occasions in 2004- 2005, e.g.  IEEE MSST 2004  http://cern.ch/castor/PRESENTATIONS/2004/MSST2004/ MSST2004-40.html http://cern.ch/castor/PRESENTATIONS/2004/MSST2004/ MSST2004-40.html  http://cern.ch/castor/PRESENTATIONS/2004/MSST2004/ MSST2004-CASTOR-1.pdf http://cern.ch/castor/PRESENTATIONS/2004/MSST2004/ MSST2004-CASTOR-1.pdf  CHEP04  http://cern.ch/castor/PRESENTATIONS/2004/chep04/index.html http://cern.ch/castor/PRESENTATIONS/2004/chep04/index.html  Operational experience  http://cern.ch/castor/PRESENTATIONS/2005/CCM- 20050118/NewStager.htm http://cern.ch/castor/PRESENTATIONS/2005/CCM- 20050118/NewStager.htm  http://cern.ch/castor/PRESENTATIONS/2005/CCM- 20050322/ALICE-experience.html http://cern.ch/castor/PRESENTATIONS/2005/CCM- 20050322/ALICE-experience.html  API documentation  http://cern.ch/castor/DOCUMENTATION/CODE/STAGE/Ne wAPI/index.html http://cern.ch/castor/DOCUMENTATION/CODE/STAGE/Ne wAPI/index.html

LCG Service Challenges – Deploying the Service CASTOR status & plans  New stager deployment  Roll-out plan is being prepared  It is too early to give a definite timescale  Gain operational experience with test instances  Configuration, monitoring, automated management  Understand the Oracle requirements  Hardware, tuning, administration  User interfaces may require more discussions with experiments  RFIO backward compatible  Stager commands are only partially backward compatible  Stager API is completely new ( http://cern.ch/castor/DOCUMENTATION/CODE/STAGE/NewAPI/index.html ) http://cern.ch/castor/DOCUMENTATION/CODE/STAGE/NewAPI/index.html  Work out a good deployment architecture exploiting the advantages of the new stager  Scalable catalog and resource sharing facilities  fewer instances

LCG Service Challenges – Deploying the Service SC3 proposal  CASTOR version:  Throughput phase  Use the new stager  Service phase  Can use new stager, but…  Sharing data with production disk pools only possible if the participating experiments have been migrated  New stager cannot share disk pools with an old stager instance

LCG Service Challenges – Deploying the Service Possible SC3 deployment using new stager SRM ALICE pool RFIO, ROOT CASTOR stager ATLAS pool RFIO, ROOT CMS pool RFIO, ROOT LHCb pool RFIO, ROOT SC3 transfer pool gridftp Internal file replication gridftp Stager managed disk pools

LCG Service Challenges – Deploying the Service SRM & SC3 – proposal  SRM version  Strongly recommend to stay with V1.1  It works and is well tested and known to interoperate but it took a long time and a lot of efforts to get there  Enough functionality for the throughput phase  Going for a newer version is associated with high risks  Will not interoperate out-of-the-box!  True (and respected) space reservations is well understood, e.g. what does 1TB reserved space mean?  I can write a 1TB file without hitting ENOSPC  I can write 1000 x 1GB files, each without hitting ENOSPC  V2.1 definition does not address this problem  The operational nightmare: “ls”  You would better gridftp the SOAP message resulting from srmLs on some castor directories

LCG Service Challenges – Deploying the Service Status of SRM definition  SRM v1.1 insufficient – mainly lack of pinning  SRM v3 not required – and timescale too late  Require Volatile, Permanent space; Durable not practical  Global space reservation: reserve, release, update (mandatory LHCb, useful ATLAS,ALICE). Compactspace NN  Permissions on directories mandatory  Prefer based on roles and not DN (SRM integrated with VOMS desirable but timescale?)  Directory functions (except mv) should be implemented asap  Pin/unpin high priority  srmGetProtocols useful but not mandatory  Abort, suspend, resume request : all low priority  Relative paths in SURL important for ATLAS, LHCb, not for ALICE CMS input/comments not included yet

LCG Service Challenges – Deploying the Service SC3 – Milestone Decomposition  File transfer goals:  Build up disk – disk transfer speeds to 150MB/s  SC2 was 100MB/s – agreed by site  Include tape – transfer speeds of 60MB/s  Tier1 goals:  Bring in additional Tier1 sites wrt SC2  PIC and Nordic most likely added later: SC4?  Tier2 goals:  Start to bring Tier2 sites into challenge  Agree services T2s offer / require  On-going plan (more later) to address this via GridPP, INFN etc.  Experiment goals:  Address main offline use cases except those related to analysis  i.e. real data flow out of T0-T1-T2; simulation in from T2-T1  Service goals:  Include CPU (to generate files) and storage  Start to add additional components  Catalogs, VOs, experiment-specific solutions etc, 3D involvement, …  Choice of software components, validation, fallback, …

LCG Service Challenges – Deploying the Service Service Goals  Expect relatively modest increase in ‘service’ components  File catalog based on agreement from Baseline Services WG  Other services agreed by BSWG  Experiment-specific components and impact on other services, e.g. Distributed Database Services, need to be clarified as soon as possible  Similarly, requirements for processing power and storage at all sites involved (T0, T1, T2)  (This is for both Service and Challenge phases: where we run the experiments’ s/w and store the output!)

LCG Service Challenges – Deploying the Service SC3 – Milestone Decomposition  File transfer goals:  Build up disk – disk transfer speeds to 150MB/s  SC2 was 100MB/s – agreed by site  Include tape – transfer speeds of 60MB/s  Tier1 goals:  Bring in additional Tier1 sites wrt SC2  PIC and Nordic most likely added later: SC4?  Tier2 goals:  Start to bring Tier2 sites into challenge  Agree services T2s offer / require  On-going plan (more later) to address this via GridPP, INFN etc.  Experiment goals:  Address main offline use cases except those related to analysis  i.e. real data flow out of T0-T1-T2; simulation in from T2-T1  Service goals:  Include CPU (to generate files) and storage  Start to add additional components  Catalogs, VOs, experiment-specific solutions etc, 3D involvement, …  Choice of software components, validation, fallback, …

LCG Service Challenges – Deploying the Service Ramp-up of FTS and T1s  As expected, this is proving to be a significant amount of work!  Need to keep up momentum:  Increasing file transfer performance;  Adding tape;  Building up network links;  Bringing additional T1s into the challenges…

LCG Service Challenges – Deploying the Service File Transfer Milestones  Need to understand how infrastructure at CERN will build up to address file transfer goals of SC3  Internal / External network infrastructure  File transfer Server nodes  Access to Experiment files  For each Tier1 need to understand how local infrastructure and network connectivity will be build up to meet requirements  For Tier2s, assume “Minimal Usable Model” (see earlier slide)…  Assume DB for FTS will run at T1 with “client” also at T2  Transfers can be initiated from both ends

LCG Service Challenges – Deploying the Service ‘SC3’ pilot services  Not really restricted to SC3, except timewise…  gLite FTS pilot  For experiments to start to get experience with it, integrating with their frameworks etc  File catalogs  Both LFC and FiReMan… and RLS continues for the time being…  Expect these to start in May, based on s/w delivered this week (code free April 15) for release end May  Max one more cycle for SC3  end June is too late for throughput phase!  LFC is already part of LCG releases

LCG Service Challenges – Deploying the Service Service Challenge 3 Network – CERN Side edoardo.martelli@cern.chedoardo.martelli@cern.ch 20050408 cernh7 T1s General Purpose Network backbone Tapes Disks Database 10Gb link 1Gb link 2x1Gb links up to 4Gbps up to 8Gbps 44/88 Machines 192.16.160.0/22 192.16.160.1 192.16.160.254 4x1Gb nx1Gb Routing cernh7, one GPN backbone router and the 44-88 machines will be connected at layer 2. The 160.16.160.0/22 prefix will be shared among their interfaces. The correct routing must be configured on each one of the 44-88 machines: default towards the GPN, tier1s prefixes towards cernh7. Security Very strict access-list must be configured on cernh7 and the GPN backbone router.

LCG Service Challenges – Deploying the Service 2005 Sep-Dec - SC4 preparation In parallel with the SC3 model validation period, in preparation for the first 2006 service challenge (SC4) – Using 500 MByte/s test facility  test PIC and Nordic T1s  and T2’s that are ready (Prague, LAL, UK, INFN,..) Build up the production facility at CERN to 3.6 GBytes/s Expand the capability at all Tier-1s to full nominal data rate SC2 SC3 SC4 Full physics run 200520072006 2008 First physics First beams cosmics Historical slides from Les / Ian

LCG Service Challenges – Deploying the Service 2006 Jan-Aug - SC4 SC4 – full computing model services - Tier-0, ALL Tier-1s, all major Tier-2s operational at full target data rates (~2 GB/sec at Tier-0) - acquisition - reconstruction - recording – distribution, PLUS ESD skimming, servicing Tier-2s Goal – stable test service for one month – April 2006 100% Computing Model Validation Period (May-August 2006) Tier-0/1/2 full model test - All experiments - 100% nominal data rate, with processing load scaled to 2006 cpus SC2 SC3 SC4 Full physics run 200520072006 2008 First physics First beams cosmics Historical slides from Les / Ian

LCG Service Challenges – Deploying the Service SC4 Planning  Discussing a joint workshop with ARDA focussing on SC4 after the summer  Tentative dates: during week of October 10 – 14  Clash with GDB and HEPiX?  After we have concrete results from SC3?  When a more concrete model for analysis has appeared?  In all events, early enough for SC4 planning…

LCG Service Challenges – Deploying the Service 2006 Sep – LHC service available The SC4 service becomes the permanent LHC service – available for experiments’ testing, commissioning, processing of cosmic data, etc. All centres ramp-up to capacity needed at LHC startup  TWICE nominal performance  Milestone to demonstrate this 3 months before first physics data  April 2007 SC2 SC3 SC4 LHC Service Operation Full physics run 200520072006 2008 First physics First beams cosmics Historical slides from Les / Ian

LCG Service Challenges – Deploying the Service Key dates for Connectivity SC2 SC3 SC4 LHC Service Operation Full physics run 200520072006 2008 First physics First beams cosmics June05 - Technical Design Report  Credibility Review by LHCC Sep05 - SC3 Service – 8-9 Tier-1s sustain - 1 Gbps at Tier-1s, 5 Gbps at CERN Extended peaks at 10 Gbps CERN and some Tier-1s Jan06 - SC4 Setup – AllTier-1s 10 Gbps at >5 Tier-1s, 35 Gbps at CERN July06 - LHC Service – All Tier-1s 10 Gbps at Tier-1s, 70 Gbps at CERN

LCG Service Challenges – Deploying the Service Key dates for Services SC2 SC3 SC4 LHC Service Operation Full physics run 200520072006 2008 First physics First beams cosmics June05 - Technical Design Report Sep05 - SC3 Service Phase May06 –SC4 Service Phase Sep06 – Initial LHC Service in stable operation

LCG Service Challenges – Deploying the Service  Additional threads started to address:  Experiment involvement;  Bringing T2s in SC3;  Longer-term goals of bringing all T2s into the LCG Service (Challenges)  The enthusiasm and support provided to these new activities is much appreciated  We have a lot of work ahead…  …but the problem is beginning to become tractable(?)

LCG Service Challenges – Deploying the Service Conclusions  To be ready to fully exploit LHC, significant resources need to be allocated to a series of Service Challenges by all concerned parties  These challenges should be seen as an essential on-going and long-term commitment to achieving production LCG  The countdown has started – we are already in (pre-)production mode  Next stop: 2020

Ramping up the LCG Service The LCG Service Challenges: Ramping up the LCG Service Jamie Shiers, CERN-IT-GD-SC April 2005 LCG.

Similar presentations

Presentation on theme: "Ramping up the LCG Service The LCG Service Challenges: Ramping up the LCG Service Jamie Shiers, CERN-IT-GD-SC April 2005 LCG."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Ramping up the LCG Service The LCG Service Challenges: Ramping up the LCG Service Jamie Shiers, CERN-IT-GD-SC April 2005 LCG.

Similar presentations

Presentation on theme: "Ramping up the LCG Service The LCG Service Challenges: Ramping up the LCG Service Jamie Shiers, CERN-IT-GD-SC April 2005 LCG."— Presentation transcript:

Similar presentations

About project

Feedback