Plans for Service Challenge 3 Ian Bird LHCC Referees Meeting 27 th June 2005

LCG Project, Plans for SC3 2 LCG Deployment Schedule

LCG Project, Plans for SC3 3 Service Challenge 3 - Phases High level view:  Setup phase  Finishes with 2 weeks sustained throughput test in July 2005  Primary goals:  150MB/s disk – disk to Tier1s;  60MB/s disk (T0) – tape (T1s)  Secondary goals:  Include a few named T2 sites (T2 -> T1 transfers)  Encourage remaining T1s to start disk – disk transfers  Service phase – must be run as the real production service  September – end 2005  Start with ALICE & CMS, add ATLAS and LHCb October/November  All offline use cases except for analysis  More components: WMS, VOMS, catalogs, experiment-specific solutions  Implies production setup (CE, SE, …)

LCG Project, Plans for SC3 4 SC3 – Deadlines and Deliverables  May 31 st 2005: basic components delivered and in place  June 2005: integration testing  June 13 – 15: SC3 planning workshop at CERN – experiment issues  June 30 th 2005: integration testing successfully completed  July 1 – 10: start disk – disk throughput tests  Assume a number of false starts / difficulties  July 11 – 20: disk tests  July 21 – 27: tape tests  July 28 – 31: T2 tests

LCG Project, Plans for SC3 5 Service Challenge Workshop  Three-day meeting (13-15 June)  First two days with presentations from Experiments. 1/2 day per experiment to cover:  Summary of Grid Data Challenges to date  Goals for SC3  Plans for usage of SC3 infrastructure  Third day focused on issues for the Tier-1 sites  Discussion focused on issues raised during previous two days  SRM requirements presentations from experiments and developers  Approximately 40 people for first two days and 60 for last day  Many CERN IT people appearing for last day  Not all sites present during first two days (??) – if present, very quiet!

LCG Project, Plans for SC3 6 Experiment Goals and Plans  All four experiments plan to be involved in SC3  Brief “one-line” summary  LHCb will evaluate the new tools via the pilot and do a data management challenge in September. Assuming ok will want to use a service from October  ALICE will also evaluate the new tools but want to run a full data challenge based on this infrastructure asap  CMS will use the resources to run two challenges in September and November, but with modest throughput. These includes T0-T1-T2 data movement and T2-T1 movement for MC Data  ATLAS plan to run a Tier-0 exercise in October along with MC production at T2 and reprocessing at Tier-1. They will use their new DDM software stack

LCG Project, Plans for SC3 7 Experiment Goals and Plans  Concern that the experiment timelines all overlap  Creating a unified timeline from the detailed presentations  We need to respond with what is possible  Pilot services for FTS and LFC are of great interest to experiments.  They’d like Fireman as well for testing  Long discussions about “VO Boxes” at all sites – neither sites, experiments or middleware providers have worked through full implications of this  First we need to list exactly what the expt requirements are  Plan is to provide an interim solution for evaluation during SC3

LCG Project, Plans for SC3 8 Tier-1 Plans and Goals  Clear message from workshop that some sites did not understand what SC3 mean in terms of compute resources  “more than a transfer test”  We need to resolve how to integrate SC3 resources into the production grid environment  “there can only be one production environment” – discussed in GDB last week  Service levels provided will be “best-effort”  We should be able to live with a site being down for a while  But we must measure site uptime/availability/response during the challenge.

LCG Project, Plans for SC3 9 Software at Tier-1s  Many SRM services are late – deadline was for end May  Many sites still haven’t got services ready for SC3  Some need to upgrade versions (BNL)  Some need to debug LAN network connections (RAL)  Some are finalizing installs (FZK, ASCC, …)  And we’re still mostly at the level of debugging SRM transfers  Many errors and retries detected at FTS level  Still need to rerun iperf tests to measure expected network throughput for all sites  Activity required from Tier-1s to run the network measurement tests and more SRM level tests  Sites need to be more proactive in testing and publishing the information

LCG Project, Plans for SC3 10 CERN Services

LCG Project, Plans for SC3 11 SC3 Services Status  FTS  SC3 service installed and configured. Limited testing undergone with Tier-1s. Many Tier-1’s still upgrading to dCache and it’s not all stable yet  BNL have a version of the FTS Server for their T1-T2 traffic  seeing many problems in getting it installed and configured  working with gLite team to try and solve these  Pilot services not ready yet  Installed but not configured yet  Experienced long delays for new software through gLite build+test process  but we now have a tag that will be ok for setup/throughput  This is part of LCG-2_5_0  Will need new version of FTS for service phase  Current version does not do inter-VO scheduling  This presents a risk since it will be a major rewrite

LCG Project, Plans for SC3 12 SC3 Services Status  LFC  Pilot and SC3 services are installed, configured and announced to experiments  POOL interface now available (POOL 2.1.0)  Not much usage yet by experiments  CASTORGRIDSC SRM  20TB setup running using old stager and old SRM code  Plan is to migrate to new CASTOR stager  fallback solution is to use old stager for setup phase  Migration of SC setup to new Castor stager is in progress

LCG Project, Plans for SC3 13 SC3 Services Status  Starting to put in place the service teams for SC3  First level support at CERN from operators  Second line support at CERN from GD SC and EIS teams  Third line support from software experts  LFC, FTS, Castor-SRM, …  Site support through site specific service challenge mailing lists  What is the level of support we will get?  Operator procedures and problem escalation steps still not clear  Reporting of problems through – tied into problem tracking system

LCG Project, Plans for SC3 14 Communication  Service Challenge Wiki  Takes over from service-radiant wiki/web-site used in SC1 & 2  hallenges hallenges  Contains Tier-0 and Tier-1 contact/configuration information and work logs for SC teams  Weekly phonecons ongoing  Daily service meetings for CERN teams from 27 th June  Technical communication through service-challenge- tech list  What else is required by Tier-1s?  Daily (or frequent) meetings during SC?

LCG Project, Plans for SC3 15 Summary  Good understanding and agreement on goals of SC3  What services need to run where,  Proposed metrics to define success  Detailed schedule  Detailed discussion of experiment goals/plans in workshop last week  Concerns about readiness of many sites to run production-level services  Preparations are late, but lots of pressure and effort  Are enough resources available to run services?  Backups, single points of failure, vacations, …  Experiments expect that SC3 leads to real production service by end of year  Must continue to run during preparations for SC4  This is the build up to the LHC service – must ensure that appropriate resources are behind it

LCG Project, Plans for SC3 16 Core Site Services  CERN  Storage: Castor/SRM  File catalogue: POOL LFC Oracle  FNAL  Storage: dCache/SRM  File catalogue: POOL Globus RLS  CNAF  Storage: Castor/SRM  File catalogue: POOL LFC Oracle  RAL  Storage: dCache/SRM  File catalogue: POOL LFC Oracle?  IN2P3  Storage: dCache/SRM  File catalogue: POOL LFC Oracle  SARA/NIKHEF  Storage:dCache/SRM  File catalogue: POOL LFC MySQL(?)  PIC  Storage: Castor/SRM  File catalogue: POOL LFC MySQL  FZK  Storage: dCache/SRM  File catalogue: POOL LFC Oracle  ASCC  Storage: Castor/SRM  File catalogue: POOL LFC Oracle  BNL  Storage: dCache/SRM  File catalogue: POOL LFC Oracle  TRIUMF  Storage: dCache/SRM  File catalogue: POOL LRC MySQL(?)  NDGF  Storage:  File catalogue: Running FTS service for T2s