1. Dirk Duellmann CERN IT/PSS and 3D http://lcg3d.cern.ch
LCG 3D Project Update
Dirk Duellmann
CERN IT/PSS and 3D

2. LCG 3D Service Architecture
Oracle Streams
http cache (SQUID)
Cross DB copy &
MySQL/SQLight Files M O S S T0
- autonomous reliable service
T1- db back bone
- all data replicated
- reliable service O O F
T2 - local db cache
-subset data
-only local service
Online DB
autonomous reliable service O M S S
R/O Access at Tier 1/2 (at least initially)
LCG 3D Status
Dirk Duellmann

3. LCG Database Deployment Plan
After October ‘05 workshop a database deployment plan has been presented to LCG GDB and MB
Two production phases
March - Oct ‘06 : partial production service
Production service (parallel to existing testbed)
H/W requirements defined by experiments/projects
Based on Oracle 10gR2
Subset of LCG tier 1 sites: ASCC, CERN, BNL, CNAF, GridKA, IN2P3, RAL
Oct ‘06- onwards : full production service
Adjusted h/w requirements (defined at summer ‘06 workshop)
Other tier 1 sites joined in: PIC, NIKHEF, NDG, TRIUMF
LCG 3D Status
Dirk Duellmann

4. Oracle Licenses For Tier 1s
3D collected license needs from experiments and s/w projects
And validated with T1 site responsibles
Some 152 processor licenses (incl. Grid Services and Castor)
CERN has negotiated with Oracle a proposal with very attractive conditions
T1 sites have been contacted and agreed to the proposal
FNAL had already acquired their licenses
Tier 1 sites should now be covered for s/w and support!
Support accounts for Tier 1s will be enable as soon as papers have been signed and site contribution has been received.
LCG 3D Status
Dirk Duellmann

5. Oracle Instant Client Distribution
The issue of client distribution has been discussed with Oracle and an agreement has been achieved
The instant client can be integrated into the LCG middleware and application distributions
As long as the included license file is preserved and adhered to
The SPI project in the Application Area will from now on bundle the software as part of AA releases.
Experiments and LCG middleware should take advantage and pick up validated client releases from this single source.
Version management will happen as for other AA packages via the established channels for external packages
LCG 3D Status
Dirk Duellmann

6. Tier 1 Hardware Setup Propose to setup for first 6 month
2/3 dual-cpu database nodes with 2GB or more
Setup as RAC cluster (preferably) per experiment
ATLAS: 3 nodes with 300GB storage (after mirroring)
LHCb: 2 nodes with 100GB storage (after mirroring)
Shared storage (eg FibreChannel) proposed to allow for clustering
2-3 dual-cpu Squid nodes with 1GB or more
Squid s/w packaged by CMS will be provided by 3D
100GB storage per node
Need to clarify service responsibility (DB or admin team?)
Target s/w release: Oracle 10gR2
RedHat Enterprise Server to insure Oracle support
LCG 3D Status
Dirk Duellmann

7. LCG 3D - Tier 1 Database Setup
Phase 1 Sites
ASGC, BNL, CNAF, IN2P3, RAL - DB cluster available, part of 3D throughput tests
GridKA - DB cluster available, expected to join soon
Phase 2 Sites
TRIUMF - regular attendance in 3D planning meetings
PIC, NIKHEF/SARA - DBA contact established
NDGF - early discussions
LCG 3D Status
Dirk Duellmann

8. LCG 3D Throughput Tests Initially scheduled for May:
Use the production database clusters at tier 1 and obtain a first estimate for the replication throughput which can be achieved with the setup
Eg as input to experiment models for calibration data flow
Tests started beginning of May, but will extend until end of June
Main reasons
Server setup problem (often db storage) at sites
Firewall configuration
Throughput optimization need Oracle experts to be involved
LCG 3D Status
Dirk Duellmann

9. June Test Setup Now tests are split into
Basic site tests with all T1s
LAN throughput optimization with Oracle (CERN local)
ATLAS and CMS online groups involved
WAN throughput (CERN-CNAF)
LFC team and LHCb involved
monitoring setup tests (CERN - RAL)
Significant load on one central expert at CERN
Trying to offload with the help of the database service team
LCG 3D Status
Dirk Duellmann

10. Database Monitoring
A central Oracle Enterprise Manager repository at CERN has been setup to collect the status and detailed diagnostics of all 3D production clusters.
Some sites will in parallel have the information integrated into their site local OEM setups
Integration into higher level monitoring tools will be based on this information
Eg experiment dashboard, GridView
In addition test jobs (a la SFT) are planned as soon as higher level services (eg COOL) are deployed by experiments
LCG 3D Status
Dirk Duellmann

11. Preliminary Streams Performance
Out of the box replication performance
Between 10 and 30 MB/min (LAN) for a single conditions insert job
Sufficient for current experiment conditions scenarios
Still quite some room for optimisation as neither source nor destination DB seem to be significantly loaded
Regular phone calls between Oracle and 3D DBA
Head of streams development attending
LCG 3D Status
Dirk Duellmann

12. Frontier/SQUID Tests
Stress tests with many clients at CERN to validate production setup for CMS
DNS based failover validated
Setting up an additional node for ATLAS Frontier tests with COOL
CMS Frontier tests at FNAL
Focus on connection retry and failover
Common s/w component shared with database acces
CMS July release may pickup LCG AA s/w release which is currently prepared
LCG 3D Status
Dirk Duellmann

13. CMS Squid Deployment Squids deployed at several Tier 1 and 2 sites.
Tier 1: LCG: ASCC,IN2P3, PIC, RAL, CNAF, FZK, CERN and OSG: FNAL
Tier 2:LCG: Bari, CIEMAT, DESY, Taiwan, OSG: UCSD, Purdue, Caltech
Remaining Tier 2 sites for SC4 goals:
LCG: Belgium, Legnaro, Budapest, Estonia, CSCS, GRIF, Imperial, Rome, Pisa, NCU/NTU, ITEP, SINP, JINR, IHEP, KNU and OSG: Florida, Nebraska, Wisconsin, MIT.
In progress, plan to finish in the next month.
Request WLCG support for Tier-2 Squid installation
Minimum specs: 1GHz CPU, 1GByte mem, GBit network, 100 GB disk.
Needs to be well connected (network-wise) to worker nodes, and have access to WAN and LAN if on Private Network.
Having 2 machines for failover is a useful option for T-2 but not required.
Sites perform Squid installation using existing instructions
Installation should be completed and tested by July 15, 2006.

14. Messages to T1 Sites
Initial set of Tier 1s Production Setups need to be usable for throughput tests of the experiments by end of June
Need consistent meeting attendance of all sites
Need all sites to join the monitoring setup asap so that we can start integration with
Need DBA coverage during the experiment tests July-Oct
New Tier 1s - NGDF, PIC, NIKHEF/SARA and TRIUMF Production setup needs to be usable for full production in October
DB h/w and s/w setup planning will need to start soon (sites should start from current h/w request)
Please let us know when your h/w acquisition could/should start
LCG 3D Status
Dirk Duellmann

15. Messages to Experiments
Need the experiments to confirm or update their current database and/or Frontier requests for October production
Need an experiment work plan from each experiment for period from July to October
Database and Frontier
A significant fraction of the service expected as of October needs to be demonstrated by middle of August
The plan should address the full chain from online databases over offline to tier 1
This activity will require significant experiment resources to drive the tests
LCG 3D Status
Dirk Duellmann

16. Proposed Timeline End of June Early-July Late-August September October
Throughput phase closed, Experiment application and throughput test start
Early-July
3D DBA day to plan database setup options with new tier 1 sites. Hosted by one of the new sites?
Late-August
3D workshop (experiments and sites) defining October setup and service
September
Experiment ramp-up test with
October
Full service open at all tier 1 sites
LCG 3D Status
Dirk Duellmann