1. Database Deployment and Operations
US ATLAS DDM Operations and MC Production Workshop
BNL, Brookhaven, NY
September 28-29, 2006
Alexandre Vaniachine (Argonne)

2. Outline WLCG 3D Project and ATLAS Database Deployment and Operations
Synergistic activities:
Project GSIMF
Project SHIELDS
Secure database deployment in production
Database release deployment
Conclusions
Alexandre Vaniachine
US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept , 2006

3. WLCG Distributed Deployment of Databases (3D) Project
To set-up database services and facilities for relational data transfers as part of the WLCG infrastructure the WLCG Distributed Deployment of Databases (3D) project is coordinating database deployment and operations activity between LHC experiments and WLCG tier sites [
The 3D project is deploying distributed database services and developing advanced technologies allowing LCG applications to access the required data
3D full production service:
Oct ‘06- onwards
Alexandre Vaniachine
US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept , 2006

4. Database Deployment & Operations: Activity
As ATLAS Computing moves from a development-centric organization to an operations-centric one a new “Database Deployment & Operations” activity is now defined:
The activity consists in the development and deployment (in collaboration with the WLCG 3D project) of the tools that allow the worldwide distribution and installation of databases and related datasets, as well as the actual operation of this system on ATLAS multi-grid infrastructure
Alexandre Vaniachine
US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept , 2006

5. Database Deployment & Operations: Domains
Distributed Deployment - Grigori Rybkine (U.K.)
LCG Deployment: Mireia Dosil (Spain) and Suijian Zhou (Taiwan)
OSG Deployment: Yuri Smirnov (U.S.)
NorduGrid Deployment: Frederik Orellana (Switzerland)
Legacy Sites (batch): John Idarraga (Canada)
Distributed Operations (ATLAS subset of WLCG 3D operations)
Tier1 operations: Gancho Dimitrov, Florbela Viegas (CERN)
Tier2 operations: Stefan Stonjek (U.K.)
Distributed calibration centers: Manuela Cirilli (U.S.)
Development - Jerome Fulachier (France)
Monitoring, SHIELDS, FroNTier/Squid, Dynamic Deployment,…
Documentation & User Support - vacant
Data access, File Transfer (for users), Best Practices, ...
Further information is at
Alexandre Vaniachine
US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept , 2006

6. ATLAS Software & Computing Workshop, Sept. 11-15
The first Database Deployment and Operations session was held during the ATLAS Software Workshop this week
The very first session was quite full
Which is a good sign for the newly defined activity
Alexandre Vaniachine
US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept , 2006

7. US ATLAS Efforts Tier1 activities – see next two presentations:
Iris Wu and Yuri Smirnov
ATLAS Tier2 Database Center Services
Marco Mambelli / Rob Gardner
ATLAS Database Grid Authentication
Wensheng Deng
ATLAS Database User Support
Jerry Gieraltowski
ATLAS FroNTier evaluation (now led by David Front - LCG support)
Richard Hansen (Argonne)
Carlos Gamboa (Brookhaven)
Focus on COOL Performance Studies (CMS is not using COOL)
Alexandre Vaniachine
US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept , 2006

8. A Side Note: ATLAS FroNTier Issues with COOL
Based on general FroNTier performance from the CMS experience prospects are good to achieve the performance required for ATLAS
ATLAS does not store large amounts of data in COOL:
The bulky payload data are stored outside of COOL - in files
It is proposed that these ATLAS COOL POOL ROOT files will be managed by the ATLAS DDM
Currently they are included in ATLAS Database Release
Which is not a scalable solution
Also to be addressed in FroNTier evaluation
Squid cache consistency with COOL
Current issues with COOL tagging interface
May result in stale cache condition
Potentially a showstopper
if there is no COOL manpower to address this issue
Alexandre Vaniachine
US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept , 2006

9. Leveraging Synergistic US Activities
Database deployment support from Open Science Grid
FroNTier will become a part of OSG release
Grid-enabled MySQL client is already a part of OSG release
Building technical collaboration with two small–scale projects
Tech-X Corp. project GSIMF:
Grid Software Installation Management Framework
Argonne is providing specs to Tech-X so that their software product could be used by individual OSG users for ATLAS Database Release deployment
Argonne project SHIELDS:
securing databases on the Grid
Alexandre Vaniachine
US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept , 2006

10. Project SHIELDS Alexandre Vaniachine
US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept , 2006

11. End-to-end transport-level Globus security
What SHIELDS is doing: End-to-End Globus Security Solution for Database Access
A drawback of a middleware approach is vulnerability of the actual database access at the back-end
To eliminate this vulnerability SHIELDS delivers end-to-end Globus security
Database
Middleware
Client
Globus-secure communications
Vulnerable back-end communications
Database
Client
End-to-end transport-level Globus security
Alexandre Vaniachine
US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept , 2006

12. Building Upon the Foundation of DASH
A collaborative DOE HEP SBIR project was completed by April 2006
A partnership of Argonne HEP Division and PIOCON Technologies:
A database consulting business with a track record of Database and Grid integration projects
Project DASH studied the feasibility of the grid-enabled MySQL database development
Building upon the foundation of previous studies project SHIELDS speeded-up the development cycle and provided the first release
Alexandre Vaniachine
US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept , 2006

13. Elaborate Globus Security Infrastructure Embedded
SHIELDS Release 1.0-alpha
ldd bin/mysqld
linux-gate.so.1 => (0x )
librt.so.1 => /lib/librt.so.1 (0x001ff000)
libz.so.1 => /usr/local/osg/MonaLisa/Service/VDTFarm/pgsql/lib/libz.so.1 (0x00e47000)
libpthread.so.0 => /lib/libpthread.so.0 (0x00fd3000)
libglobus_gsi_proxy_core_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libglobus_gsi_proxy_core_gcc32dbgpthr.so.0 (0x00ecb000)
libglobus_gsi_credential_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libglobus_gsi_credential_gcc32dbgpthr.so.0 (0x )
libglobus_gsi_callback_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libglobus_gsi_callback_gcc32dbgpthr.so.0 (0x00c37000)
libglobus_oldgaa_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libglobus_oldgaa_gcc32dbgpthr.so.0 (0x )
libglobus_gsi_sysconfig_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libglobus_gsi_sysconfig_gcc32dbgpthr.so.0 (0x00d3f000)
libglobus_gsi_cert_utils_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libglobus_gsi_cert_utils_gcc32dbgpthr.so.0 (0x00a9f000)
libglobus_openssl_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libglobus_openssl_gcc32dbgpthr.so.0 (0x )
libglobus_openssl_error_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libglobus_openssl_error_gcc32dbgpthr.so.0 (0x009cb000)
libglobus_proxy_ssl_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libglobus_proxy_ssl_gcc32dbgpthr.so.0 (0x0012a000)
libglobus_common_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libglobus_common_gcc32dbgpthr.so.0 (0x0012f000)
libssl_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libssl_gcc32dbgpthr.so.0 (0x0016a000)
libcrypto_gcc32dbgpthr.so.0 => /usr/local/osg/globus/lib/libcrypto_gcc32dbgpthr.so.0 (0x )
libltdl_gcc32dbgpthr.so.3 => /usr/local/osg/globus/lib/libltdl_gcc32dbgpthr.so.3 (0x00c47000)
libdl.so.2 => /lib/libdl.so.2 (0x008f7000)
libcrypt.so.1 => /lib/libcrypt.so.1 (0x006a3000)
libnsl.so.1 => /lib/libnsl.so.1 (0x00c82000)
libstdc++.so.6 => /usr/lib/libstdc++.so.6 (0x )
libm.so.6 => /lib/libm.so.6 (0x0041f000)
libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x001a0000)
libc.so.6 => /lib/libc.so.6 (0x )
/lib/ld-linux.so.2 (0x )
Alexandre Vaniachine
US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept , 2006

14. Secure Database Deployment in Production
The deployment of SHIELDS software product and its integration into the existing distributed data management system in production requires additional efforts, e.g., the
connection protocol string proposed by Wensheng Deng:
A dedicated meeting on that and other technical issues of grid-enabled databases took place at Brookhaven during this workshop
In particular we discussed plans for deployment in production at
US ATLAS Tier1 Computing Center at Brookhaven Lab
As well as deployment of latest versions for testing at
US ATLAS Tier2 Computing Center at the University of Chicago
Can we accomplish the deployment soon?
Alexandre Vaniachine
US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept , 2006

15. Database Release Deployment Technologies
Common LHC software used by ATLAS for data is technology neutral – that is why various technologies can be used for actual data storage:
POOL ROOT files
Relational Databases
Server-based:
Oracle
MySQL
File-based: SQLite
Database-resident data replication technologies:
For server-resident data
Oracle streams
FroNTier
For file-based data
ATLAS Distributed Data Management technology (dataset subscription)
ATLAS Software Installation technology
Alexandre Vaniachine
US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept , 2006

16. Validation and Decoupling
Software installation is not a simple data transfer (also compilation)
Thus, the Software Installation technology includes integrated release validation procedures
In contrast, the validation of the data transfer is much simpler
e.g. the checksum
Database Release deployment does not need complex validation after the data transfer
DECOUPLING
The event data and the software release are decoupled
Any data can be processed by any software release
A similar decoupling for the Database Release data is required
A detailed discussion thread in the Computing Operations mailing list showed that ATLAS software installation technology will not support decoupling of the Database Release from the Software Release
Note, that on the grid the software and the event data are installed at each site in different SE areas
Alexandre Vaniachine
US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept , 2006

17. Technical Choices For now, the two candidate technologies are the
ATLAS Software Release Installation technology
Push Model
ATLAS DQ2 dataset subscription technology
Pull Model
During the Recent ATLAS Software Workshop we tried to reach conclusion on the database release deployment strategy
I recommended to implement proven DQ2 dataset subscription technology
Alexandre Vaniachine
US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept , 2006

18. Database Replicas Deployment: Implementation
Two proven ATLAS technologies – SW Release installation and the DDM DQ2 dataset subscription - are available for implementation:
Integration with the Software Release installation technology
Possible if DB Release and SW Release are tightly coupled
i.e. each DB Release results in new SW Release installation
not sustainable in the long term
Attempts to decouple DB Release and SW Release installations revealed technical ‘showstoppers’ in the current production system
Integration with the DDM DQ2 dataset subscription mechanism
Implemented now thanks to Martin, Vakho, Pavel and Grigori
To collect experience before making long term deployment decisions the deployment technology will be tested in Release production
Alexandre Vaniachine
US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept , 2006

19. Conclusions
New ATLAS Database Deployment and Operation Activity has been defined
US ATLAS contributes efforts to the Activity
To secure production operations the grid-enabled database has to be deployed
Before making long term deployment decisions
FroNTier cache consistency must be validated for COOL
The new database deployment technology will be tested in Release production
Alexandre Vaniachine
US ATLAS DDM Operations Workshop, BNL, Brookhaven, Sept , 2006