2. Ruben.Gaspar.Aparicio_@_cern.ch On behalf DBoD team, IT Department HEPIX 2014 Nebraska Union, University of Nebraska – Lincoln, USA

3. Agenda 3 Manifesto Current status Architecture Demo Management Infrastructure High Availability Monitoring Data protection: backups and recoveries in detail Future development Summary

4. Agenda 4 Manifesto Current status Architecture Demo Management Infrastructure High Availability Monitoring Data protection: backups and recoveries in detail Future development Summary

5. Manifesto 5 https://cern.ch/twiki/bin/view/DB/DBOnDemandManifesto Making users database owners Full DBA privileges Covers a demand from CERN community not addressed by the Oracle service Different RDBMS: MySQL, PostgreSQL and Oracle (private use) No access to underlying hardware Foreseen as single instance service No DBA support or application support No vendor support (except for Oracle) It provides tools to manage DBA actions: configuration, start/stop, upgrades, backups & recoveries, instance monitoring

6. Agenda 6 Manifesto Current status Architecture Demo Management Infrastructure High Availability Monitoring Data protection: backups and recoveries in detail Future development Summary

7. Current status 7

8. 8 Openstack Puppetdb (MySQL) Lhcb-dirac Atlassian databases LCG VOMS Geant4 Hammercloud dbs Webcast QC LHC Splice FTS3 DRUPAL CernVM VCS IAXO UNOSAT … Database usually supports an open source/commercial software

9. Agenda 9 Manifesto Current status Architecture Demo Management Infrastructure High Availability Monitoring Data protection: backups and recoveries in detail Future development Summary

10. Architecture 10 https://cern.ch/dbondemand Oracle VM & physical servers Storage network https://cern.ch/resources DataLogs Diag (Oracle) FIM DBDBOD DBRACMON DB RACMON DAEMON FIM WEB APP CERN AI MONITORING ORACLE EM https://oem.cern.ch DBOD WS Syscontrol DB client

11. Agenda 11 Manifesto Current status Architecture Demo Management Infrastructure High Availability Monitoring Data protection: backups and recoveries in detail Future development Summary

12. 12 https://cern.ch/dbondemand (Please check http://indico.cern.ch/event/313869/ from 12’13’’ )http://indico.cern.ch/event/313869/

13. Agenda 13 Manifesto Current status Architecture Demo Management Infrastructure High Availability Monitoring Data protection: backups and recoveries in detail Future development Summary

14. DBOD Daemon 14 Small program which: Fetches to-be executed jobs from the database Manage jobs execution (via IT-DB framework) Carries job post-execution tasks, if necessary Updates the application DB with job results and instance status Executes around ~350 jobs per day for DBoD and ~ 900 jobs for MiddleWareOnDemand Modular design with focus on expansion Easy to add support for new systems (MiddleWareOnDemand) Reusable code

15. DBOD State Checker 15 Part of the daemon package. Cron managed script which periodically checks each instance availability and accordingly updates its status in the DB Necessary to correctly display externally caused changes to the status of the service instances (e.g. host downtime, network issues, etc) in the user interface

16. Migration to the CERN Agile Infrastructure 16 IT-DB Virtualization infrastructure is being migrated from RHEL + OVM to the standard CERN AI OpenStack setup (KVM + SLC) Storage access performance is vital to DB applications IT-DB runs its own OpenStack installation on servers physically connected to its storage servers for performance reasons

17. Migration to the CERN Agile Infrastructure 17 DBOD customized RPM packages for MySQL and PostgreSQL servers already built using Koji A Puppet module configures each host according to the instance-resource relations stored on the Syscontrol LDAP directory NAS Volumes, service startup scripts, users, etc.

18. Agenda 18 Manifesto Current status Architecture Demo Management Infrastructure High Availability Monitoring Data protection: backups and recoveries in detail Future development Summary

19. High Availability 19 Driven by demand/need, not initially in the plans Not relying on virtualization features so far (it may change in the future, as OpenStack evolves) 4 node clusters Nowadays two clusters running under Oracle cluster ware 12.1.0.1. Clusterware controls: Virtual IP RDBMS instance PostgreSQL and MySQL instances can co-exist, different versions supported.

20. High Availability 20 For instances running on an Oracle cluster ware, care must be taken in case of server crash for MySQL instances. "InnoDB: Unable to lock./ibdata1, error: 11" Error Sometimes Seen With MySQL on NFS (Doc ID 1522745.1) Failover test\DowntimeAvg. (s)Min. (s)Max(s) Kill process16.9439 Kill process (different node)21.71034 Network down39.93747 Server down373343 Relocate6.257 Testing the cluster (MySQL & Postgresql instances)

21. Agenda 21 Manifesto Current status Architecture Demo Management Infrastructure High Availability Monitoring Data protection: backups and recoveries in detail Future development Summary

22. 22 Infrastructure: Hardware servers Dell blades PowerEdge M610 2x Quad-Core Intel Xeon @ 2.53GHz 48 GB RAM Transtec Database server 2x eight-core Intel(R) Xeon(R) CPU E5-2650 0 @ 2.00GHz 128 GB RAM NetApp cluster Next release 10GbE Public Network Private Network

23. Agenda 23 Manifesto Current status Architecture Demo Management Infrastructure Monitoring Data protection: backups and recoveries in detail Future development Summary

24. Monitoring 24 Critical component for both DBoD managers and DBAs (our clients) Different monitoring tools, two main categories Servers, OS, Storage → + Scripts + Netapp tools DBoD instance: + RACmon Trying to outsource DBoD instance monitoring Very demanding task: configure alerts, adapt to new RDBMS releases, add functionality like reporting, execution plans,etc.

25. Monitoring 25 Three main axes IT GNI service, IT dashboards Oracle being monitored by → fine grained access per pluggable database in Oracle12c MySQL, PostgreSQL monitored by Very intuitive interface Database – storage volumes – protocol correlation DB activity view and SQL analysis

26. Monitoring (nowadays view) 26

27. Monitoring: Enterprise manager 12c 27 Pluggable database Container database

28. Monitoring: AppDynamics 28

29. Monitoring: AppDynamics 29

30. 30 Monitoring: AppDynamics

31. Agenda 31 Manifesto Current status Architecture Demo Management High Availability Monitoring Data protection: backups and recoveries in detail Future development Conclusions

32. Storage evolution 32 FAS3240FAS8060 NVRAM1.0 GB8.0 GB System memory8GB64GB CPU1 x 64-bit 4-core 2.33 Ghz2 x 64-bit 8-core 2.10 Ghz SSD layer (maximum) 512GB8TB Aggregate size180TB400TB OS controllerData ONTAP® 7-mode Data ONTAP® C-mode* scaling up scaling out * Cluster made of 8 controllers (FAS8060 & FAS6220). Shared with other services.

33. Data protection 33 2 file systems: data + redo logs on different Netapp appliances Storage is monitored: Netapp tools + home made tools Multipath access to disks (redundancy + performance) → disks are seen by two controllers (HA pair) → Transparent interventions RAID6 Automatic scrubbing (based on checksum) Rapid RAID Recovery + Disk Maintenance Center

34. 34 Backup management Same backup procedure for all RDBMS. Only data volume is snapshot. Backup workflow: mysql> FLUSH TABLES WITH READ LOCK; mysql> FLUSH LOGS; or Oracle>alter database begin backup; Or Postgresql> SELECT pg_start_backup('$SNAP'); mysql> UNLOCK TABLES; Or Oracle>alter database end backup; or Postgresql> SELECT pg_stop_backup(), pg_create_restore_point('$SNAP'); snapshot resume … some time later new snapshot

35. 35 Snapshots Taken programmatically via our API using ZAPI ( NetappManagementSDK) Logs can be controlled via DB On Demand site It is a very fast operation. Example: Hc_atlas (MySQL) about 3 secs Indico_p (PostgreSQL) about 3 secs

36. Tape backups 36 Driven by demand/need, not initially in the plans Likely to be removed Possible only on PostgreSQL and MySQL Oracle12c solution comes already with a tape backup! Consistent snapshot + redo logs sent to tape Database activity is not impacted Tape backups are not validated Manual process to set them up, need to contact us (DBOD + TSM service)

37. 37 Instance restore 37 Binary logs TIME Data files Manual snapshot Now Automatic snapshots Point-in-time recovery

38. Agenda 38 Manifesto Current status Architecture Demo Management High Availability Monitoring Data protection: backups and recoveries in detail Future development Summary

39. 39 High density consolidation: LXC Scaling up servers (128GB RAM, 32 CPUs), LXC should help to consolidate even more. Red Hat 7 Atomic Host Fine control on memory and CPU using control groups MySQL 5.5.30 - sysbench 0.5 query test - data set fits into innodb buffer

40. 40 Data protection: SnapVault Based on snapshots It should cover tape backup functionality → Disaster Recovery location *Image taken from Netapp documentation.

41. Agenda 41 Manifesto Current status Architecture Demo Management Monitoring Data protection: backups and recoveries in detail Future development Summary

42. 42 Summary Many lessons learned during the design and implementation of the DBoD service Building Database as a Service helped CERN DB group to Gain experience with MySQL, PostgreSQL and multi-tenancy Oracle 12c Provide a solution for Oracle database with special needs e.g. Unicode character sets Improve tools and operations Standardize on tools and frameworks Consolidate Face new use cases from CERN community e.g. Increase data protection On-going integration with IT central services

43. 43 Acknowledge IT-DB colleagues Our former colleagues Daniel Gomez Blanco and Dawid Wojcik Ignacio Coterillo and David Collados as members of DBoD team

44. Questions 44