1. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Load testing & benchmarks on Oracle RAC Romain Basset – IT PSS DP

2. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Load testing & Benchmarking Framework for Oracle RAC - 2 Outline Motivations DB load testing & benchmarks –Existing solutions The application –Description –Examples Conclusion

3. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Motivations Database workload is rising until LHC startup. → Database architecture should be tested to study performance and availability issues. Internal software (COOL, CORAL) has to be tested in production-like environment. Load testing & Benchmarking Framework for Oracle RAC - 3

4. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Atlas example Load testing & Benchmarking Framework for Oracle RAC - 4

5. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t DB load testing & benchmarks To ensure the quality of service, three main aspects should be looked at : –Performance –Scalability –Reliability Benchmarking is focusing on the first two. Load testing aims at proving the last one. Load testing & Benchmarking Framework for Oracle RAC - 5

6. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t TPC TPC : Transaction Processing Performance Council Benchmark specifications –TPC-C : Order-Entry system (OLTP) select, update, insert & delete –TPC-H : Decision support system complex select queries & few insert/delete statements No out-of-the-box implementation Simulated behaviors do not match with our use cases Load testing & Benchmarking Framework for Oracle RAC - 6

7. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t HammerORA Tcl/Tk open source software Possibility to replay a workload from Oracle trace files (10046) Load testing Implementation of TPC-like benchmarks –TPC-C and TPC-H Master/Slave module out of order No real monitoring / reporting feature No C++ support Load testing & Benchmarking Framework for Oracle RAC - 7

8. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Swingbench Free load generator & benchmarks Implementation of OLTP-like benchmarks Comprehensive graphical toolset for performance metrics and monitoring Distributed execution Not open source No C++ support Load testing & Benchmarking Framework for Oracle RAC - 8

9. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t SwingBench Load testing & Benchmarking Framework for Oracle RAC - 9

10. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Requirements Possibility to plug new tests (C++) Distributed execution Monitoring feature Performance report for analysis Load testing & Benchmarking Framework for Oracle RAC - 10

11. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Technologies Python AFS SSH XML HTML, PHP, JPGraph CORAL LEMON Oracle Load testing & Benchmarking Framework for Oracle RAC - 11

12. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Typical scenario 1.Spawn the clients 2.Query the DB 3.Write results on files (on AFS) 4.Gather results 5.Query the DB for statistics Load testing & Benchmarking Framework for Oracle RAC - 12

13. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Configuration Configuration file : - node1 node2 node3 server1 server2 - Object inserted Memory usage in MB - client.py testConf.xml Authentication.xml (CORAL): - - - Dblookup.xml (CORAL): - Load testing & Benchmarking Framework for Oracle RAC - 13

14. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Monitoring (RAC CPU) RAC CPU monitoring Get the values from Oracle : GV$SYSMETRIC Load testing & Benchmarking Framework for Oracle RAC - 14

15. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Monitoring (wait events) Average active sessions (OEM style) Get the values from Oracle : GV$WAITCLASSMETRIC Load testing & Benchmarking Framework for Oracle RAC - 15

16. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t LEMON LEMON plots for client and server nodes –Cpu –Load –Network (in & out) Linked or downloaded from LEMON web pages Load testing & Benchmarking Framework for Oracle RAC - 16

17. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Performance (client) Metric measured by the clients Get the values from the log files Load testing & Benchmarking Framework for Oracle RAC - 17

18. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Performance (aggregated) Aggregated measures from the clients Get the values from the log files Number of sessions (GV$SESSION) Load testing & Benchmarking Framework for Oracle RAC - 18

19. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Possibility to write new tests No specific language No specific metrics –Dynamic, time dependant (throughput) –Static, time independant (response time) Specific input –Client ID –Result file path –Result output frequency Specific output (CSV file) –Timestamp –Time elapsed since last result –Values for the different metrics Load testing & Benchmarking Framework for Oracle RAC - 19

20. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Examples CORAL : generic relational database access API for Physics applications COOL based on CORAL, defines a schema and provides C++ API to store/retrieve contitions data Load testing & Benchmarking Framework for Oracle RAC - 20

21. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Examples COOL performance tests –Insertions : storeObject() –Retrievals : findObjects(), browseObjects() –Benchmarking (using different COOL versions) –Tests on production sized dataset (using 1TB database ~1-year Atlas PVSS data) Stress testing using COOL insertion test –10 clients –4-node RAC –1TB of data in ~10 hours Load testing & Benchmarking Framework for Oracle RAC - 21

22. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t Conlusion Tests already conducted : –Performance when changing Oracle configuration –Performance in COOL package –Stress testing on 11g (by Balys) Tutorial available (on PSS wiki) Future work –Comparison of 2 test runs (new report) –Integration with COOL performance suite –Automatic checking of performance issues before a software release Load testing & Benchmarking Framework for Oracle RAC - 22

23. CERN IT Department CH-1211 Genève 23 Switzerland www.cern.ch/i t References Pro Oracle Database 10g RAC on Linux, J. Dyke and S. Shaw, chap. 5 & 23. TPC, http://tpc.org/ HammerORA, http://hammerora.sourceforge.net/ SwingBench, http://www.dominicgiles.com/swingbench.html CORAL, http://pool.cern.ch/coral COOL, http://cool.web.cern.ch/cool Oracle Load Testing Tools Documentation, https://twiki.cern.ch/twiki/bin/view/PSSGroup/OracleRACLoadTesting Tutorial, https://twiki.cern.ch/twiki/bin/view/PSSGroup/OracleRACLoadTestingTuto Load testing & Benchmarking Framework for Oracle RAC - 23