1. DB-5: Simulating User Load Tom Bascom VP Technology White Star Software tom@wss.com

2. DB-5: Simulating User Load - 2 Sanity Check! n This Presentation is NOT about simulating your user interface. n We are going to discuss a method for creating a database load that mimics a target usage profile.

3. DB-5: Simulating User Load - 3 The Challenge n A Volatile Business Environment n System Load expands to consume all resources… n Something is Always changing… n Fighting Misconceptions: –Load testing is too difficult. –Load simulation tools are expensive. –Load testing tools don’t support Progress.

4. DB-5: Simulating User Load - 4 Why Simulate User Load? n Regression Testing n Finding the Limits / Capacity Planning n Root Cause Analysis n Minimizing Cost / Optimizing Resources n Quality of Service (QoS) Assurances n To support other testing efforts more realistically

5. DB-5: Simulating User Load - 5 Considerations of Fidelity n Assumptions n Characterizing the Load n Where are the Bottlenecks? n Users are fickle… n How do we (realistically) drive test scenarios? (Or: Are we really simulating user load?)

6. DB-5: Simulating User Load - 6 Assumptions n We are primarily interested in the DATABASE. n READ activity is, by far, the largest consumer of database resources. n WRITE activity is also important. n An ABL session is an ABL session… (in terms of resource use) n The functionality of the application is not being tested (by these tests).

7. DB-5: Simulating User Load - 7 Characterizing the Load n Historical data collection is a must. n Application Metrics –Orders Taken –Widgets Produced n Baseline db Performance –Logical IO ops/sec –Disk IO ops/sec –Latch & Resource Waits –TRX, Record Locks & Blocked Clients n IO by User n TableStats

8. DB-5: Simulating User Load - 8 Where are the Bottlenecks? n Database –Latch contention –Transaction throughput n Operating System –Disk –CPU –Memory –Network

9. DB-5: Simulating User Load - 9 Users are fickle… n They work on different things… n at different rates … n pretty much randomly … n and sometimes at their leisure ;-) n Our best friends are: –IO by User (PROMON, ProTop) –_TableStat (VST, ProTop)

10. DB-5: Simulating User Load - 10 IO Requests by User

11. DB-5: Simulating User Load - 11 IO Requests by Table

12. DB-5: Simulating User Load - 12 IO Requests by Table Create Read Update Delete 5,940,439 1,662,688,907 2,540,165 2,500,292 0.35% 99.34% 0.15% 0.15% 83.8/s 23,454.5/s 35.8/s 35.3/s Table Name Created % Read % Updated % Deleted % CHART_ENTRY 0 0.0 1,061,326,658 63.8 0 0.0 0 0.0 BUREAU_QUEUE 0 0.0 241,087,053 14.5 25,150 0.9 0 0.0 APP_INFO 4,711 0.0 82,913,516 4.9 213,944 8.4 26 0.0 BTE_QUEUE 0 0.0 27,612,013 1.6 14,312 0.5 0 0.0 SCORE_EXPWGHT 0 0.0 20,349,269 1.2 0 0.0 0 0.0 APP_REQUEST 4,711 0.0 13,491,724 0.8 45,119 1.7 668 0.0 APP_TRADE 105,260 1.7 11,499,880 0.6 143,465 5.6 20,672 0.8 APP_OPTION 23,791 0.4 11,341,471 0.6 50,809 2.0 9,614 0.3 APP_TRANSTAGG 3,347,503 56.3 8,603,941 0.5 0 0.0 1,919,357 76.7

13. DB-5: Simulating User Load - 13 Data Distribution

14. DB-5: Simulating User Load - 14 Idle Users? Current Users: 192 Idle Users: 0:01 18 0:02 8 0:03 10 0:04 4 0:05 4 0:06 2 0:07 4 0:08 4 0:09 1 10m - 1hr: 51 Hour+ old: 32

15. DB-5: Simulating User Load - 15 How do we drive test scenarios? n With some ABL code that reads and writes records as the application does. n Distributed among many sessions. n Profiled according to TableStats.

16. DB-5: Simulating User Load - 16 Code !!!

17. DB-5: Simulating User Load - 17 WARNING n Using the CREATE, UPDATE or DELETE routines will modify your database! n Use these destructive options only with test copies of your database. n Use the READ-ONLY option with production databases!

18. DB-5: Simulating User Load - 18 A Simulation Controller n You already have everything that you need… n A generic Load driver n Tuning the driver to your Load n Controlling test runs

19. DB-5: Simulating User Load - 19 Control Program (pacectl.p) cmd = “mbpro &1 -pf pace.pf -param &2 -U &3". input from value( "io.usr" ). do while true: import io_usr nm_usr. xcmd = substitute( cmd, pdbname(1), io_usr, nm_usr ). message xcmd. if opsys = "unix" then os-command silent value( xcmd ). else run spawn( input xcmd, input "", output pid ). pause 1. end.

20. DB-5: Simulating User Load - 20 Config File (io.usr) 0,500,50,0 jami 0,100,10,0 tucker 0,50,5,0 jami 0,40,4,0 julia 0,30,3,0 peter 0,20,2,0 emily

21. DB-5: Simulating User Load - 21 pace.p do while true: x = random( 1, pickLimit ). find first tt_pace where tblPick >= x. run xread( tblName, r_rate ). t = time - s. do while (( i / t ) > r ): pause 1 no-message. t = time - s. end.

22. DB-5: Simulating User Load - 22 xread() create buffer bh for table tblName. qh:set-buffers( bh ). qh:query-prepare( “for each “ + tblName ). j = random( j / 2, j * 10 ). do while k < j: qh:get-next( no-lock ) no-error. if qh:query-off-end then do: qh:get-first( no-lock ) no-error. next. end. k = k + 1. end.

23. DB-5: Simulating User Load - 23 Demo !!!

24. DB-5: Simulating User Load - 24 Selected Case Studies n Benchmarking OpenEdge 10.0 –PSDN Whitepaper: http://www.psdn.com/library/entry!default.jspa?categoryID=256&externalID=876 n Ensuring Disk Array Performance n Verifying a Failover Plan n Validating Parameter Changes n Providing a background load for development or testing environments. n Supporting Demos ;-)

25. DB-5: Simulating User Load - 25 Benchmarking OE10 n PSDN Whitepaper: http://www.psdn.com/library/entry!default.jspa?categoryID=256&externalID=876 n Problem: –How Does OE10 Compare to Progress 9?

26. DB-5: Simulating User Load - 26 Ensuring Disk Array Performance n Problem: –Will a new disk array configuration support the production workload? n Several previous attempts to go live had failed. n Load simulation was used to prevent a repeat of that fiasco. n Problems were found and solutions were verified and confidence was repaired prior to a successful implementation.

27. DB-5: Simulating User Load - 27 Verifying a Failover Plan n Problem: –Redundant (and expensive) systems were purchased and configured for mutual failover. n A method to realistically test a failover with significant load on the hardware was needed. n The failover worked very well! n But the “fail back” failed miserably  n As a result a serious OS bug was identified and fixed and downtime avoided.

28. DB-5: Simulating User Load - 28 Validating Parameter Changes n Problem: –Changing many Progress and OS parameters can result in unanticipated limits being reached. n # of file handles, semaphores, memory size etc… n Solution: –Use the load simulator to start sessions and stress these limits.

29. DB-5: Simulating User Load - 29 Background Load n Problem: –Developers and testers are often working on over-powered servers with no competition for resources. –Which leads to a false sense of good performance when testing. n Solution: –Run load simulator to provide a continuous background load.

30. DB-5: Simulating User Load - 30 Demos! n Problem: –Single user demos are boring. n Solution: –Create some fake users that do interesting things.

31. DB-5: Simulating User Load - 31 Summary n Basic Load Simulation is easier than you think! n And doesn’t necessarily require fancy (and expensive) tools. n It does require fore-thought and planning. n And can be immediately useful to you!

32. DB-5: Simulating User Load - 32 Questions http://www.greenfieldtech.com/downloads.shtml