2. Web Performance, Load, and Stress Tests Mainframe and Open System Network Natural Conference Boston Dieter W. Storr Los Angeles Times May 2004 info@storrconsulting.com

3. May 2004Dieter W. Storr -- info@storrconsulting.com 2 Performance, Load, and Stress Tests Contents çWhat are performance, load, and stress tests? çWhy and when should you test? çWhat response time do you expect? çWhat parts play a role in the network? çNetwork basics çLogical and physical transactions

4. May 2004Dieter W. Storr -- info@storrconsulting.com 3 Performance, Load, and Stress Tests Contents çWhat tools can you use to test these parts? çWhen should you start performance tests? çWeb site performance, analysis and tuning çChecklist çLinks

5. May 2004Dieter W. Storr -- info@storrconsulting.com 4 What are Performance Tests? çTest each part of the system, such as web server, web applications, databases, etc. çTests can run usually without requesting page images in order to concentrate the testing on the script and code itself ? ? ? ?

6. May 2004Dieter W. Storr -- info@storrconsulting.com 5 What are Load Tests? çLoad tests are performed by testing the website using the best estimate of the traffic your website needs to support. çDefine the maximum time it should take for a page to load -- from a usability and customer experience standpoint, not from a technical one. ? ? ? ?

7. May 2004Dieter W. Storr -- info@storrconsulting.com 6 What are Stress Tests? çTo simulate brute force attacks that apply excessive load to system, such as web server, web applications, databases, etc. çTo estimate the maximum load that your system can support çTo learn the traffic thresholds of your system çHow it will respond after exceeding its threshold ? ? ? ?

8. May 2004Dieter W. Storr -- info@storrconsulting.com 7 What are Stress Tests? Ramp test çA variation of stress test çThe number of users is increased over the life of the test ? ? ? ?

9. May 2004Dieter W. Storr -- info@storrconsulting.com 8 Why Performance Tests? çDetermine that all functions do what they meant to do (function test) çEnsure that tasks and objectives are easily to be accomplished (usability test) çGuarantee a result from the website in an acceptable time (performance test)

10. May 2004Dieter W. Storr -- info@storrconsulting.com 9 Why Performance Tests? çOptimize each part of the system for increased web traffic çEnsure that the end-users mouse click is not met with silence

11. May 2004Dieter W. Storr -- info@storrconsulting.com 10 Why Load Tests? çTo compare the maximum time it should take for a page to load with the effective used time. çCalculate the impact of exceeding that maximum time. Will you lose sales or prospective customers? çCalculate the anticipated load and load pattern for the website. çDetermine to do some work on the application or webserver.

12. May 2004Dieter W. Storr -- info@storrconsulting.com 11 Why Stress Tests? çDetermine the maximum load çTest excessive system load çDetermine the traffic thresholds of your system çExperience the reaction of the system after exceeding its threshold çRamp tests can help to determine what maximum load a server can handle while providing optimal access to web resources

13. May 2004Dieter W. Storr -- info@storrconsulting.com 12 When Performance Tests? çYou cannot start early enough! çStart before a single line of code is written ç Network ç Load balancer ç Application ç Database ç Web servers çDuring development, all software engineers and QA team should have access to performance test tools

14. May 2004Dieter W. Storr -- info@storrconsulting.com 13 When Load Tests? çAs soon as web pages are working çQA team should conduct the first load tests çPerformance testing should be part of the regular testing routine each day for each build of the software

15. May 2004Dieter W. Storr -- info@storrconsulting.com 14 What Response Time Do You Expect? çDoes your company set limits? çKEEP in mind: çMany websites today have a serious business mission => to take money çReason: Advertisement opportunities, selling retail products, etc. çWebsites and applications need to be up and running at all times

16. May 2004Dieter W. Storr -- info@storrconsulting.com 15 What Response Time Do You Expect? Source: Jakob Nielsen, Designing Web Usability, New Riders Publishing, 2000

17. May 2004Dieter W. Storr -- info@storrconsulting.com 16 What Response Time Do You Expect? è95% of all web requests should be processed in less than 10 seconds! BE AWARE: If performance slows even a little, fickle web users are likely to jump quickly to a competitors site

18. May 2004Dieter W. Storr -- info@storrconsulting.com 17 What Parts Play a Role in The Network? ADABAS NATURAL CGI Web Server Web Browser Basic Connections

19. May 2004Dieter W. Storr -- info@storrconsulting.com 18 What Parts Play a Role in The Network? çHubs çSwitches çRouters çFirewalls çLoad Balancer çApplication Servers çDatabase Servers çWeb Servers

20. May 2004Dieter W. Storr -- info@storrconsulting.com 19 Network Basics Hub / Switch / Bridge / Router / Firewall çIn data communications, a hub is a place of convergence where data arrives from one or more directions and is forwarded out in one or more other directions. A hub usually includes a switch of some kind. See USB 7-Port Hub To connect several USB devices

21. May 2004Dieter W. Storr -- info@storrconsulting.com 20 Network Basics Hub / Switch / Bridge / Router / Firewall çIn telecommunications, a switch is a network device that selects a path or circuit for sending a unit of data to its next destination. See 8-Port Network Switch

22. May 2004Dieter W. Storr -- info@storrconsulting.com 21 Network Basics Hub / Switch / Bridge / Router / Firewall çA switch may also include the function of the router, a device or program that can determine the route and specifically what adjacent network point the data should be sent to. çIn general, a switch is a simpler and faster mechanism than a router, which requires knowledge about the network and how to determine the route

23. May 2004Dieter W. Storr -- info@storrconsulting.com 22 Network Basics Hub / Switch / Bridge / Router / Firewall çIn telecommunication networks, a bridge is a product that connects a local area network (LAN) to another local area network that uses the same protocol (for example, Ethernet or token ring).

24. May 2004Dieter W. Storr -- info@storrconsulting.com 23 Network Basics Hub / Switch / Bridge / Router / Firewall çOn the Internet, a router is a device or, in some cases, software in a computer, that determines the next network point to which a packet should be forwarded toward its destination. See 4-Port Cable/DSL Router

25. May 2004Dieter W. Storr -- info@storrconsulting.com 24 Network Basics Hub / Switch / Bridge / Router / Firewall çA system designed to prevent unauthorized access to or from a private network. Firewalls can be implemented in both hardware and software, or a combination of both. PIX Firewall from CISCO Private Internet Exchange Firewall Gauntlet Firewall from NAI Network Associates Inc.

26. May 2004Dieter W. Storr -- info@storrconsulting.com 25 Network Basics Bridge

27. May 2004Dieter W. Storr -- info@storrconsulting.com 26 Web Server Firewall Appl. Server Internet Screen Scraper Firewall Com-plete NATURAL ADABAS CICS COBOL VSAM Firewall SSL Oracle FirstLogic DBs Web Server Appl. Server Screen Scraper CISCO Router e.g. Apache e.g. WebLogic BEA e.g. Attachmate Main- frame Open System

28. May 2004Dieter W. Storr -- info@storrconsulting.com 27 Logical and Physical Transactions? ADABAS NATURAL Appl. Server Web Server Web Browser

29. May 2004Dieter W. Storr -- info@storrconsulting.com 28 Response Time / Click Time Tests ADABAS NATURAL Appl. Server Web Server Web Browser Function: Display my account

30. May 2004Dieter W. Storr -- info@storrconsulting.com 29 What Tools Can You Use to Test? Web Server Stress Tool çUser simulation çLogging period çSelect URLs çURL recorder çData merging çBrowser simulation çReviewing summary and detailed log results çAnalyzing graphical results Web Browser

31. May 2004Dieter W. Storr -- info@storrconsulting.com 30 CSS Stress Test 5 Users 10 Clicks 25 Sep 2003, 11:52-11:53

32. May 2004Dieter W. Storr -- info@storrconsulting.com 31 CSS Stress Test 5 Users 10 Clicks 14 Oct 2003, 14:05

33. May 2004Dieter W. Storr -- info@storrconsulting.com 32 09/25/2003 11:52:02 AM --- WEBSERVER STRESS TOOL User #1 Click #1 started, used URL #1 (CSS-Prod-1) POST-DATA= "email=dieter.storr@latimes.com&password= User #1 Click #1: Request 1 HTTP State: Initialized (0 ms) User #1 Click #1: Request 1 HTTP State: DNS Lookup begin (7 ms) User #1 Click #1: Request 1 HTTP State: DNS Lookup done (17 ms) User #1 Click #1: Request 1 HTTP State: Connected (504 ms) User #1 Click #1: Request 1 HTTP State: Waiting for header (512 ms) User #1 Click #1: Request 1 HTTP State: Receiving Header (5937 ms) (=Time to First Byte) User #1 Click #1: Request 1 HTTP State: Waiting for body (5947 ms) User #1 Click #1: Request 1 HTTP State: Done (5999 ms) User #1 Click #1: CLICK-Request 1: Time=6008 ms, TFB=5937ms, Bytes=16038, HTTP-StatusCode=200 (OK) User #1 Click #1: !Success! All requests of click done (6046 ms)

34. May 2004Dieter W. Storr -- info@storrconsulting.com 33 What Tools Can You Use to Test? APAS/INSIGHT ADABAS DWSCWUSE: SUMMARIZE (LS=150) COUNT (HD='COMMANDS') PERCENT (HD='% OF TOTAL' PF=G10) SUM(TOTAL-IO) (HD='TOTAL IO') PCT(TOTAL-IO) (HD='% OF TOTAL' PF=G10) MEAN(TOTAL-IO) (HD='AVG IO') MAX(TOTAL-IO) (HD='MAX IO') MEAN(DURATION) (HD='AVG DUR') MAX(DURATION) (HD='MAX DUR') MEAN(ENQ-TIME) (HD='AVG ENQ') MAX(ENQ-TIME) (HD='MAX ENQ') BY USER-ID WHERE USER-ID(2) = 'CW' INSIGHT-LINES = 30 PRINT TO STRESS01;

35. May 2004Dieter W. Storr -- info@storrconsulting.com 34 USER ADABAS ID COMMANDS % OF TOTAL -------- ---------- ----------- CW0016 269 |====== CW0017 269 |====== CW0018 269 |====== CW0019 369 |========= CW0020 369 |========= CW0021 369 |========= CW0022 369 |========= CW0023 369 |========= CW0024 269 |====== CW0025 269 |====== ---------- ----------- ======== 3,190 |========> AVG DUR SEC MAX DUR SEC ------------ --------- 0.00033 0.006 0.00052 0.008 0.00036 0.005 0.00058 0.014 0.00041 0.011 0.00043 0.008 0.00045 0.008 0.00038 0.007 0.00034 0.006 0.00049 0.009 ------------ --------- 0.014 ADABAS on Enterprise Server CSS Stress Test 5 Users 10 Clicks 25 Sep 2003, 11:52-11:53

36. May 2004Dieter W. Storr -- info@storrconsulting.com 35 What Tools Can You Use to Test? çAPAS/INSIGHT ADABAS 0090 DWSCWOBX: EXTRACT (LS=150) 0100 NAT-LOAD-COUNT (HD='OBJ-LOAD') 0110 DATE TIME 0120 NAT-PROGRAM CMD 0130 USER-ID 0140 GLOBAL-ID 0150 WHERE USER-ID(2) = 'CW' AND 0160 (NAT-PROGRAM = 'IWS00P00' OR 0170 NAT-PROGRAM = 'IWD10P10') 0180 INSIGHT-LINES = 200 0190 PRINT TO STRESS01 0200 ;

37. May 2004Dieter W. Storr -- info@storrconsulting.com 36 NATURAL PROGRAM OBJ-LOAD -------- -------- IWD10P10 50 -------- 50 ADABAS on Enterprise Server CSS Stress Test 5 Users 10 Clicks 25 Sep 2003, 11:52-11:53

38. May 2004Dieter W. Storr -- info@storrconsulting.com 37 Response Time / Click Time Tests ADABAS NATURAL Appl. Server Web Server Web Browser Function: Display my account 200ms - 20s 0.00033s - 0.014s

39. May 2004Dieter W. Storr -- info@storrconsulting.com 38 What Tools Can You Use to Test? STRESS -- self-written ADABAS 1 Start process with heavy CPU workload 2 Start process with many ADABAS Calls 3 Start process with heavy CPU workload and many ADABAS calls 4 Start process with many ADABAS calls and excessive terminal I/O's 5 Start asynchronous processes (ADABAS & CPU)

40. May 2004Dieter W. Storr -- info@storrconsulting.com 39

41. May 2004Dieter W. Storr -- info@storrconsulting.com 40

42. May 2004Dieter W. Storr -- info@storrconsulting.com 41

43. May 2004Dieter W. Storr -- info@storrconsulting.com 42

44. May 2004Dieter W. Storr -- info@storrconsulting.com 43

45. May 2004Dieter W. Storr -- info@storrconsulting.com 44 What Tools Can You Use to Test? Com-plete çUCTRL çNETSPY çTMON/MVS TP Monitor

46. May 2004Dieter W. Storr -- info@storrconsulting.com 45 UCTRL 17:49:14 TID 6 COMTEST USER ZINDWS 03/16/04 --- COM-PLETE CONTROL FACILITY --- UCT1 Function FC Function FC Function FC -------------------- -- -------------------- -- -------------------- -- ADABAS interface... AI Program lookaside.. IP Task groups........ PG ADABAS statistics.. AS Program in-stor dir. OP Task list.......... PL Buffer activities.. BA Resident programs.. RP Thread groups...... TG Common storage..... CS Find program....... FP Thread subgroups... SG DB2 thread status.. D2 Queue overview..... QO Thread list........ TL FBPM statistics.... FB Roll activities.... RA User activities.... UA Message log........ CM Server overview.... SO User status......... US Terminal overview.. TO VSAM statistics.... VS Terminal status.... TS Print statistics... PR Select function.... Operand............ Operator command: Enter-PF1---PF2---PF3---PF4---PF5---PF6---PF7---PF8---PF9---PF10--PF11--PF12--- End

47. May 2004Dieter W. Storr -- info@storrconsulting.com 46 NETSPY

48. May 2004Dieter W. Storr -- info@storrconsulting.com 47 TP Monitor Com-plete Consumption

49. May 2004Dieter W. Storr -- info@storrconsulting.com 48 TMON çAlerts çAverage response time çCPU usage çI/O usage Number of ended transactions rate

50. May 2004Dieter W. Storr -- info@storrconsulting.com 49 What Tools Can You Use to Test? Attachmate Run Time Application çManagement & Control Services (MSC) -- monitoring pools and sessions çJava console running in foreground Screen Scraper

51. May 2004Dieter W. Storr -- info@storrconsulting.com 50 What Tools Can You Use to Test? Attachmate Run Time Application çNumber of sessions to host (mainframe ) çMaximum sessions çNumber of failures çQueue size Screen Scraper

52. May 2004Dieter W. Storr -- info@storrconsulting.com 51 What Tools Can You Use to Test? Attachmate Servers çNT utilities çSampling at 10-second intervals Screen Scraper

53. May 2004Dieter W. Storr -- info@storrconsulting.com 52 What Tools Can You Use to Test? Attachmate Servers Analyze results: çCPU utilization çMemory utilization çRun queues çSwap space utilization çDisk I/O - most busy which disk hit the hardest çI/O wait > 10% çPage file - swaps Screen Scraper

54. May 2004Dieter W. Storr -- info@storrconsulting.com 53 What Tools Can You Use to Test? Cisco 2900 Switch çManual testing/sampling Analyze results çLoad on network components çPacket throughput Content Server Switch

55. May 2004Dieter W. Storr -- info@storrconsulting.com 54 What Tools Can You Use to Test? Cisco 4000 Router (10BaseT) çManual testing/sampling Analyze results çLoad on network components http://www.scd.ucar.edu/nets/devices/routers/ios_analy sis.html çPacket throughput çRouter Configuration: http://www.swcp.com/~jgentry/topo/cisco.htm Router

56. May 2004Dieter W. Storr -- info@storrconsulting.com 55

57. May 2004Dieter W. Storr -- info@storrconsulting.com 56 What Tools Can You Use to Test? Oracle çOracle utilities çApplication-specific log files çPerformance Monitoring and Tuning Tools TKProf UTLBSTAT.SQL and UTLESTAT.SQL - Begin and end stats monitoring Statspack Oracle Enterprise Manager - Tuning Pack RDBMS

58. May 2004Dieter W. Storr -- info@storrconsulting.com 57 What Tools Can You Use to Test? Oracle çPerformance Monitoring and Tuning Tools (cont) DBXRAY: A real-time graphical snapshot of the health of your Oracle database- anytime, anywhere. Patrol: Intelligent and non-intrusive for proactive database monitoring, including Oracle Parallel Server and Oracle Symmetric Replication environments. SQL-Explorer for Oracle The most advanced SQL tuning tool available. Identify, model and tune problematic SQL statements to boost application performance. RDBMS

59. May 2004Dieter W. Storr -- info@storrconsulting.com 58 What Tools Can You Use to Test? Oracle Analyze results: çNumber of Oracle processes çBuffer hit cache çBlock I/O rate çNumber of cursors created çNumber of connections çLog information regarding number of registrations, starts, etc. RDBMS

60. May 2004Dieter W. Storr -- info@storrconsulting.com 59 What Tools Can You Use to Test? SSL Accelerator Secure Socket Layer (protocol between HTTP and TCP) -- appliance or card çManual testing/sampling Analyze results: çLoad on network components çPacket throughput Optimizing SSL processing for Web çhttp://www.nwfusion.com/news/tech/2000/1023tech.html?nf SSL

61. May 2004Dieter W. Storr -- info@storrconsulting.com 60 What Tools Can You Use to Test? PIX çManual testing/sampling Analyze results: çLoad on network components çPacket throughput Firewall

62. May 2004Dieter W. Storr -- info@storrconsulting.com 61 What Tools Can You Use to Test? Gauntlet (several) çManual testing/sampling Analyze results: çMemory çDisk I/O çFull system log Firewall

63. May 2004Dieter W. Storr -- info@storrconsulting.com 62 What Tools Can You Use to Test? Sun OS çSolaris utilities çSampling at 10-second intervals Web Server

64. May 2004Dieter W. Storr -- info@storrconsulting.com 63 What Tools Can You Use to Test? Sun OS Analyze results: çCPU utilization çMemory utilization çRun queues çSwap space utilization çDisk I/O -- most busy which disk hit the hardest çI/O wait > 10% çPage file -- swaps Web Server

65. May 2004Dieter W. Storr -- info@storrconsulting.com 64 What Tools Can You Use to Test? Apache Application çLog Analyzer Analyze results: çNumber of sessions çBytes per session çSource address çDestination address Web Server

66. May 2004Dieter W. Storr -- info@storrconsulting.com 65 What Tools Can You Use to Test? Sun OS çSolaris utilities çSampling at 10-second intervals Application Server

67. May 2004Dieter W. Storr -- info@storrconsulting.com 66 What Tools Can You Use to Test? Sun OS Analyze results: çCPU utilization çMemory utilization çRun queues çSwap space utilization çDisk I/O -- most busy which disk hit the hardest çI/O wait > 10% çPage file -- swaps Application Server

68. May 2004Dieter W. Storr -- info@storrconsulting.com 67 What Tools Can You Use to Test? BEA WebLogic Application çWebLogic administrative console çSpotlight Application Server

69. May 2004Dieter W. Storr -- info@storrconsulting.com 68 What Tools Can You Use to Test? BEA WebLogic Application Analyze results: çNumber of active queues çNumber of connections çNumber of active sockets çTypes of access çTime of access Application Server

70. May 2004Dieter W. Storr -- info@storrconsulting.com 69 Web Site Performance Analysis and Tuning çIn-House or Web Caching çNear the network edge, e.g. proxy cache or browser: static HTML, JavaScript, and image files çCache-control header no-cache re-fetch expiry date çCompression çHTML is compressed before being sent through the network.

71. May 2004Dieter W. Storr -- info@storrconsulting.com 70 Web Site Performance Analysis and Tuning çSSL Acceleration (box or card) çConsider the number of transactions per second (TPS). TPS will be more important on SSL Acceleration appliances than cards. çAn SSL Acceleration hardware appliance (box) can deal with transactions for all your web servers and scale quickly. çNetwork Load Balancing çRound Robin çLowest response time çLeast connections çProximity

72. May 2004Dieter W. Storr -- info@storrconsulting.com 71 Checklist -- Getting Started? çDetermine your environment Mainframe, Open System, Middleware çEvaluate Stress Test Software Write your own or buy one çUnderstand the different components of a web site download DNS lookup Redirect First byte download Base page download Content download

73. May 2004Dieter W. Storr -- info@storrconsulting.com 72 Checklist -- Getting Started? çWhat kind of content? Dynamic or static.html or image çNumber of servers çWhere does traffic comes from Internet, Intranet, or Extranet

74. May 2004Dieter W. Storr -- info@storrconsulting.com 73 Create a Stress Test Plan çTest Organization Overview Participants Technical team Coordination team Application team Objectives Infrastructure Application Assumptions / pretest conditions Communication plan

75. May 2004Dieter W. Storr -- info@storrconsulting.com 74 Create a Stress Test Plan çEnvironment / Test Components Overview Overall success criteria Transactions to be tested Logical and physical Application functions Data indentification Script development High-level system diagram Components Responsibility Metrics Tool / method

76. May 2004Dieter W. Storr -- info@storrconsulting.com 75 Create a Stress Test Plan çSequence of Events Summary Process Schedule of events

77. May 2004Dieter W. Storr -- info@storrconsulting.com 76 Web Speed is no Magic!

78. May 2004Dieter W. Storr -- info@storrconsulting.com 77 Links to Interesting Websites çhttp://www.edwardtsai.com/web-site- performance.html çhttp://www.nwfusion.com/news/tech/2000/102 3tech.html?nf çhttp://www.scd.ucar.edu/nets/devices/routers/ ios_analysis.html çhttp://www.swcp.com/~jgentry/topo/cisco.htm

79. May 2004Dieter W. Storr -- info@storrconsulting.com 78 Thank You!