1. 8th Sakai Conference4-7 December 2007 Newport Beach Collaborative Performance Testing Healthy Medicine for Robust Sakai Code

2. Collaborative Performance Testing Using the example of Samigo, how performance testing can help developers find and fix performance defects before production.

3. Overview Introductions Some terms Michigan performance test setup Perspectives: –Tester –DBA –Systems Administrator –Developer Questions and Comments

4. Who are we? Lydia Li – Sakai Developer, Stanford University Chris Franz – Senior System Administrator, Unicon Drew Zhu – Oracle DBA, University of Michigan Chris Kretler – Performance Tester, University of Michigan

5. What does performance testing mean to me? Replication of middleware calls Tests have different goals: –“Load” –“Stress” –“Benchmark” –“Capacity Planning”

6. What does it mean, part II High data requirements –500 x 60 x 5 = 150,000 rows Response times Infrastructure metrics –CPU –Memory Concurrent “virtual” users

7. What it does not mean: A Simulation –Actual traffic is generated –System cannot tell the difference. GUI test High-volume functional test –Scope is narrower

8. When do we test? All major releases –New tools –“Significant” builds Hardware Evaluations Configuration Changes Sakai performance testing –Expand our scope

9. Michigan Test Environment Dedicated testing environment: –Load Balancers (Netscaler RS9000s) –8 application servers (Dell 2650s running RedHat) –1 Database server (Sun T2000 running 10g) System Administrators (Jeff, Adi, Rick, Ken, Jenny) Database Administrator (Drew) LoadRunner software

10. Samigo Setup Process Working Sakai build w/Samigo Determined requirements and focus Courses and students were created Sample tests imported Script creation –Lydia’s help was invaluable here.

11. Sample Script Portion Understanding expected result is crucial to script creation.

12. Load Test Results

13. DBA Tasks in Load Tests Monitoring Database and DB Server Produce AWR reports Analyze AWR Reports and Identify Issues Communicate with Developers, Testers, and Sysadmins Query Details and Analyzing Execution Plans Provide suggestions

14. Oracle OEM Screenshot -1

15. Oracle OEM Screenshot -2

16. Oracle OEM Screenshot -3

17. Samigo Load Test AWR CPU Elapsed CPU per % Total Time (s) Time (s) Executions Exec (s) DB Time SQL Id ---------- ---------- ------------ ----------- ------- ------------- 3,184 3,296 68,972 0.05 43.8 38xh90r4gx5cz select itemgradin0_.ITEMGRADINGID as ITEMGRAD1_139_, itemgradin0_.ASSESSMENTGRAD INGID as ASSESSME2_139_, itemgradin0_.PUBLISHEDITEMID as PUBLISHE3_139_, itemgra din0_.PUBLISHEDITEMTEXTID as PUBLISHE4_139_, itemgradin0_.AGENTID as AGENTID139_, itemgradin0_.SUBMITTEDDATE as SUBMITTE6_139_, itemgradin0_.PUBLISHEDANSWERID a 858 858 254,115 0.00 11.4 0byxm85k79suc select answerfeed0_.ANSWERID as ANSWERID1_, answerfeed0_.ANSWERFEEDBACKID as ANS WERFE1_1_, answerfeed0_.ANSWERFEEDBACKID as ANSWERFE1_137_0_, answerfeed0_.ANSWE RID as ANSWERID137_0_, answerfeed0_.TYPEID as TYPEID137_0_, answerfeed0_.TEXT as TEXT137_0_ from SAM_PUBLISHEDANSWERFEEDBACK_T answerfeed0_ where answerfeed0_.A 834 980 79,456 0.01 13.0 3px7qs284b3xk select answerset0_.ITEMTEXTID as ITEMTEXTID7_, answerset0_.ANSWERID as ANSWERID7 _, answerset0_.ANSWERID as ANSWERID136_6_, answerset0_.ITEMTEXTID as ITEMTEXTID1 36_6_, answerset0_.ITEMID as ITEMID136_6_, answerset0_.TEXT as TEXT136_6_, answe rset0_.SEQUENCE as SEQUENCE136_6_, answerset0_.LABEL as LABEL136_6_, answerset0_ 203 203 43,210 0.00 2.7 9xmw3p1px195k select count(1) from SAKAI_REALM_RL_FN where REALM_KEY in (select REALM_KEY from SAKAI_REALM where SAKAI_REALM.REALM_ID IN (:1,:2,:3,:4,:5)) and FUNCTION_KEY i n (select FUNCTION_KEY from SAKAI_REALM_FUNCTION where FUNCTION_NAME = :6) and ( ROLE_KEY in (select ROLE_KEY from SAKAI_REALM_RL_GR where ACTIVE = '1' and USER_ 142 142 55,652 0.00 1.9 5n3rvywwp9y7u select itemmetada0_.ITEMID as ITEMID1_, itemmetada0_.ITEMMETADATAID as ITEMMETA1 _1_, itemmetada0_.ITEMMETADATAID as ITEMMETA1_134_0_, itemmetada0_.ITEMID as ITE MID134_0_, itemmetada0_.LABEL as LABEL134_0_, itemmetada0_.ENTRY as ENTRY134_0_ from SAM_PUBLISHEDITEMMETADATA_T itemmetada0_ where itemmetada0_.ITEMID=:1

18. Samigo Load Test AWR CPU Elapsed CPU per % Total Time (s) Time (s) Executions Exec (s) DB Time SQL Id ---------- ---------- ------------ ----------- ------- ------------- 10,105 10,659 2,125,189 0.00 15.2 0byxm85k79suc select answerfeed0_.ANSWERID as ANSWERID1_, answerfeed0_.ANSWERFEEDBACKID as ANS WERFE1_1_, answerfeed0_.ANSWERFEEDBACKID as ANSWERFE1_137_0_, answerfeed0_.ANSWE RID as ANSWERID137_0_, answerfeed0_.TYPEID as TYPEID137_0_, answerfeed0_.TEXT as TEXT137_0_ from SAM_PUBLISHEDANSWERFEEDBACK_T answerfeed0_ where answerfeed0_.A 8,184 8,459 99,794 0.08 12.1 8wcadnu8rwd5b select count(assessment0_.ASSESSMENTGRADINGID) as col_0_0_ from SAM_ASSESSMENTGR ADING_T assessment0_ where assessment0_.FORGRADE=1 and assessment0_.AGENTID=:1 a nd assessment0_.PUBLISHEDASSESSMENTID=:2 7,290 8,974 665,235 0.01 12.8 3px7qs284b3xk select answerset0_.ITEMTEXTID as ITEMTEXTID7_, answerset0_.ANSWERID as ANSWERID7 _, answerset0_.ANSWERID as ANSWERID136_6_, answerset0_.ITEMTEXTID as ITEMTEXTID1 36_6_, answerset0_.ITEMID as ITEMID136_6_, answerset0_.TEXT as TEXT136_6_, answe rset0_.SEQUENCE as SEQUENCE136_6_, answerset0_.LABEL as LABEL136_6_, answerset0_ 5,979 6,189 77,580 0.08 8.8 3cswn2dndzcm0 select assessment0_.ASSESSMENTGRADINGID as ASSESSME1_140_, assessment0_.PUBLISHE DASSESSMENTID as PUBLISHE2_140_, assessment0_.AGENTID as AGENTID140_, assessment 0_.SUBMITTEDDATE as SUBMITTE4_140_, assessment0_.ISLATE as ISLATE140_, assessmen t0_.FORGRADE as FORGRADE140_, assessment0_.TOTALAUTOSCORE as TOTALAUT7_140_, ass 2,174 2,232 317,661 0.01 3.2 9xmw3p1px195k select count(1) from SAKAI_REALM_RL_FN where REALM_KEY in (select REALM_KEY from SAKAI_REALM where SAKAI_REALM.REALM_ID IN (:1,:2,:3,:4,:5)) and FUNCTION_KEY i n (select FUNCTION_KEY from SAKAI_REALM_FUNCTION where FUNCTION_NAME = :6) and ( ROLE_KEY in (select ROLE_KEY from SAKAI_REALM_RL_GR where ACTIVE = '1' and USER_ 1,950 2,026 22,183 0.09 2.9 8su2pcsy7fbq5 select assessment0_.ASSESSMENTGRADINGID as col_0_0_, publisheda1_.ID as col_1_0_, publisheda1_.TITLE as col_2_0_, assessment0_.AGENTID as col_3_0_, assessment0_.SUBMITTEDDATE as col_4_0_, assessment0_.ISLATE as col_5_0_, assessment0_.FORGRA DE as col_6_0_, assessment0_.TOTALAUTOSCORE as col_7_0_, assessment0_.TOTALOVERR

19. New Indexes for Samigo create index idx_ASSGRAD_AID_PUBASSEID on SAM_ASSESSMENTGRADING_T(AGENTID,PUBLISHED ASSESSMENTID) tablespace ctools_indexes; create index IDX_ITEMGRADING_A_GRADINGID on SAM_ITEMGRADING_T(ASSESSMENTGRADINGID) tablespace ctools_indexes; create index IDX_PUBANS_ITEMTEXTID on SAM_PUBLISHEDANSWER_T(ITEMTEXTID) tablespace ctools_indexes; create index IDX_PUBMETDATA_ASSESSMENTID on SAM_PUBLISHEDMETADATA_T(ASSESSMENTID) tablespace ctools_indexes;

20. Samigo Load Test AWR CPU Elapsed CPU per % Total Time (s) Time (s) Executions Exec (s) DB Time SQL Id ---------- ---------- ------------ ----------- ------- ------------- 4,840 5,046 1,007,039 0.00 21.0 0byxm85k79suc select answerfeed0_.ANSWERID as ANSWERID1_, answerfeed0_.ANSWERFEEDBACKID as ANS WERFE1_1_, answerfeed0_.ANSWERFEEDBACKID as ANSWERFE1_137_0_, answerfeed0_.ANSWE RID as ANSWERID137_0_, answerfeed0_.TYPEID as TYPEID137_0_, answerfeed0_.TEXT as TEXT137_0_ from SAM_PUBLISHEDANSWERFEEDBACK_T answerfeed0_ where answerfeed0_.A 3,489 4,245 315,353 0.01 17.7 3px7qs284b3xk select answerset0_.ITEMTEXTID as ITEMTEXTID7_, answerset0_.ANSWERID as ANSWERID7 _, answerset0_.ANSWERID as ANSWERID136_6_, answerset0_.ITEMTEXTID as ITEMTEXTID1 36_6_, answerset0_.ITEMID as ITEMID136_6_, answerset0_.TEXT as TEXT136_6_, answe rset0_.SEQUENCE as SEQUENCE136_6_, answerset0_.LABEL as LABEL136_6_, answerset0_ 1,072 1,096 152,652 0.01 4.6 9xmw3p1px195k select count(1) from SAKAI_REALM_RL_FN where REALM_KEY in (select REALM_KEY from SAKAI_REALM where SAKAI_REALM.REALM_ID IN (:1,:2,:3,:4,:5)) and FUNCTION_KEY i n (select FUNCTION_KEY from SAKAI_REALM_FUNCTION where FUNCTION_NAME = :6) and ( ROLE_KEY in (select ROLE_KEY from SAKAI_REALM_RL_GR where ACTIVE = '1' and USER_ 817 839 220,813 0.00 3.5 5n3rvywwp9y7u select itemmetada0_.ITEMID as ITEMID1_, itemmetada0_.ITEMMETADATAID as ITEMMETA1 _1_, itemmetada0_.ITEMMETADATAID as ITEMMETA1_134_0_, itemmetada0_.ITEMID as ITE MID134_0_, itemmetada0_.LABEL as LABEL134_0_, itemmetada0_.ENTRY as ENTRY134_0_ from SAM_PUBLISHEDITEMMETADATA_T itemmetada0_ where itemmetada0_.ITEMID=:1 623 643 15,350 0.04 2.7 c55kf9dj75jy4 select count(1) from SAKAI_REALM_RL_FN MAINTABLE LEFT JOIN SAKAI_REALM_RL _GR GRANTED_ROLES ON (MAINTABLE.REALM_KEY = GRANTED_ROLES.REALM_KEY AND MAINTABLE.ROLE_KEY = GRANTED_ROLES.ROLE_KEY), SAKAI_REALM REALMS, SAKAI_REAL M_ROLE ROLES, SAKAI_REALM_FUNCTION FUNCTIONS where ( ROLES.ROLE_NAME in(

21. Server Side Monitoring Ran Perl script to capture CPU utilization on one of eight app servers as well as the DB server The amount of free space in the JVM on one of the application servers was also captured Ran ‘tail –f *.log’ on the app servers looking for stack traces, 404 errors, etc. When necessary, send a ‘kill –QUIT ’ to get a thread dump of what’s going on inside

22. JAVA_OPTS JAVA_OPTS="-server -Xms1500m -Xmx1500m" JAVA_OPTS="$JAVA_OPTS -XX:NewSize=400m XX:MaxNewSize=400m" JAVA_OPTS="$JAVA_OPTS -XX:PermSize=192m - XX:MaxPermSize=192m" JAVA_OPTS="$JAVA_OPTS -verbose:gc -XX:+PrintGCTimeStamps - XX:+PrintGCDetails" JAVA_OPTS="$JAVA_OPTS -Dsakai.home=/usr/local/sakai/home - Dsakai.components.root=/usr/local/sakai/components" JAVA_OPTS="$JAVA_OPTS -Dsakai.security=/usr/local/sakai/home" JAVA_OPTS="$JAVA_OPTS -Dnetworkaddress.cache.ttl=0 - Dnetworkaddress.cache.negative.ttl=0" JAVA_OPTS="$JAVA_OPTS -Dsun.net.inetaddr.ttl=0 - Dsun.net.inetaddr.negative.ttl=0" JAVA_OPTS="$JAVA_OPTS -Djava.awt.headless=true"

23. CPU Utilization Both application server and database server utilization was spiky Application server was in the 20-60% range Database server was in the 60-100% range

24. CPU Utilization Graphs

25. App Server Heap Usage

26. Application Profiling If bottleneck was determined to be on application server side, we were prepared to use JProfiler to capture internal JVM statistics Due to its overhead, a profiler can generally only capture statistics under lighter loads

27. Purpose for the trip Identify bottlenecks under load. –no local performance test environment and expertise Establish a baseline for future regression testing. Gain confidence in our local deployment. –Samigo is an important tool for our users. Publish performance test report to the Sakai community.

28. Prepare for Performance Test What test case scenarios are we testing? –Focus on Samigo’s delivery feature. What is the expected usage pattern –Our user support team provided the expected # of concurrent users at Stanford –Emailed and collected other schools’ usage information Prepare for the test –Created sample assessments with question types. –Send Chris XMLs to be pre-loaded to the test environment. –Assisted Chris with load test scripts.

29. During the test Monitored Oracle DB Monitored Application Servers, Monitored LoadRunner’s test results/graphs. Iterative performance tuning based on test results.

30. What we learned Database: –Added indexes upon Drew’s recommendation. Application: –Optimized application code to reduce excessive DB calls (SAK-10153) No memory leaks found.

31. Things we didn’t test in delivery Focused on Samigo’s Delivery only –Did not test Audio and File Upload questions due to limited time –Audio/File upload questions are heavily used at Stanford. Please let us know if you have questions.

32. Summary Performance test helps to identify bottlenecks under load. Establishes a baseline for future testing. –Identify performance improvement/degradation in future releases. Load test should be an integral part of release. Performance test report can be found at http://bugs.sakaiproject.org/confluence/display/SAM/Home http://bugs.sakaiproject.org/confluence/display/SAM/Home