1. How to Design and Implement Load Testing for Sakai The Michigan Experience

2. 8 Dec 2006lmp@umich.edu 2 Why Load Test? –Measure end-user response time How long does a specified operation take? Currently a minor part of our testing –Define optimal hardware configuration Enables pre-tuning of Oracle Testing various JVM settings –Check reliability How long or hard can system work without producing errors (failures)?

3. 8 Dec 2006lmp@umich.edu 3 Why Load Test.... –Check hardware & software upgrades Does new host or software perform as well as previously tested versions? –Evaluate new products Used to evaluate hardware for database replacement –Measure system capacity How much can the system handle before performance degradation occurs? –Identify bottlenecks What part of the environment causes performance degradation?

4. 8 Dec 2006lmp@umich.edu 4 Test Environment Performance Testing Tool Infrastructure Courses & Users Data

5. 8 Dec 2006lmp@umich.edu 5 Performance Testing Tool Mercury Interactive LoadRunner –VuGen (virtual user scripting) –Controller (scenario and load generation) –Analyzer (analysis and reports) Approach appropriate to any tool Presentation will not focus on tool

6. 8 Dec 2006lmp@umich.edu 6 Infrastructure As close to Production as possible –Hardware –Software Same configuration unless testing something specific –Authentication Use application rather than CoSign ›UM-developed, open-source single web login service –UMIAC Warehouse for needed administrative data ›UM-developed, PERL scripts accessing Peoplesoft system –Database servers not always identical Where less “powerful,” larger host will likely perform better than observed in tests –AFS resource storage

7. 8 Dec 2006lmp@umich.edu 7 Courses & Users UMIACload provides course names, instructor owners, and rosters of students –11,000 courses –11,000 “instructors” –43,000 “students” Why so many? –To permit high level of activity without excessive concurrency

8. 8 Dec 2006lmp@umich.edu 8 Data Testing requires that all courses be pre- populated with materials to be used by students Both AFS and Oracle data used Saved versions of “base” data maintained to enable rollback to “clean” data Always start with data refresh when adding new tools to test suite

9. 8 Dec 2006lmp@umich.edu 9 Testing Process Scripts Scenarios Analysis Results

10. 8 Dec 2006lmp@umich.edu 10 Script Setup Analyze tool to be scripted Understand tool usage –Value of user profiling Design test plan –Goals –Single or multiple scripts? –Student or Instructor activity? –Are course materials required? Design data files –Used by Virtual Users (VU) to change input parameters

11. 8 Dec 2006lmp@umich.edu 11 Script Setup... Create course materials Populate courses with materials –Must be done sequentially to ensure all courses get materials –Multiple resources can be added to same course with same script –Need more efficient method Record activity to be tested

12. 8 Dec 2006lmp@umich.edu 12 Script Parameterization Scripts recorded as single user Load iterations need to change users each iteration –Data files used to manage user & input changes All object-oriented elements in Sakai receive associated ID numbers that must change whenever a course or user changes –Recording rules used to manage ID number changes –Generic parameter identifiers changed to make analysis easier

13. 8 Dec 2006lmp@umich.edu 13 Script Pseudocode - Chat Login as “u10001” –Moves to “My Workspace” Select course tab –Use “LOADTEST 102 001” Select “Chat” tool from left-hand navigation column –Moves to Chat tool Enter message into Chat text entry box Return to “My Workspace” Logout

14. Chat() { web_set_max_html_param_len("1024"); web_url("xlogin", "URL=https://ctoolsload.ds.itd.umich.edu/portal/xlogin", "Resource=0", "RecContentType=text/html", "Referer=", "Snapshot=t1.inf", "Mode=HTML", EXTRARES, "Url=../ctlib/skin/ctools/images/logo_inst.gif", "Referer=https://ctoolsload.ds.itd.umich.edu/ctlib/skin/tool_base.css", ENDITEM, "Url=../ctlib/skin/ctools/images/topleft-p-title.gif", "Referer=https://ctoolsload.ds.itd.umich.edu/ctlib/skin/ctools/tool.css", ENDITEM, "Url=../ctlib/skin/ctools/images/help.gif", "Referer=https://ctoolsload.ds.itd.umich.edu/ctlib/skin/ctools/tool.css", ENDITEM, "Url=../ctlib/skin/ctools/images/topright-p-title.gif", "Referer=https://ctoolsload.ds.itd.umich.edu/ctlib/skin/ctools/tool.css", ENDITEM, "Url=../ctlib/skin/ctools/images/reload_h.gif", "Referer=https://ctoolsload.ds.itd.umich.edu/ctlib/skin/ctools/tool.css", ENDITEM, "Url=../ctlib/skin/ctools/images/chatpresicon.gif", "Referer=https://ctoolsload.ds.itd.umich.edu/ctlib/skin/ctools/tool.css", ENDITEM, LAST); Script Walkthrough - Chat

15. // [WCSPARAM WCSParam_UserID 36 b2de7f47-7c74-40d5-0024-98d191450a5d] Parameter {WCSParam_UserID} created by Correlation Studio web_reg_save_param( "WCSParam_UserID", "LB=site/", "RB=\"", "Ord=2", "Search=Body", "RelFrameId=1.1", LAST ); web_submit_form("xlogin_2", "Snapshot=t2.inf", ITEMDATA, "Name=eid", "Value={CTlogin}", ENDITEM, "Name=pw", "Value={CTpwd}", ENDITEM, "Name=submit", "Value=Login", ENDITEM, EXTRARES, "Url=../ctlib/skin/ctools/images/topright-tab.gif", "Referer=https://ctoolsload.ds.itd.umich.edu/ctlib/skin/ctools/portal.css", ENDITEM, "Url=../ctlib/skin/ctools/images/topright-curtab.gif", "Referer=https://ctoolsload.ds.itd.umich.edu/ctlib/skin/ctools/portal.css", ENDITEM, "Url=../ctlib/skin/ctools/images/topright-tab_hov.gif", "Referer=https://ctoolsload.ds.itd.umich.edu/ctlib/skin/ctools/portal.css", ENDITEM, LAST);

16. web_link("LOADTEST {Course2} 001 W06", "Text=LOADTEST {Course2} 001 W06", "Snapshot=t3.inf", LAST); lr_think_time(5); // [WCSPARAM WCSParam_ChatID 36 194d7683-e9fc-41d3-0012-f0ac6fea0d60] Parameter {WCSParam_ChatID} created by Correlation Studio web_reg_save_param( "WCSParam_ChatID", "LB=title/", "RB=?", "Ord=1", "Search=Body", "RelFrameId=1.3", LAST ); web_link("Chat Room", "Text=Chat Room", "Snapshot=t4.inf", EXTRARES, "Url=/courier/194d7683-e9fc-41d3-0012-f0ac6fea0d60/1145294730027-0.2702248719455739?auto=true", "Referer=", ENDITEM, "Url=/courier/194d7683-e9fc-41d3-0012-f0ac6fea0d60/1145294740215-0.4220853133253527?auto=true", "Referer=", ENDITEM, "Url=/courier/194d7683-e9fc-41d3-0012-f0ac6fea0d60/1145294750293-0.2283065980084248?auto=true", "Referer=", ENDITEM, "Url=/courier/{WCSParam_UserID}-presence/1145294759434-0.23544053982281854?auto=true", "Referer=", ENDITEM, LAST);

17. web_url("{WCSParam_UserID}", "URL=https://ctoolsload.ds.itd.umich.edu/portal/presence/{WCSParam_UserID}", "Resource=0", "RecContentType=text/html", "Referer=https://ctoolsload.ds.itd.umich.edu/portal/presence/{WCSParam_UserID}", "Snapshot=t5.inf", "Mode=HTML", EXTRARES, "Url=/courier/{WCSParam_ChatID}/1145294760356-0.31310668668084407?auto=true", "Referer=", ENDITEM, LAST);

18. web_submit_data("{WCSParam_ChatID}", "Action=https://ctoolsload.ds.itd.umich.edu/portal/tool/{WCSParam_ChatID}?panel=Control", "Method=POST", "RecContentType=text/html", "Referer=https://ctoolsload.ds.itd.umich.edu/portal/tool/{WCSParam_ChatID}?panel=Control", "Snapshot=t6.inf", "Mode=HTML", ITEMDATA, "Name=message", "Value=Who me? Wadya want, meathead? sheesh.....", ENDITEM, "Name=eventSubmit_doSend", "Value=x", ENDITEM, "Name=send", "Value=Add message", ENDITEM, EXTRARES, "Url=/courier/{WCSParam_ChatID}/1145294761621-0.647618403375133?auto=true", "Referer=", ENDITEM, LAST); web_url("{WCSParam_ChatID}_2", "URL=https://ctoolsload.ds.itd.umich.edu/portal/tool/{WCSParam_ChatID}?panel=List", "Resource=0", "RecContentType=text/html", "Referer=https://ctoolsload.ds.itd.umich.edu/portal/tool/{WCSParam_ChatID}?panel=Main", "Snapshot=t7.inf", "Mode=HTML", LAST);

19. lr_think_time(6); web_url("%7E{CTlogin}", "URL=https://ctoolsload.ds.itd.umich.edu/portal/site/%7E{CTlogin}", "Resource=0", "RecContentType=text/html", "Referer=https://ctoolsload.ds.itd.umich.edu/portal/site_tabs/{WCSParam_UserID}", "Snapshot=t8.inf", "Mode=HTML", LAST); return 0; }

20. 8 Dec 2006lmp@umich.edu 20 Scenarios One or more scripts to be run in load test –Number of virtual users running script –Number of iterations per virtual user –Pacing (how much delay between iterations) –Think time (emulates user variability) –Browser emulation –Network bandwidth emulation –Results detail (step or full transaction level)

21. 8 Dec 2006lmp@umich.edu 21 13 “Core” Tests Instructor –CreateAnnounce –CreateAssign –ReviseResource –ScheduleEvent –UploadLgFile –UploadSmFile Student –Chat –CreateProject –DownloadSmFile –DownloadLgFile –NewDiscussion –ReadBrowse –SubmitAssign

22. 8 Dec 2006lmp@umich.edu 22 “New” Tool Tests Instructor –AddSoMod –AddVzMod –EditSyllabus –EditWiki3L –SubmitGrades Student –ReadTxtMod –ReadSoMod –ReadVzMod –ReadSyllabus –ReadWiki3L –ViewGrade

23. 8 Dec 2006lmp@umich.edu 23 Scenario Preparation Single script runs –1 VU –100 iterations –No wait time between iterations –No think time –Showing response times for each step of transaction Identify serious script problems Take average response time for scenario

24. 8 Dec 2006lmp@umich.edu 24 Scenario Iteration Timing (125%) Script name(VU, Wait) CreateSite/Project3VU, 0sec UploadSmFile (85%)4VU, 0sec UploadLgFile (15%)2VU, 7sec DownloadSmFile (85%)78VU, 0sec DownloadLgFile (15%)10VU, 0sec CreateAnnounce2VU, 0sec ScheduleEvent6VU, 0sec NewDiscussion1VU, 1sec CreateAssign1VU, 57sec SubmitAssign8VU, 0sec ReviseResource2VU, 4sec Chat10VU, 0sec LoginLogout & ReadBrowse76VU, 0sec Script name(VU, Wait) EditWiki3L1VU, 136sec ReadWiki3L1VU, 1sec EditSyllabus1VU, 157sec ReadSyllabus1VU, 29sec SubmitGrades2VU, 67sec ViewGrades1VU, 12sec AddVzMod1VU, 140sec AddSoMod1VU, 141sec ReadTxtMod1VU, 36sec ReadVzMod1VU, 23sec ReadSoMod1VU, 44sec

25. 8 Dec 2006lmp@umich.edu 25 Analysis & Results Scenario runs for 1 hour plus ramp up and down time Upon completion of run, use LoadRunner Analyzer to crunch data Compare resulting graphs with previous runs Developers, DBA, SysAdmins look for performance bottlenecks Fixes proposed and test rerun

26. 8 Dec 2006lmp@umich.edu 26 CTload Scenario Results: 2.0_r3636w8app (100%)

27. 8 Dec 2006lmp@umich.edu 27 CTload Scenario Results: 2.2.1_r14159w6app (125%)

28. 8 Dec 2006lmp@umich.edu 28 Sharing the Burden? Multiple campuses participate –Ideally a few dedicated partners Localized hardware Localized test tool choice Centralized resources –Test design –Pseudocode for scripts –Course data files –Results –Support and discussion

29. 8 Dec 2006lmp@umich.edu 29 Community Testing? Share common core tests –Baseline for comparison Focus on new tools of interest to campus –Contribute resources back to central repository –Contribute results to community Greater stress on performance in development