Presentation on theme: "Load Testing UNIX Systems Peter Harding"— Presentation transcript:
Load Testing UNIX Systems Peter Harding
System & Load testing Two main circumstances you might want to test: –Sizing new hardware (system acquisition) –Deploying new application (new or upgraded software, eg. Y2K)
Two Types of Tests In either situation there are two types of test we are interested in: –System testing: Backups End of Day/End of Month –Load testing: Capacity – throughput/response time Scalability
A Load Testing Methodology Typically want to run a system up with a client’s application and databases in a hardware configuration reflecting actual client production infrastructure. Want to find out such things as maximum throughput, expected response times under a designated load. May want to validate modelled predictions for performance (& capacity).
Environments ASCII (telnet, rlogin, ssh) Windows X Windows (X11, KDE, Gnome, …) Web –http –https (SSL) Other protocols –ftp, nntp, –Tuxedo –Corba –SQL*Net Java –See:
Commercial Tools Mercury Interactive (www.mercuryinteractive.com)www.mercuryinteractive.com –WinRunner –LoadRunner Rational (now owned by IBM) –Rational Robot Seague (www.seague.com)www.seague.com –SilkPerformer V CompuWare (www.compuware.com)www.compuware.com –QALoad Others –Lots – especially in the web space
Single Tier (Insitu) Benchmark Architectures System Under Test (SUT) + RTE System(s) (Client - Driver System)
Two Tier Benchmark Architectures RTE System(s) (Client - Driver System) System Under Test (SUT) RTE System(s) (Client - Driver System) TTY Lines Network Connection (X25 or Ethernet) System Under Test (SUT)
Three Tier Benchmark Architecture RTE System(s) (Client System) System Under Test (SUT) RTE System (Driver System)
The General Approach Planning Phase Development Phase Testing Phase Review Phase Reporting Phase
Planning Phase Conversation with client determine scope of exercise. Consultation with client resources are required to identify WHAT is to be tested – ie. The TASKS. Further conversation will hopefully yield indications of expected mix of activities (TASKS). This is often subject to wild speculation. This allows us to specify the load scenarios from which we build the workload files.
Development Phase Familiarize yourself with the application environment (probably using a development environment for SUT). Set up test harness and RTE/data server libraries on driver(s). Record basic navigation through the application. Record and test each identified TASK. Handling error conditions will require most effort.
Test Phase Setup of System Under Test (SUT) environment will be required about now. Setup data to be fed to robots during actual tests. Build workload files. Run tests. Do analysis. Publish report.
The Basic Components Recorder (record) RTE Library (librte.so) Test Harness (sim.c et al.) Controller (start /regulate/review) Data Server (dserver) Workload setup (wcalc) Post-processing scripts (postproc.pl)
The Test Harness The test harness consists of a directory containing the C source code for the test framework and each of the robots. It also contains: –a Makefile (which links in librte.so and libdcl.so) –Various text files used to configure each run (.regrc,.rterc, Parameters, SeqNo, RTE, Offset, Host, etc.). –A directory (or symbolic link to one) in which the tree of log files and other test output are stored. –Symbolic links into the log directoy of the current and last run. (These are set up by the start script)
The Controller The controller is a C executable, regulate, which takes a large number of arguments many of which are passed down into the individual robots. It is typically invoked using the start script which crafts these arguments automagically using various config files in working directory (SeqNo, Offset, RTE, Host, etc.).
Controller Operation On startup the controller (regulate) reads.regrc file which controls rate of start up. It establishes shared memory and semaphores used to provide communication between robots, regulate and review (the viewer). It enters a loop which fork/execs the child processes and overlays the executable specified in ‘RTE’ Once all child processes are initiated it falls through into a loop where it executes wait() until all children have terminated and been cleaned up. The controller can optionally restart children as they terminate if desired. This is specified in.regrc file.
Test Monitoring Tests are monitored by the review program which reads the global shared memory segment to access status information written by each running robot. Review uses a ps command in the background to establish the pid of the version of regulate it should monitor. If multiple tests are running the pid of the controlling regulate to be monitored may be specified on the command line.
review The command line interface within review mimics that used by vi. A text help file is provided which summarises commands available. review also provided access to log, stdout, stderr and trace files of running robots.
The Data Server The data server uses System V IPC (queues) to pass data from a single server to each of the robots (initiated by each robot). Data sources are CSV (text) files customized for the tests. Typically CSV files will be crafted from output of SQL on underlying database but may be hand crafted (say by client personnel using Excel)
Data Server Operation Starting it up: plh]$ dserver [dserver] Server working directory is /u/pha/plh/DATA Stale dserver pid file! [dserver] Server pid is 2227 Used 0 MByte of malloc space Shutting it down: plh]$ dserver -T [dserver] Terminating server with pid, 2227 plh]$ [shutdown] dserver shut down Wed Apr 16 15:28: On shutdown unused data in memory is flushed to disk. Makefile is used to reset state of data files from master copies matched to databases of system under test (SUT).
Data Server Configuration Data server files are contained in a directory which should be defined in the environment as $DSERVER_DIR. Configuration is contained in dserver.ini in that directory. Data is stored in.dat files, used data in.used files and the master copies in.master files.
dserver.ini Format of file is as follows: DATA]$ cat dserver.ini [Data] Description=Address,CSV #Description=Name,CSV #Description=Note,CSV #Description=UniqNo,Sequence #Description=WhsItem,Keyed #Description=Keyed,Keyed #Description=Indexed,Indexed Description=CreditorId,CSV Description=CredActivity,CSV Description=ItemId,CSV Description=ItemQryTimes,CSV Description=ProdCode,CSV … Description=FinCredATB_Out,CSV Description=FinGLAcctInq,CSV Description=FinGLAcctInq_Out,CSV Description=FinBankRecSR,CSV Description=FinBankRecSR_Out,CSV
Workload Generation The various load scenarios are converted into workload files which completely choreograph each run by the workload calculator (wcalc). This is a UNIX command line tool which takes a configuration file and produces a workload file. (eg. wcalc –c BuyerLookupPPCE) The workload file is read by each of the robots when they start to extract the tasks to be performed and their start-times.
Workload Definition File (eg Scen01_3600.wcd) # # An initial mean load workload definition # # Date: Tue Jun 4 12:41:32 EST 1996 [Config] NoUsers=40 Duration=3600 # seconds StaggerDelay=0 # seconds RestartTimer=FALSE [Threads] # Name,NoSessions,NoTx,GivenCycleTime Thread=BuyLookupPPCE,16,3200,0,120 Thread=BuyLookupPPCE_PF6,5,3000,0,120 Thread=BuyLookupPPCE_PF7,5,3000,0,120 Thread=BuyCyberSADD,5,1800,0,120 # # Very quick financial reports # Thread=FinAPCredEnq,1,600,0,120 Thread=FinChqListing,1,240,0,120 Thread=FinCredATB,1,600,0,120 Thread=FinGLAcctInq,1,120,0,120 Thread=FinGLTBF1,1,120,0,120 Thread=FinMiscCRDR,1,120,0,120 # # Long reports # Thread=FinCyberGLTR,1,10,0,120 Thread=BuyBuyerPOR,1,10,10,120 Thread=BuyCyberPOR,1,20,20,120
Outputs from Robots Each robot produces four output files: –Log fileNNNN.log –StdoutNNNN.stdout –StderrNNNN.stderr –Trace fileNNNN.trc
Log File Format The log file encapsulates: –Benchmark start and end times –Task elapsed times –Query response times
Log file sample ******** S Mon Feb 17 16:10: [ ] RTE version > I Identifier "99_40" > I SessionIdx 9 RangeOffset U Concurrency = U Display = U DelayMode = U UseCycleTimes = U UseStaggerDelay = U LoopMax = U MaxWait = U FixedSequence = U SendRate = U StaggerDelay = U TimeoutRestart = U TimerRestartInterval = U Tracing = U **** "170203" U **** Fixed sequence initialization (Seed = 9) U Adding "BuyLookupPPCE" (idx 12) start 32 sec U Adding "BuyLookupPPCE" (idx 12) start 90 sec
Log file sample (Cont) U Workload file "Scen04_3600.wld" contains 60 threads U Telnet to " " M S "t01" S "^J" M S "tst1ng" S "^J" M S "Ush^M" M S "^[[M" M S "FSMENU^M" M U %Benchmark start at 51.9 sec U Idle U [nextTask] currentThread 12 Task U Task 0 sleeping with delay U checkRunning() U CMD_RUN
Log file sample (Cont) U %Start BuyLookupPPCE at 32.0 sec S "^[[P" M S "^[[T" M S "^[[O" M U Looking up " "^M S " " S "^M" M M U %Query BuyLookupPPCE LookUp 1.4 sec U , S "^[[d" M S "^[[d" M S "^[[d" M S "^[[d" M U %Finish BuyLookupPPCE (OK) at 47.8 Proc 15.8 Elap 15.8 CSD 0.0 CPD U [nextTask] currentThread 12 Task U Task 1 sleeping with delay U checkRunning() U CMD_RUN U %Start BuyLookupPPCE at 90.8 sec
Starting a Run Use the script start.sh to initiate run. REJECTSHOP]$./start.sh 20 Log files will be written to directory results/ /104_20 Starting run 104 on hostname at rate 1 Duration of run will be 1800 seconds Taking 60 samples at intervals of 30 seconds Running "regulate -h e sim -r 1 -W TstThread0.wld -w results/ /104_20/logs -i 104_20 -D o 0 20" Regulate Process PID 2305 Logfile Generation... Run the review executable (in working directory). Wait until all robots have paused (P). Type ‘:S’ (Start) to initiate test. Terminate by typing ‘:E’ (End) – robots will complete current task and then exit gracefully. Abort by typing ‘:X’ (eXterminate) – robots all terminate on signal.
Review Screen Review is a curses application whose screen looks like this: [V3.2.12] Controlling PID: 2535 Run Duration: 1800 User: plh Started ramp up at: Wed Apr 16 16:04: Elapsed: 00:00:25 -- Sessions: started 20 finished 0 failed 0 concurrent 0 of 20 Ident: 105_20 Sess: P 5P 5P 5P 5P 5P 5P 5P 5P 5P 10 5P 5P 5P 5P 5P 5P 5P 5P 5P 5P For each session (robot) the number (5) is the send/match count and the letter (P) is the status code. This is set by rteSleep() call in robot [e.g. rteSleep( 15, ‘C’);].
review (once started) Once running review will display progress: [V3.2.12] Controlling PID: 2535 Run Duration: 1800 User: plh Started ramp up at: Wed Apr 16 16:04: Elapsed: 00:04:27 S- Sessions: started 20 finished 0 failed 0 concurrent 0 of 20 Ident: 105_20 Sess: A 13A 13A 5i 13A 5i 13A 5i 5i 5i 10 5i 5i 5i 5i 5i
Post-Processing The running robots each write elapsed and query times into a log file. These files are post processed to produce analysis for reports. The post-processing phase produces: Histograms of query response time and task elapsed time. CSV versions of the above data which is folded into an Excel spreadsheet using a macro. Various graphs (in PostScript using GnuPlot which are converted into PDF files for distribution).
Post-processing Scripts Historically a combination of shell and Perl scripts have been utilized for rehashing the log files to produce. Most recently Perl has been used in conjunction with GnuPlot and Acrobat.
Response Time Histogram ***** BuyLookupPPCE_PF6-PF6 ************************************** No of queries: 580 Mean: 0.23 sec Standard Deviation: 0.81 Variance: 0.65 Median: 0.0 sec (no = 280) Mode: 0.0 sec (no = 280) Fraction at mode: (Mode/Total) Percentile Distribution: 50% 60% 70% 75% 80% 85% 90% 95% 99% Time: Query Response Time Distribution: Time: Freq: Time: Freq: Time: Freq:
Analysis Typically the analysis is performed in conjunction with tools that report on system performance during the test runs: –sar, iostat, netstat, etc. –Various shell and Perl scripts wrapped around sar –TeamQuest
Reporting Reports typically consists of two parts –Overall system performance & capability at various workload levels (scenarios). –Throughput and response time at various load levels (scenarios).
Conclusions Labour intensive. Complex. Fun (if you like breaking systems and software). Useful for leveling the playing field in system acquisitions. Useful for determining load limit and throughput thresholds and bounds.