Presentation on theme: "Profiling Suspicious Code"— Presentation transcript:
1Profiling Suspicious Code Tom BascomWhite Star Software
2A Few Words about the Speaker Tom Bascom; Progress 4gl coder & roaming DBA since 1987President, DBAppraise, LLCRemote database management service for OpenEdge.Simplifying the job of managing and monitoring the world’s best business applications.VP, White Star Software, LLCExpert consulting services related to all aspects of Progress and OpenEdge.I have been working with Progress since 1987… and today I am both President of DBAppraise;The remote database management service…where we simplify the job of managing and monitoring the worlds best business applications;and Vice President of White Star Software;where we offer expert consulting services covering all aspects of Progress and OpenEdge.
3Users want the right answer, with the best response time at the lowest cost.
4The performance enhancement possible with a given improvement is limited by the fraction of the execution time that the improved feature is used.-- Amdahl’s Law
5Performance is not just about the database. The most bang for the performance tuning buck is often in the application code.But figuring out where to look is often hard.It’s the code that drives the database…But figuring out where to look is hard
6Target the largest response time component of the most important Business process first.
8Finding “Likely Suspects” User ComplaintsCompile with XREFcompile “program.p” xref “tmp/program.xrf” debug-list “tmp/program.dbg”.“I/O By User” DataCRUD DataTesting – Add Performance Criteria to Test Plans for New Releases
10PROMON – IO By Process 04/08/04 I/O Operations by Process 11:00:00 Database BI AI -----Usr Name Access Read Write Read Write Read Write0 lakewood10 lakewood11 lakewood12 lakewood13 smcnulty14 eratclif16 sstout17 lakewood18 lakewood19 aracey20 lakewood
11ProTop – IO By User09:37: ProTop -- Progress Database Monitor (release xv) /19/04Sample sports2000 [/data/s2k/sports2000] RateHit Ratio: : : Commits: Local:Miss% : % % Latch Waits: Remote:Hit% : % % Tot/Mod Bufs: Batch:Log Reads: Evict Bufs: Server:OS Reads: Lock Table: Other:Chkpts: Lock Tbl HWM: TRX:Flushed: Old/Curr BI: Blocked:Area Full: % After Image: DISABLED Total:UIOUsr Name Flags PID DB Access OS Reads OS Writes Hit%31 julia SB %30 jami SB %34 tucker SB %9 tucker SB* %6 julia SB %32 peter SB %16 julia SB %
13Why “Logical I/O” ??? Consistent and Repeatable Measurement The same query against any given dataset will always return the same result.Not subject to external factors such as CPU speed, disk throughput, user activity or the buffer cache hit ratio.Shows Hidden Problems even with small datasets.Shows Impact on Other Users.“Chokepoint” on rate of Logical IO ops.
14Why NOT etime() ??? Non-Repeatable Subject to a host of external factorsCPU speed, disk throughput, other user activity, buffer cache efficiency, phase of the moonGranularity is too gross (millisecond)Does measure non-db activity…
15LRTEST.p define variable i as integer no-undo. define variable lr as integer no-undo.find _myconnection no-lock.find _userio no-lock where_userio-usr = _myconn-userid no-error.lr = _userio._userio-dbaccess.etime( yes ).find <table> no-lock where <whatever> no-error.display i ( _userio-dbaccess - lr ) etime().
16Example Production Development QA for each loanfile no-lock where loan-amount > :RecordsLog I/ORatioetime()Production383335,597876:16,685ms 2,788msDevelopment4128,517695:1647ms 394msQA365327,284896:14,880ms 2,965msfor each loanfile no-lock where price-locked >= 9/01/2011 and price-locked <= 9/30/2011:1,4983,0012:164ms 52ms3N/A19ms 0ms1,5013,010
17The performance enhancement possible with a given improvement is limited by the fraction of the execution time that the improved feature is used.-- Amdahl’s Law
18Target the largest response time component of the most important Business process first.
19Wouldn’t it be nice if… Description: XYZZY Date: 10/04/11 Top Total Time LinesProgram Line Avg Time Time Callsxtabsms2.pgetdocprep2.pproc_create_sitm xtabsms2.pxtcountry_x2_x3.p ,967proc_read-database sysval.p ,879xtcountry_x2_x3.p ,894getdocprep2.pproc_upd_nref xtmfintb2.pproc_upd_nref xtmfintb2.pproc_process_tasks xttskscn.rfindClient sysval.p ,763
21Profiler First introduced with version 8.2 (-zprofile) “Unsupported” (meaning the analysis tool)Improved with version 9.0 (session:profiler handle, no more -zecret)Microsecond timingsDoes not include “think time”
22Using the Profiler -profile profiler: handle Analysis tools Non-intrusiveNon-selectiveprofiler: handleSelectiveBut requires code insertion or “wrappering”Analysis tools$DLC/src/samples/profiler
23PROFILER AttributesDESCRIPTION – optional text describing this sessionLISTINGS – whether or not to create debug listingsDIRECTORY – where to create debug listings (default to –T)FILE-NAME – name of output file (default profile.out)ENABLED – yes/no; initializes listings and so forthPROFILING – turn profiling on or off
24Other PROFILER Attributes TRACE-FILTER – CSV list of “matches” criteria for procedure tracingTRACING – line level tracingCOVERAGE - % coverage support
25PROFILER Methods Write-Data() – flush accumulated data to output file. User-Data(char) – write user defined data, such as VST statistics, to the output file.
26Minimal Embedded Usage assign profiler:enabled = yes profiler:profiling = yes . do i = 1 to : /* do something */ end. assign profiler:enabled = no profiler:profiling = no . profiler:write-data().
27Targeted Profiling Embedded in Code Being Investigated define variable i as integer no-undo.run profiler/on.p ( “batch001” ).do i = 1 to 10:find customer no-lock where customer.cust-num = 1 no-error.end.find customer no-lock where customer.phone = "(702) " no-error.run profiler/off.p ( “batch001” ).
29Profiling a SessionCreate file called profiler.cfg with 3 lines: -OUTFILE /tmp/profiler.out -LISTINGS /tmp -DESCRIBE someDescriptionAdd –profile to session startup: mpro dbName –p start.p –profile profiler.cfgRun normally.Terminate cleanly & analyze the output.
30Sample Profiling Output Description: XYZZY Date: 10/04/11Top Total Time LinesProgram Line Avg Time Time Callsxtabsms2.pgetdocprep2.pproc_create_sitm xtabsms2.pxtcountry_x2_x3.p ,967proc_read-database sysval.p ,879xtcountry_x2_x3.p ,894getdocprep2.pproc_upd_nref xtmfintb2.pproc_upd_nref xtmfintb2.pproc_process_tasks xttskscn.rfindClient sysval.p ,763
31Profiler Example A Calculation Bottleneck? p = 4 * ( )
32Profiler Example A Calculation Bottleneck? 0009 function piterm returns decimal ( input n as integer ).return ( 1.0 / (( n * 2 ) + 1 )).0011 end.00120013 do while abs( newpi - oldpi ) > precision:oldpi = newpi.if i modulo 2 = 0 thenpi = pi + piterm( i ).elsepi = pi - piterm( i ).newpi = ( 4.0 * pi ).display i newpi oldpi.i = i + 1.0022 end.
33Sample Profiling Output Description: pi Date: 10/07/11Top Total Time LinesProgram Line Avg Time Time Calls./pi.p ,001piterm ./pi.p ,001./pi.p ,002./pi.p ,000./pi.p ,001./pi.p ,001./pi.p ,001./pi.p ,001./pi.p ,001/home/tom/p26226_Untitled1.pedpiterm ./pi.p ,001./pi.p ,001
34Profiler Example A Calculation Bottleneck? 0009 function piterm returns decimal ( input n as integer ).return ( 1.0 / (( n * 2 ) + 1 )).0011 end.00120013 do while abs( newpi - oldpi ) > precision:oldpi = newpi.if i modulo 2 = 0 thenpi = pi + piterm( i ).elsepi = pi - piterm( i ).newpi = ( 4.0 * pi ).if i modulo 100 = 0 then display i newpi oldpi.i = i + 1.0022 end.
35Sample Profiling Output Description: pi Date: 10/07/11Top Total Time LinesProgram Line Avg Time Time Callspiterm ./pi.p ,001./pi.p ,002./pi.p ,000./pi.p ,001./pi.p ,001./pi.p ,001./pi.p ,001./pi.p ,001./pi.p ,001piterm ./pi.p ,001./pi.p ,001
36Caveat!define variable i as integer no-undo. assign profiler:enabled = yes profiler:profiling = yes . do i = 1 to : end. i = 0. do while i < : i = i + 1. end. i = 0. do while i < : i = i + 1. end. assign profiler:enabled = no profiler:profiling = no. profiler:write-data().
37Caveat!1 02/28/2007 "Generic" 07:55:03 "tom" . 2 "profile.p" ""Note the result for line 16 (the last line shown in prof.out). (The module 0, line 0 line is also curious but it isn't my focus right now.) I can assure you that it took much longer the seconds for that code to execute It would seem that source lines with multiple statements on them are not correctly accounted for.
38Reminders & Hints Line numbers are DEBUG LISTING line numbers You need to have .DBG filesThey need to be created with the same source and PROPATH as your .r filesSession profiling must terminate cleanly – no “kill” & no crash.Temp files can become very, very large.
39Summary Things to be suspicious of. Tools to narrow your search. A better way to gauge query effectiveness.An introduction to Profiling.Strategies for attacking code performance problems.The Stuff That We’ve Learned Today!
40Target the largest response time component of the most important Business process first.