Presentation on theme: "Performance Testing and OBIEE"— Presentation transcript:
1 Performance Testing and OBIEE WatchWaterRobin Moffatt, WM Morrisons plc
2 Introduction Oracle BI specialist at Morrisons plc Big IT development programme at its early stages implementing OBIEE, OBIA, ORDM, all on Oracle 11g & HP-UXAt Morrisons we’re on the latest (current) version of OBIEE 10g, and use Oracle 11g on HP.We’re using OBIA, ORDM, and we’ve built our own ODS using Oracle Data IntegratorThe performance work that I’ve done so far has been with OBIA, but what I’ll be talking about today should be applicable to any OBIEE installation.I’m interested to hear other people’s experience with OBIA and performance. Come and speak to me afterwards!We did Performance work forExisting prob with OBIAMethodology – future projects
3 The aim of this presentation A Performance Tuning MethodologyOBIEE techie stuffLearn from my mistakes!Three things to take awayQuestions – quick as we go / Q&A discussion at endNB – testing, not //tuning//
4 What is performance testing all about? Response timesReportETL batchOLTP transactionSystem impactResource usageScalabilityQuantified & EmpiricalSo– what is performance testing?I want to go through this pretty briefly, as it’s a huge area of theory that I can’t do proper justice to.I’ll try and cover off what I see as some of the basicsIt’s important to really understand this when you’re doing it, otherwise you’re not going to get valid test results and you’ll probably end up wasting a lot of time.For a proper understanding of it I’d really recommend reading papers written by Cary Millsap.Performance testing is a term used to cover quite a few different things.The way I define it – and this may or may not be industry standard, so apologies – is this:Performance testing itself is this [click] –Does it go fast enough?Does my report run in a time that meets user specification, or expectation?(This is within the context of OBIEE – for ETL we’d be talking about job runtimes and batch windows)The next stage after performance testing is generally load testing [click]This is to answer the questionWill it still go fast enough, once everyone’s using it?The other big consideration with load testing is -- Will it break my server? Or to be a bit more precise, what kind of impact is a new system going to have on an existing one? When you put your new reporting system live what will happen to the existing one that’s been running happily for a year?A logical extension of load testing is stress testing - how far can it scale? This applies to both the reporting system you’re putting in, and the servers themselves (which therefore feeds into capacity planning).I’ve heard these terms used interchangeably, and there’s quite a difference between them in my mind.Why does it matter? Well it dictates how you design and execute your testing.When you move from performance testing to load testing you generally take a step back in terms of level of detail.My definition: Load Testing is not Performance Testing. It may be an area within it, but it is most definitely not one and the same thing. This presentation is not about performance TUNING Of course, testing feeds into tuning which in turn feeds into testing, so the two are inextricably linked. But to try and keep things focussed - I will try to avoid discussing tuning specifics otherwise we’ll be here all day….Performance testing is the repeatable running of a request to obtain metrics (primarily response time), to determine whether its performance is acceptable or notAcceptable is a subjective term, but would typically be driven by user requirementsIn the context of OBIEE then we’re basically talking about does an Answers report or dashboard run fast enough to keep the users happy?Load Testing is aboutquantifying the effect an application is going to have on a systemwhether the application is going to perform acceptably on the system when its under loadOne report run on its own might be fine, but what happens on a Monday morning when a thousand users all log on at the same time and all run the report?
5 Why performance test? (Isn’t testing just for wimps?) Check that your system performsAre the users going to be happy?BaselineHow fast is fast?How slow is slow?Validate system designDo it right, first timeCapacity planningThis may be stating the obviously, but there are some pretty hefty reasons why you should incorporate multiple iterations of performance testing in any new development.- To test if the new system performs wellIs it going to return the data to the users in the time that they’re expecting?If you don’t test for this reason alone then you’re either brave, or foolhardy!What are BI systems about, if not providing the best experience to the end user- To provide baselinesHow do you know if a system is performing worse if you don't know how it performed beforeHow often have you had the problem reported to you “my report’s running slow”?If it took 10 seconds to run when you put the system live, then you’ve got a problem.Well what’s slow? 9 minutes?If it took 9 minutes to run when you put the system live then you’ve possibly just got an impatient user with unrealistic expectations.When you do get a performance problem, assuming you did your performance test packs beforehand you’ll be all set to diagnose where the problem liesBy their definition, performance tests generate a lot of lovely metricsOnce you think you’ve fixed the performance problem, you need to validate two thingsHave you fixed it?Have you broken anything else?Your performance test packs will give you the basis on which to prove it- To validate the way you are building the systemFor example, partitioning or indexing methodsHow often have you heard “It Depends” from a DBA?Optimal Parallelism setting or Partitioning strategy this time round may be different from what was optimal on the last project you didAn index may help one report, how do you know it doesn’t hinder another? Unless you have a pack of repeatable tests with timings then you can’t quickly tell what the impact isIn effect, you do your performance tuning up-front, as part of the build, rather than with a thousand angry users furious that their reports have stopped working
6 Why performance test? It’s never too late “You’ll never catch all your problems in pre-production testing. That’s why you need a reliable and efficient method for solving the problems that leak through your pre-production testing processes.”— Cary Millsap - Thinking Clearly About PerformanceIt’s not too late to put in place a solid performance test methodology around an existing system.If you set up your performance tests now you will have a set of baselines and a full picture of how your system behaves normallyThen when it breaks or someone complains you’re already set to deal with itIf you don’t, then when you do have problems you have to start from scratch, finding your way through the process.Which is better – at your leisure, or with the proverbial gun of unhappy users held to your head?Performance testing isn’t optional. It’s mandatory. It’s just up to you when you do it.Even if you’re running Exadata or similar and a so confident that you don’t have performance problems and never will – how are you going to capacity plan? Do you know how your system currently behaves in terms of CPU , IO, etc? How many more users can you run?Which is easier, simulate and calculate up front, or wait until it starts creaking?
7 Why performance test? Because it makes you better at your job “At the very least, your performance test plan will make you a more competent diagnostician (and clearer thinker) when it comes time to fix the performance problems that will inevitably occur during production operation.”— Cary Millsap - Thinking Clearly About PerformancePerformance Testing requires an extremely thorough understanding of the system.If you do it properly there is no doubt you will come out of it better equipped to support and develop the system further
8 Performance Testing – What & Why Quantifying response timesSystem impactUser expectationsProblem diagnosisDesign validationAny questions so far?
9 Performance Testing - How? Iterative approachDefineMeasureAnalyseReviewImplementTimebox!Do it rightDon’t “fudge it”Evaluate design / config optionsDo more testingThis is the methodology we’ve developedIt’s a high level view – subsequent slides will give detailRedefine testDo more testingBe Methodical
10 Define & build your test MeasureAnalyseReviewImplementDefine & build your testDefine – what are you going to testAim of the testScopeAssumptionsSpecificsData, environment, etcBuild – how are you going to test itOBIEE specificE.g. :Check that the system performsBaseline performanceProve system capacityValidate system designMake sure you have a specific aimEasier to have two clear tests than try to cover everything in oneLoose analogy - 10,000 ft view vs low-level flybyDon’t forget predicates – One report may have many filters and behave a lot differently depending on how they’re setWRITE IT ALL DOWNThink of breadth of test (# of reports) vs depth (# of metrics)If your test aim doesn’t define clearly enough how you’re going to run it, consider these two options:Option 1: Top down / Start big, see what breaks, follow standard troubleshooting of trying to isolate the problemShortest time to get initial resultsDifficult to isolate any problems thoughBetter for firefighting an existing problem where time is limited or there are obvious quick-winsimpreciseFor example – look in usage tracking for all reports that run > 5 minutes. Test only those reports.Option 2: Bottom up / Start small - define each test component, record behaviour for each test run, combine test components into bigger test runs, scale up to load testingUltimately more preciseMore metrics = more precision = quicker to identify issues and resolutionsBut it’s boring !Where’s my gigs of throughput bragging rights, or smoking server groaning under the load?What’s the point running a ten thousand users through your system just to prove it goes bang (or doesn’t)?Longer process, needs more accuracyFor example, report response time requirements from users, representative workloadsyour testing may give rise to some tuning, your tests must be isolatable and repeatable, otherwise how do you validate that what you’ve changed has fixed the problem and not made it worse?Repeatable –what’s the complete set of things you’d need to run the same test somewhere else? Eg: Schema definition, DB config parameters, OBIEE NQSConfig, OBIEE RPD, OBIEE reports , [plus presentation services config & web cat]Isolatable – Do you have a dedicated performance environment?, What else is running at the same time?
11 Consider your test scope DefineMeasureAnalyseReviewImplementConsider your test scopeFewer components = easier to manage = more precise = more efficientMore components = more complex = more variables = larger margin of errorReduce complexity intelligentlyDon’t cut corners, but distil the problem to its essencePick the closest point up stream from the bottleneck.More components = greater margin of error!So you’ve seen the different places in which you can test OBIEE. But how do you choose the one most applicable to the testing you’re doing? Should you just run everything against the database directly?Basically, it’s about keeping it simple, and avoiding unnecessary complexity.Say someone gives you a Ducati engine to fix. [click]You’ve already identified the area of the problem.Would you still spend all your time looking at the whole engine, or isolate where you know the problem lies and work on it from there? [click]Now clearly this isn’t an absolute, because there could be more than one problem with the engine, and so on – but the principle is sound. Reduce the complexity, intelligently.The same principle applies to testing OBIEE.Depending on the kind of reports, your RPD, and your data model and database, your performance bottlenecks will be somewhere across all of the stack.You shouldn’t be looking to cut corners, but look at it as distilling down what you’re testing to its essence and nothing more.Hopefully this is pretty obvious, as it’s going to be a more efficient use of your time, but here’s the two reasons why:1) Your tests show a slow response time and you need to track down and diagnose this. Would you rather be considering three elements or three hundred?When I was working on some performance testing, it was clear that very little time was spent after the BI Server passes data back up to the Presentation Services.So what I did was cut out presentation services entirely, because the aim of my testing was to resolve reports that were taking 5, 10 minutes to run, and these 5/10 minutes were always down-wind of presentation servicesBear in mind where your dependencies lie, where the stack is coupled. The time itself was always in the database, but you can’t just edit the SQL, because that comes from the BI server. So you intelligently pick the closest point up stream from the bottleneck.2) The other reason why you should reduce complexity is the impact that a change will have on your test plans. If you decide you want to implement a change, maybe based on the results of your testing. How many stages in the test would you like to have to go and change and re-configure your monitoring for – two or twenty?
12 OBIEE stack Report / Dashboard Rendered report Presentation Services DefineMeasureAnalyseReviewImplementOBIEE stackReport / DashboardRendered reportPresentation ServicesLogical SQLData setPhysical SQL statement(s)BI ServerHopefully everyone’s familiar with this picture. Here’s a quick refresherA rather grumpy looking user runs a Request in answersPS sends LSQL to BI ServerBI server sends SQL to DB serverDB server returns results to BI ServerBI Server processes data (aggregates, stitches, etc)BI Server returns data to PS serverPS Server renders and returns through web/app server to the user, who’s hopefully now happy …DatabaseData set(s)Excludes App/Web server & presentation services plug-in
13 Presentation Services DefineMeasureAnalyseReviewImplementOBIEE testing optionsLoad Testing tool(eg. LoadRunner, OATS)User & StopwatchReport / DashboardRendered reportPresentation ServicesLSQLnqcmdData setLSQLSQL ClientBI ServerPhysicalSQLPhysicalSQLSo, how can we do our testing? I’m going to cover first all the options, and then discuss why you’d choose one rather than the otherThe whole point of testing is that it is repeatable, which will normally mean automated.From the top down, here are the ways of doing it[click]- To simulate the complete end-to-end system, you’ve two ways:A user and a stop watch :-)A web-capable testing tool such as LoadRunner or Oracle Application Testing Suite, which simulates a user interacting with OBIEE dashboards or answersMaybe something clever with web services too?- If you want to test from the BI Server onwards only, then you have these options:A utility that comes with OBIEE is called nqcmd. It interfaces with the BI Server using ODBC. I’m going to spend a lot of this presentation talking about it, and will come back to it shortly.You could use another ODBC-capable tool to generate the workload.- Finally, you could run the SQL on the database only. This isn’t as simple as it sounds, because remember that BI Server generates the SQL. How often have you seen the SQL being run on the database and winced or had a DBA shout at you for it? BI Server is a black box when it comes to generating the SQL, all you can do is encourage it through good data modelling both in the database schema and the RPD However, if the focus of your performance testing (remember I talked about defining why you’re doing it) is more to the load testing side of things and you are you happy that there’s no big wins to be had from the BI Server but from tuning the database itself, then you could consider running the SQL directly against it. If all that will change is the execution plan then you can save yourself a lot of time by effectively cutting out OBIEE entirely and treating it purely as a database tuning exercise. Candidates for this approach would be if you were doing things like evaluating new indexes, or parallelism or compression settings. If you’re just interested in the database then its database vendor specific. For Oracle I’d consider:SQL file run through SQL*Plus from the command line (lends itself to scripting)SQL Tuning Sets, which you can feed into SQL Performance Analyzer to run on another databaseOracle RAT - real application testing (which is made up of Database Replay and SQL Performance Analyser)DatabaseData set(s)
14 OBIEE testing options BI Server Database LSQL Physical SQL Data set(s) DefineMeasureAnalyseReviewImplementOBIEE testing optionsnqcmdLSQLBI ServerPhysicalSQLThis is what we opted forDatabaseData set(s)
15 nqcmd nqcmd is part of the OBIEE installation on both unix and windows Command: nqcmd - a command line client which can issue SQL statements against either Oracle BI server or a variety of ODBC compliant backend databases. SYNOPSIS nqcmd [OPTION]... DESCRIPTION -d<data source name> -u<user name> -p<password> -s<sql input file name> -o<output result file name> -D<Delimiter> -C<# number of fetched rows by column-wise binding> -R<# number of fetched rows by row-wise binding> -a (a flag to enable async processing) -f (a flag to enable to flush output file for each write) -H (a flag to enable to open/close a request handle for each query) -z (a flag to enable UTF8 instead of ACP) -utf16 (a flag to enable UTF16 instead of ACP) -q (a flag to turn off row output) -NoFetch (a flag to disable data fetch with query execution) -NotForwardCursor (a flag to disable forwardonly cursor) -SessionVar <SessionVarName>=<SessionVarVAlue>nqcmd is part of the OBIEE installation on both unix and windowsYou can use nqcmd interactively, or from a script
16 nqcmdsetup]$ . ./sa-init.sh setup]$ nqcmd Oracle BI Server Copyright (c) Oracle Corporation, All rights reserved Give data source name: RNMVM01 Give user name: Administrator Give password: Administrator [T]able info [C]olumn info [D]ata type info [F]oreign keys info [P]rimary key info [K]ey statistics info [S]pecial columns info [Q]uery statement Select Option: C Give catalog pattern: Give user pattern: Give table pattern: Time Give column type pattern: TABLE_QUALIFIER TABLE_NAME COLUMN_NAME A_TYPE TYPE_NAME PRECISION LENGTH SCALE RADIX NULLABLE Sample Sales Reduced Time Day Date 9 DATE Sample Sales Reduced Time Week 12 VARCHAR Sample Sales Reduced Time Month 12 VARCHAR Sample Sales Reduced Time Quarter 12 VARCHAR Sample Sales Reduced Time Year 12 VARCHAR Row count: 5Interactively, you can use it to query the logical data model that the RPD exposesNB When you use nqcmd on unix you need to make sure you’ve set the environment variables for OBIEE first, by dot-sourcing sa-init.sh
17 nqcmdperftest]$ cat /data/perftest/lsql/test01.lsql SELECT "D0 Time"."T01 Per Name Week" saw_0 FROM "Sample Sales" WHERE ("D01 More Time Objects"."T31 Cal Week" BETWEEN 40 AND 53) AND ("D01 More Time Objects"."T35 Cal Year" = 2007) ORDER BY saw_0 perftest]$ . /app/oracle/product/obiee/setup/sa-init.sh perftest]$ nqcmd -d AnalyticsWeb -u Administrator -p Administrator -s /data/perftest/lsql/test01.lsql Oracle BI Server Copyright (c) Oracle Corporation, All rights reserved Connection open with info: [State: 01000] [DataDirect][ODBC lib] Application's WCHAR type must be UTF16, because odbc driver's unicode type is UTF saw_ Week Week Week Week Week Week Week Week Week Week Week Week Week Week 53 Row count: 14 Processed: 1 queriesUsing nqcmd to execute a given logical sql script is where the real power in it isHere we take a simple logical sql statement in the file test01.lsqlIt’s run as an input parameter to nqcmd using the s flag
18 nqcmd load test demoThe good bit about being able to run nqcmd from the command line is that you can then call it from scripts.nqcmd on its ownI’ve populated a directory with logical SQL statements that were generated when I ran some dashboards on the SH repository.Using a script I’ve written, I can simulate running each of these reports and dashboardsI’ve disabled caching in BI Server so as to not complicate things.The script I’m going to run basically scans the directory specified and runs everything in it through nqcmd.Setup:- Run Oracle OS Watcher (every 5 seconds, for an hour)~/osw/OSWatcher.sh 60 1- Run Oracle OS Watch grapherjava -jar ~/osw/oswg.jar -i ~/osw/archive/- Run SQL Plus to show Usage Trackingsqlplus / as sysdba@ut1- Tail NQQuery.logtail -f /app/oracle/product/obiee/server/Log/NQQuery.logFirst, check what’s in usage tracking, and NQQuery logOpen two more windows:sqlplus / as./obiee_replay.sh /data/perftest/lsql_test1/This runs a fifth of the scripts (there are 25 in total).< run obiee_replay.sh for a handful of lsql files. >Once we’ve successfully set up the generation of the work, we need to monitor it.Here’s what I’d do.NQQuery – useful for diagnostics. Parse it with grep (get_nq_stats.sh)Usage Tracking – there’s so much you can do with this, and I’ll come back to this in a minuteAdd in SQL Tuning Set captureDB server stats – eg. kc_io, Oracle OS Watcher, sarWhat I’d then do is go to Usage Tracking and extract the run times from there.I’ve built my own table, OBIEE_REPLAY_STATS, which I populate with
19 Usage Tracking or NQQuery.log DefineMeasureAnalyseReviewImplementnqcmdUsage Tracking or NQQuery.logBI ServerLogical SQLDataLogical SQLLogical SQLVery versatile, here’s how you can use itUnix shell scripting, or powershell on WindowsGenerate a set of Logical SQL filesRun them sequentially (twice)Test scriptnqcmd
20 DefineMeasureAnalyseReviewImplementnqcmdMaster test scriptTest scriptnqcmdBI ServerTest scriptDatanqcmdLogical SQLTest scriptAim – script that encompasses the whole test – press a button, does it allLess interaction = less effort, less errorScript– simulate user sleep, randomness- Automation of metric collectionRun nqcmd in parallelit’s just a script, invoke it twice, three times, four timesWrite “user” scripts, that include random report choice and “sleeping”Script include:Random report choiceSleepingLogging to fileSQL interaction, eg triggering SQL Tuning Set collectionTarget is - Press a button to run script and get response time numbers outnqcmdTest scriptnqcmd
21 LoadRunner a.k.a. HP Performance Centre DefineMeasureAnalyseReviewImplementLoadRunner a.k.a. HP Performance CentreSimulates user interaction – HTTP trafficPowerful, but can be difficult to set upAjax complicates thingsDo you really need to use it?ToolsFiddler2FireBugReference:My Oracle Support – Doc ID
22 Defining your test - summary DefineMeasureAnalyseReviewImplementDefining your test - summaryBe very clear what the aim of your test isYou probably need to define multiple testsDifferent points on the OBIEE stack to interfacePick the most appropriate oneWrite everything down!Any questions?
23 DefineMeasureAnalyseReviewImplementOnce you’ve defined your test you need to execute it – and measure the resultsHere’s the ways you should consider measuring the different parts of the stack
24 OBIEE measuring & monitoring DefineMeasureAnalyseReviewImplementOBIEE measuring & monitoringPerfMon (Windows)Enterprise Manager (Oracle)Server metricse.g. : IO, CPU, MemoryPresentation Services plug-inApp ServerWeb ServerApache logOracle OS Watcher (unix)OAS logUsage Trackingsawserver.logAnalytics logPresentation ServicesNQServer.logEnterprise ManagerNQQuery.logASH, AWR, SQL MonitorOnce you’ve decided how much of the stack you’re going to test, you need to set about designing the test and how you’re going to capture your metricsPerformance tests are all about collecting metrics that allow you to make statistically valid and quantifiable conclusions about your system.The primary metric of interest is time. What’s the end-to-end response time, from request to answer, and where’s the time in between spent?If a user complains that a report take five minutes to run but the DBA says they don’t see the query hit the database for the first two, and then it executes in 30 seconds, what’s happened to the other two and a half minutes?Other metrics of interest are the environmental statistics like CPU, memory, and IO, and diagnostic statistics such as the execution plan on the database and lower-level information like buffer gets etc.So – from the top down:[click]Web server, eg. Apache log – first log of the user request coming inApp server, eg. OASPresentation Serivces plugin, Analytics – (this is where you see the error logs when you get 500 Internal Server Error from analytics)sawserver.log - by default this doesn’t record that much, but by changing the logconfig.xml file you can enable extremely detailed logging.This is useful for diagnosing lots of problems, but also if you’re looking to do an accurate profile of where the time in an Answers request is spent. You can see when it receives the user request, when it sends on the logical SQL to the bI Server, and when it receives the data backSee for detailsBI Server – spoilt for choice here. For a production environment I strongly recommend enabling Usage Tracking. For performance work you should also be using NQQuery.log where the variable levels of logging show you logical and physical SQL, BI Server execution plans, response times for each database query run, etc.As well as these two features there is the systemsmanagement functionality which exposes some very detailed counters through windows PerfMon or the BI Management Pack for OEM. You can also use the jmx protocol to access the data through clients like Jconsole or jManageFor the database all the standard monitoring practices apply, depending on what your database is. For Oracle you should be using OEM, ASH, SQL Monitor, etc.And finally, for getting a complete picture of the stack’s performance -- Speak to your users! Maybe not as empirically valid as the other components, but just as important.BI ServerDatabasePerfMon(windows only)systems managementjConsole etcEnterprise ManagerBI Management PackPresentation services
25 DefineMeasureAnalyseReviewImplementNQQuery.logQuery Status: Successful Completion Rows 1, bytes 96 retrieved from database query id: <<10172>> Physical query response time 1 (seconds), id <<10172>> Rows 621, bytes 9246 retrieved from database query id: <<10188>> Physical query response time 10 (seconds), id <<10188>> Physical Query Summary Stats: Number of physical queries 2, Cumulative time 11, DB-connect time 0 (seconds) Rows returned to Client 50 Logical Query Summary Stats: Elapsed time 14, Response time 12, Compilation time 2 (seconds)Useful when prob is suspected on DB, only place that individual physical SQL query response times are keptDatabase query times & row counts
26 Database metrics Oracle DefineMeasureAnalyseReviewImplementDatabase metricsOracleEnterprise Manager’s Performance functionality is fantasticFor pure testing metrics capture you need to go to the tablesV$SQL_MONITOR, etcEM is goodFor pure testing – need to capture data :SQL Tuning SetsGood for capturing behaviour of a set of SQL over time – longest running, most IO, etcLess good for focussing on individual queries because stats are aggregatedSQL Monitor – export from EM (next slide)SQL ServerDMVs, SQL Profiler, PerfMon counters, etcOther RDBMS – pass
27 DefineMeasureAnalyseReviewImplementOracle SQL MonitorGot to mention this – from EM you can export a standalone HTML file that renders like thisbrilliant
28 Measure - summary Lots of different ways to measure DefineMeasureAnalyseReviewImplementMeasure - summaryLots of different ways to measureBuild measurement into your test planAutomate where possibleEasierLess errorLots of different ways to measuredecide what metrics are relevant to your testingLoad testing – system metrics v. ImportantPerf testing indiv report – maybe just response timePlan your measurements as part of the testTrigger collection scripts automagicallyInclude manual collection in test instructions
29 better to use a non-meaningful label analyse it - visualisation DefineMeasureAnalyseReviewImplementAnalysis step is :Collate datastore in sensible way- Raw data- label your testsbetter to use a non-meaningful labelanalyse it- visualisation- analysis will depend on aim of test- eg loadtesting – identify bottlenecks
30 Analysing the data [click] raw data, DefineMeasureAnalyseReviewImplementAnalysing the data[click] raw data,[click] compared to a previous baseline – illustrate varience[click] host metrics - IO graph[click] response time, over time
31 Analysing the data Illustrating the data – colours! [click] DefineMeasureAnalyseReviewImplementAnalysing the dataIllustrating the data – colours![click]Use Excel, conditional formatting is great
32 Analysing the data Average (mean) 3.4 50th percentile (Median) 2 DefineMeasureAnalyseReviewImplementAnalysing the dataResponse time193210Response time123910Average (mean)3.450th percentile (Median)2Think about the data statistically, what do the number represent?Average / mean – often used, but ignores variancePercentile – more representativeStandard Deviation – indication of the varianceSample quantity – statistically validGood site:90th percentile9.1
33 Recording data about the test DefineMeasureAnalyseReviewImplementRecording data about the testDashboardRequestsLogical SQLORA_HASH(QUERY_TEXT)Physical SQLSQL IDsExecution planExecution plan hash idAs well as metrics – record data about your testsWhat are you recording “Response Time” against – report? Physical SQL? etc[click] Ways to capture themFor each test execution aim to record how each level relates to the nextEg lsql -> SQL IDsSQL IDs -> exec plan idMight seem constant, but could change between tests:- changing the RPD could change logical SQL – and therefore physical SQL, SQL ID, exec plan- new index wouldn’t change physical SQL or SQL ID, but exec plan mightHelps with retrospective analysisBe aware how each element links to the nextUseful when analysing test results, to be able to identify a SQL ID in Oracle AWR etc
34 Extending Usage Tracking DefineMeasureAnalyseReviewImplementExtending Usage TrackingOBIEE_REPLAY_STATEMENTSqt_ora_hashquery_textsaw_pathdashboardS_NQ_ACCTSTART_TSROW_COUNTTOTAL_TIME_SECNUM_DB_QUERYQUERY_TEXTQUERY_SRC_CDSAW_SRC_PATHSAW_DASHBOARDOBIEE_REPLAY_STATStestidtestenvqt_ora_hashstart_ts response_time row_count db_query_cntFollows on from how to capture dataSomething i put together to help with analysing statement performance across systemsOra_hash(query_text) common linkDatabase tables obvious place to store raw perf test dataKeeping track of logical SQL statements, often 2-3k in size, difficultUsed ORA_HASH to encodeBuilt new lookup table and new fact tableRunning queries on diff systems, could compare equal statements this way
35 Analyse Iterative approach At end of analyse - options Timebox! DefineMeasureAnalyseReviewImplementDo it rightDon’t “fudge it”Evaluate design / config optionsAt end of analyse - optionsMost likely – change something and re-measure (partitioning, system config, etc)Choose what to doMore tests?IndexesConfig settingsEtcis the test wrong?- redefinecompleted all tests, or reached end of timebox- reviewTimebox!
36 Review Iterative approach Summarise analysis Compare to the test aim DefineMeasureAnalyseReviewImplementRedefine testContinue testingSummarise analysisCompare to the test aimImplementation optionseffort vs benefitBranch – both implement and continue testingWhat did tests show – summariseHave you proved anythingHave you disproved anythingDo you need to test some moreHave you got time to test moreDo you need to define a new set of testsWhat’s practical to implement? Return vs effort.Implement
37 Review Example from one of the review stages DefineMeasureAnalyseReviewImplementReviewExample from one of the review stagesEvaluate IO profile from four different iterationsDefault parallelism – bottleneck at 800MB/sOutput:- implement: reduce DOP- branch : look at auto-DOP in 11gR2
38 Implement Iterative approach DefineMeasureAnalyseReviewImplementDon’t forget to validate your implementationUse your perf test scriptsImplement the chosen optionBASELINE & test it using your perf testsBranch the code line and do more testing if you wanteg us and parallelism, compressionWhen you hit perf problems in Prod, you can use your pre-defined perf tests to assess the scope and nature of the problem
39 Lessons Learnt You won’t get your testing right first time There’s no shame in thatDon’t cook the booksBetter to redefine your test than invalidate its resultsStick to the methodologyDon’t move the goalpostsVery tempting to pick off the “low-hanging fruit”If you do, make sure you don’t get indigestion…TimeboxTest your implementation!Testing – understand more about system as you go – prob want to redefine test – that’s part of the process!Performance Testing is an iterative process. I can’t stress this enough.You will not get it right the first time you do itWhatever you do, you’ll probably miss something or invalidate your tests.Remember that an iterative approach is entirely valid, don’t feel you “got it wrong” and have to fudge the results to cover your mistake. Better to abandon a test and learn from the mistake than produce a “perfect” test that’s complete rubbish.Stick to methodBenefit of also enforces justification for changes, avoid “we’ve always done it that way”don’t move the goalposts.You might find some horrible queriesas you dig into them you notice some obvious “quick wins”If you rush the fix in without completing your first round of testing, you risk invalidating itBE METHODICAL!!!!Timebox the execute/measure/analyse iterations -don’t get lost in diminishing returnsIt’s a good idea to timebox your work, and have regular review pointsTest your implementation!parallel config, not tested properly after implement in test env, nearly got to prod without realisingDon’t get so bogged down in the detail that you miss the wood for the treesYou can end up focussing on perfecting one element of the system at the expense of all the others.
40 How to approach performance testing Think clearlyThis presentation has shown you how to run big workloads against your OBIEE systemBut, resist the temptation to dash off and see what happens when you run a thousand users against your system at once.It’ll be fun, but ultimately a waste of time.You have to define what you’re going to do.You need to define what the ultimate aim is.Are you proving a system performs to specific user requirements? In which case your test definition is almost written for you, you just have to fill in the gapsIf you’re building a performance test for best-practice and all the good reasons I spoke about before then you need to think carefully about what you’ll test.What’s a representative sample of the system’s workload?For example: -Analyse existing usage, pick the most frequently run reportsSPEAK TO YOUR USERS! Which reports do they care about?Be wary of only analysing the reports that users complain about though – you want to be colleting lots and lots of good metrics. What happens when you fix the slow reports – the old “fast” reports will now appear slow in comparison, so you want to have some baselines for them tooI can’t stress this strongly enough.Cary Millsap writes excellently on the whole subject of performance. I can’t recommend highly enough his paper “Thinking Clearly About Performance”, as well as many of the articles on his blog.There are books and books written on how you should approach performance testing and tuning, people like Mr Millsap have built their whole careers around it. It’s way outside the scope of this, but I believe it’s essential to understand the approach to follow, otherwise all your testing can be in vain.It’s not the same as dashing off an OBIEE report that you can bin and recreate next week. Imagine designing your DW schema without good modelling knowledge – or think of ones that you’ve worked with where the person who created it didn’t understand what they were doing. The wasted time and misleading results can be potentially disastrous if you don’t get it right up front.Take my word for it – time invested up front reading and understanding will repay itself ten-fold.Preaching over.
41 Performance Testing OBIEE Iterative approachDefineMeasureAnalyseReviewImplementDo it rightDon’t “fudge it”Evaluate design / config optionsDo more testingQuestions & DiscussionRedefine testDo more testingBe Methodical· ·
42 #EOF Performance tuning links: ** Rittman’s article ** OBIEE EMG threadOracle® Database Performance Tuning Guide 11g Release 1 (11.1)Performance Improvement MethodsI/O Configuration and DesignOracle® Database 2 Day + Data Warehousing Guide 11g Release 1 (11.1)Optimizing Data Warehouse OperationsEliminating Performance BottlenecksOracle® Database VLDB and Partitioning Guide 11g Release 1 (11.1)Oracle® Database Data Warehousing Guide 11g Release 1 (11.1)Part V Data Warehouse PerformanceDBMS_SQLTUNE