1. 1

2. Oracle's Approach to Infrastructure and Application Testing Mughees A. Minhas Vice President, Product Management Oracle

3. 33 The following is intended to outline our general product direction. It is intended for information purposes only, and may not be incorporated into any contract. It is not a commitment to deliver any material, code, or functionality, and should not be relied upon in making purchasing decisions. The development, release, and timing of any features or functionality described for Oracle’s products remains at the sole discretion of Oracle.

4. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 16 4 Why is Testing Important? Cost of 1 hour unplanned production downtime Percent of unplanned downtime can be eliminated through pre-deployment testing and better QA/Development collaboration $100,000+ Incremental cost of fixing problems in production 80-10,000x 80% Source: Gartner Percent of applications deployed without testing 75% Proper testing saves money and reduces unplanned downtime.

5. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 16 5 Testing Challenges  Test quality: despite significant testing, critical issues go undetected  Test development: Test tools and development of comprehensive tests is error prone and time-consuming  Operational agility: Peripheral activities like setting up hardware and deploying applications consumes up to 50% of the time Why are changes not being tested? Provison application under Test Provision testing tool Run functional tests Run load tests Detect bottlenecks Tune Application under Test Apply patches & re-test Develop test scripts Deploy Test scripts Detect issues Define test requirements

6. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 16 6 Testing Challenges Test quality: despite significant testing, critical issues go undetected Test development: Test tools and development of comprehensive tests is error prone and time-consuming  Operational agility: Peripheral activities like setting up hardware and deploying applications consumes up to 50% of the time Why are changes not being tested? Provison application under Test Provision testing tool Run functional tests Run load tests Detect bottlenecks Tune Application under Test Apply patches & re-test Develop test scripts Deploy Test scripts Detect issues Define test requirements

7. 7 Application Infrastructure Changes Application Changes O/S Storage DBMS Middleware Test infrastructure changes separately from application changes Test one change at a time Specialized tools can result in much higher quality and efficient testing Proper test systems and test data mandatory for good testing results Best Practice Approach to Quality Management Infrastructure Testing Tools Infrastructure Testing tools Application Replay Real Application Testing Application Testing tools Application Testing Suite

8. 8

9. 9 Infrastructure Testing Response time testing –Unit testing for response time Throughput testing –To uncover concurrency related issues Significant benefits over traditional tools –Production system behavior emulation –No script development or maintenance Basic Requirements

10. 10 Data-Tier Testing: SQL Response Time

11. 11 Test and measure change impact on response time using real, production SQL statements Identify all plan changes and/or regressions Eliminate SQL regressions with SQL Tuning Advisor and SQL Plan Baselines For data warehouse workloads SPA Exadata simulation helps estimate I/O savings without provisioning Exadata hardware Response Time Testing with SPA SQL Plans + Stats before change Compare SQL Performance Analysis Report SQL Workload STS SQL Plans + Stats after change

12. 12 Data-Tier Testing: Throughput and Concurrency

13. 13 Throughput Testing using DB Replay Database load and performance testing with real production workloads –Production workload characteristics such as timing, transaction dependency, think time, etc., fully maintained Test and measure transaction throughput improvements Identify application scalability and concurrency problems Remediate issues pre-production for risk-free migration Test Production Replay Driver Clients Storage

14. 14 Mid-Tier Testing: Application Replay Captures and replays actual production workload Maintains all production characteristics during replay such as timing, order, concurrency, dependency, etc. Like DB Replay but for the mid-tier Benefits –Faster testing –No script development or maintenance –Better testing –Real production workloads –Broader workflow coverage: captures all flows that arise in practice Authentic replay with minimal divergence Automatic user requests capture Automatic Http-DB Synchronization Automatic request correlation

15. 15 Benefits of Infrastructure Testing Solutions Comparison of LoadRunner & Database Replay 2 20 5 4 80 24 20 DB Replay: ½ month LoadRunner: 7 ½ months Total Testing Time 5 00 15X

16. 149 Days To:From: Complete workflows Partial workflows Low risk High risk Automated Manual intensive Production workloads Artificial workloads Days of development Months of development Benefits of Infrastructure Testing Solutions

17. 17

18. 18 Application Testing If change is within the application –New version or patch –Any customization of application –New functionality Application Testing Suite –Synthetic testing of any changes to application stack or the application logic

19. 19 Application Testing Modularization Maintainability Script Robustness Portability Auto-correlation Basic Requirements

20. 20 Ability to break complex flow into chained flows that are shorter and self-contained –ERP apps like EBS have test flows that often span multiple modules –Test data (input as well as output/results) must move across scripts Enter & Book order Pick Release & Ship Order Generate Invoice Create Receipt Order Mgmnt Shipping Receivables Result output Order to Cash business flow Modularization

21. 21 Maintainability Ability to create function libraries for frequently used functions –Function blocks like "log-in"/"log-out" or other well used functions may be added as function libraries to make test assets easier to maintain and use

22. 22 Portability Ability to port test assets to different environments –Example: Parameterize FORMS connection for better ”portability” nca.connect(317,ConnectMode.Socket,"{{db.myEBSsystem.hostname,EBSServer.mycompa ny.com}}", Integer.parseInt(eval("{{db.myEBSsystem.Forms_Port,9000}}")),""); Recorded value Databank file column used Databank file Indicates Databank variable Update Databank file with new HOST and PORT info when a new test system is used instead of changing in ALL scripts

23. 23 Script Robustness Ability to identify objects that can withstand minor changes –Eliminates need to redevelop scripts frequently Oracle Functional Testing’s accelerators for Oracle applications provides pre-built and optimized object identification rules for EBS, Fusion, Siebel –Provides best possible and most reliable/robust scripts –Can reduce script development by up to 50% Functional Testing

24. 24 Auto-correlation Ability to auto-correlate so that manual modifications are not required –Quicker scripting and more time to do actual testing Oracle Load Testing accelerators for Oracle applications provides pre-built correlation rules for: –E-Business Suite –Fusion –Siebel –JD Edwards Enterprise One Load Testing

25. 25 Oracle Application Testing Suite Supports –Functional/regression testing –Load/Performance testing –Test Management –Web, Siebel, EBS/Forms, ADF, Flex, DB & WebServices Reduce scripting time by ~50% Automatic test asset (script) generation –Load scripts auto-generated using RUEI and Real Application Testing Smart integration with EM diagnostics –JVM Diagnostics for mid-tier –DB diagnostics for data-tier

26. 26 Application Testing Suite Integrated solution for automated functional, load testing and test process management Custom testing accelerators for Oracle Applications, e.g., EBS, Siebel, Fusion, JD Edwards Test Starter Kits for both functional and load testing OpenScript integrated scripting platform for load and functional testing –Intuitive visual scripting interface –Java IDE for powerful extensibility Built-in server monitors and integration with EM Diagnostics to identify performance bottlenecks under load Scalable enterprise architecture built on WLS and Oracle DB Key Features

27. 27

28. 28 Data Masking Application integrity –Generation of data compliant with the application’s requirements Data integrity –Maintaining PK/FK relationship integrity even when primary keys are masked Automated data-relationship discovery –Discovery of sensitive data and relationships across enterprise databases Enterprise-wide rule definition –Defining masking policies for all enterprise databases from a central dashboard Performance –Time taken to execute the entire masking process Basic Requirements

29. 29 Data Masking Reporting –Rich reporting to enable enterprises track and report masking processes Preset pack of rules –Pre-defined set of rules that allows enterprises to use the solution immediately Integration into SLC platforms –Integration of data masking with application development and testing tools DB agnostic –Support for all major databases from mainframe to relational databases O/S agnostic –Availability on all the major operating system platforms Basic Requirements

30. 30 Oracle Data Discovery and Modeling Application Data Model (ADM) –Scans application schemas to model relationships between tables and columns –Extract data relationships from Oracle Applications meta-data –Store referential relationships stored in repository –Enables test data operations such as data sub-setting, masking Sensitive Data Discovery –Pattern-based database scanning –Import from pre-built mask templates –Pre-built Data Masking templates for Oracle applications

31. 31 Oracle Data Masking Deploy secure test system by masking sensitive data Sensitive data never leaves the production database Extensible template library and policies for automation Sophisticated masking: Condition-based, compound, deterministic, at-source Integrated masking and cloning LAST_NAMESSNSALARY HODGES111 — 23-111160,000 JOHNSON222-34-134540,000 LAST_NAMESSNSALARY AGUILAR203-33-323440,000 BENSON323-22-294360,000 ProductionTest

32. 32 Masking Techniques Support Compound masking –Multi-column groups, e.g. address (street, city, state, zip) Condition-based masking –Mask national identifiers, e.g. social security number, by country-specific format Deterministic (Repeatable) –Repeatable consistent masking, e.g. referential integrity across databases, or consistency across refreshes Reversible –Key-based format preserving masking, e.g. outsourced data processing Shuffle –Randomization within population set. Perturbation –Random mask generation around a value, e.g. date of birth At-Source Masking –Ability to mask data as it leaves production, i.e., at the source

33. 33 Automatic data extraction rules from ADM Estimate subset before execution Parameter-driven dynamic subset generation Unique to Oracle: Data Pump-based subset creation Oracle Data Subsetting Define new Application Data Model Create Data Subset Definition Extract Data Subset Export / Import In-Place Delete Production Test

34. 34

35. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 16 35 Testing Challenges  Test quality: despite significant testing, critical issues go undetected  Test development: Test tools and development of comprehensive tests is error prone and time-consuming  Operational agility: Peripheral activities like setting up hardware and deploying applications consumes up to 50% of the time Why are changes not being tested? Provison application under Test Provision testing tool Run functional tests Run load tests Detect bottlenecks Tune Application under Test Apply patches & re-test Develop test scripts Deploy Test scripts Detect issues Define test requirements

36. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 16 36 Testing Challenges  Test quality: despite significant testing, critical issues go undetected  Test development: Test tools and development of comprehensive tests is error prone and time-consuming Operational agility: Peripheral activities like setting up hardware and deploying applications consumes up to 50% of the time Why are changes not being tested? Provison application under Test Provision testing tool Run functional tests Run load tests Detect bottlenecks Tune Application under Test Apply patches & re-test Develop test scripts Deploy Test scripts Detect issues Define test requirements

37. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 16 37 How To Get Agility  Automate time consuming, repetitive tasks – Provision test tools and application – Provision scripts – Orchestrate testing process end-to-end What can be improved?  Simplify  Reporting  Deep diagnostics & root cause analysis Provison application under Test Provision testing tool Run functional tests Run load tests Detect bottlenecks Tune Application under Test Apply patches & re-test Develop test scripts Deploy Test scripts Detect issues Define test requirements Automate the testing process

38. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 16 38 What Is The Solution?  Infrastructure-as-a-Service (IaaS) – Provisioning of Virtual Machines on provided infrastructure  This will only resolve the provisioning of Application under Test and Test tools – only 10-15% of the full solution IaaS based testing cloud Provison application under Test Provision testing tool Run functional tests Run load tests Detect bottlenecks Tune Application under Test Apply patches & re-test Define test requirements and process Deploy Test scripts Detect issues IaaS Develop test scripts Define test requirements

39. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 16 39 What Is The Solution?  Software-as-a-Service (SaaS) – Software (for test automation) as a cloud solution. Available on-demand  Only addresses the test execution and (possibly) issue/bottleneck identification  No provisioning of the Application under Test  Possibly no monitoring of internal applications, depending on solution SaaS based testing cloud Provison application under Test Provision testing tool Run functional tests Run load tests Detect bottlenecks Tune Application under Test Apply patches & re-test Deploy Test scripts Detect issues SaaS Define test scripts Define test requirements

40. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 16 40 What Is The Solution?  Combination of – Provisioning of cloud resources (IaaS) – Software on demand (SaaS) – Test orchestration and automation of the whole testing process including test scripts – Application tuning and patching – Result reporting  Built with semantic understanding of testing artifacts like testing tools, applications, test scripts, etc. – it is not just VMs IaaS and Saas Combination Solution Oracle Testing-as-a-Service Platform IaaS SaaS Orchestration Orchestration, patching, tuning...

41. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 16 41 Oracle Testing-as-a-Service Self-service private testing cloud IaaS Platform Cloud Policy Framework Self-Service Solutions http(s) Test Admin, Test Designer & Managers Test Engineers

42. 42

43. © 2012 Oracle Forrester Real Application Testing TEI Study Multi-customer Study Demonstrates Strong ROI, Business Value Risk adjusted ROI of 224% with a payback period of 5.9 months Increased productivity gain to nearly $4.8M over 3 years Reduced capital spending on servers by 50% Increased availability and uptime of critical business applications (Excludes $20,655 annual business up time savings)

44. 44 Oracle Testing Solutions Testing Application Changes Testing Infrastructure Changes Test Data and Lab Management Oracle Functional Testing Oracle Load Testing Oracle Test Manager SQL Performance Analyzer Application Replay Data Masking Data Sub-setting Database Replay Data Discovery and Modeling Cloud Services Testing Tools Test Data Management Cloud Delivery Platform Testing as-a-Service

45. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 16 45 Key Take-Aways  Distinguish between infrastructure vs. application changes  Don’t employ “one-tool-fits-all” approach – Select right tool for right problem  Proper test systems and test data mandatory for good results  Gain operational agility with testing cloud services

46. Copyright © 2012, Oracle and/or its affiliates. All rights reserved.Insert Information Protection Policy Classification from Slide 16 46

47. 47

48. 48