Produced by Wellesley Information Services, LLC, publisher of SAPinsider. © 2016 Wellesley Information Services. All rights reserved. Advanced Performance Optimization with SAP BW 7.3 and SAP BW 7.4 Dr. Bjarne Berg COMERIT

1 In This Session Get practical tips and techniques for maintaining and cleaning an SAP BW system for optimal performance, including PSA optimization, compression, maintaining statistical cubes, and controlling growth, reducing log file sizes, removing DTP temporary storage, DTP error logs, and temporary database objects Reduce the size of an SAP BW system by as much as 70% by taking steps such as removing PSAs, aggregating, and optimizing InfoCubes, and implementing the new LSA++ architecture See how to clean batch tables and reduce the footprints of un-needed data Learn how to take advantage of new performance features in SAP BW 7.3 and BW 7.4

2 What We’ll Cover SAP BW 7.3 performance basics Housekeeping tasks in SAP BW 7.3 and 7.4 Demo: Optimal SAP BW 7.4 on HANA performance 12-step pre-HANA migration cleanup tasks Demo: SAP BW 7.4 performance monitoring BW optimization after HANA migration LSA vs. LSA++ optimization and SPOs Wrap-up

3 Explore the use of line item dimensions for fields that are frequently conditioned in queries. This model change can yield faster queries. BW 7.3 (Non-HANA) InfoCube Design — Line Item Dimensions Line item dimensions are basically fields that are transaction-oriented Once flagged as a line item dimension, the field is actually stored in the fact table and has no table joins This may result in improvements to query speeds for cubes not in BWA or HANA Source: SAP AG

4 BW 7.3 (Non-HANA) InfoCube Design — High Cardinality Flags High-Cardinality flag for large InfoCubes with more than 10 million rows At this company there were 11 InfoCubes with a ratio of more than 30% of the records in the dimensions vs. fact table SAP recommends for Indexing and performance reasons to flag these as “high-cardinality” dimensions. However, it has minor impact to smaller cubes. In this example, there were four medium and large InfoCubes that are not following the basic design guidelines, and subsequently had slow performance Many companies should redesign large InfoCubes with high-cardinality to take advantage of the standard performance enhancements available Real Example

5 BW 7.3 (Non-HANA) DSO Design and Locks on Large Oracle Tables In this example, many of the very large DSOs are not partitioned, and several objects have over 250 million records Additionally, 101 DSO objects were flagged as being reportable. This resulted in System IDs (SIDs) being created during activation. Combined, these resulted in frequent locks on the Oracle database and failed parallel activation jobs Partition DSOs. The lock on very large DSOs during parallel loads are well known and SAP has issued several notes on the topic: “ODS object: Activation fails – DEADLOCK” and “Oracle deadlocks, ORA ” Real Example

6 SAP BW 7.3 has a new, step-by-step wizard that allows you to generate data flows from flat files or existing data sources The BW 7.3 DataFlow Generation Wizard A great benefit is that the wizard works against any InfoProvider; i.e., you can use the wizard to create loads from DSOs to DSOs or InfoCubes This wizard reduces the number or manual steps needed to load data. It also simplifies the development process and makes ETL work much easier.

7 Database Performance (Non-HANA Systems) Database statistics are used by the database optimizer to route queries. Outdated statistics leads to performance degradation. Outdated indexes can lead to very poor search performance in all queries where conditioning is used (i.e., mandatory prompts) The current sampling rates for this example were too low, and statistics should only be run after major data loads, and could be scheduled weekly For many systems, database statistics are outdated and may cause database performance to perform significantly poorer than otherwise would be the case. Sampling should often be changed and process chains may be re-scheduled. Real Example

8 Statistics and Indexes in BW: Top-10 InfoCubes Performance In this real example, several of the largest InfoCubes have outdated Database statistics, which may lead to sub-optimal performance. These should be updated as soon as possible. Updating DB stats can be done in a weekly job that runs ever weekend. Real Example

9 Another example: Global Cache and Query performance Today over 40% of all queries access the database instead of the cache. This means that the ‘cache hit-ratio’ is only 60%. That ratio is very low, and can be improved upon by changing the size of the cache on the server BW comes with the default setting of 100 MB for local cache and 200 MB for global cache. In the last month, the system had 31,955 query execution, of which 12,782 queries accessed the database instead of the cache. If the hit-ratio is improved by 20%, it would result in almost 59 minutes of less total wait times per day for the users. Real Example

10 The OLAP Memory Cache Size Utilization A global cache size of only 200 MB (default setting) is too small. Most companies increase this to 400 MB. The settings can be changed in RSCUSTV14. Unfortunately, in this example someone has decreased the global cache setting from 200 MB to 136MB in the production system, instead of increasing it Real Example The decreased cache size may result in lower hit-ratio, since the cache results gets ‘invalidated’ once the max size is filled.

11 Changing the Cache Size The size of OLAP Cache is also physically limited by the amount of memory set in system parameter rsdb/esm/buffersize_kb. The settings are available in RSPFPAR and RZ11. Since the global cache size is a logical value and the Export/Import SHM gives a physical limit, and also considering that other applications (such as BCS) might use the Export/Import SHM, SAP recommend to set the global cache parameter maximally to 90% of the Export/Import SHM buffer.

12 Example: Query Performance of most used InfoCubes While many SD query executions are very fast (13-14 seconds), other reports in Finance and Inventory Management is very slow Real Example

13 Example: Query Performance – 4,300 reports per month Of the total queries executed in the system in April-May 2015, 4,075 were from end-users. The average query run-time took 39.7 seconds, while some reports took 8-9 minutes to run. Of the slowest interface, the reports in BEx Web Java was executed 2,809 times and took an average of 45 seconds to execute. In an HANA environment, the same query would have completed in an estimated time of 7 second. Installing HANA could result in 426 hours less waiting for reports by the business users each year Real Example

14 Another Example: Query Performance - Details 1,095 query executions took over 69.9 seconds to execute. The daily inventory report, which is run 7 times each workday takes 2.5 min. Real Example

15 Temporary Tables and Inconsistencies There are several temp tables that can be obsolete in the older systems (mostly from previous jobs and internal reports run). This could be cleaned up using the program SAP_DROP_TMPTABLES when users and batch jobs are not running on the system (note: this program will remove all temp tables except temp hierarchy tables and may stop jobs in-progress). It is therefore recommended that this is done during a weekend outage. Once this is done, you could consider running SAP_UPDATE_DBDIFF to clean all obsolete temporary entries. These two efforts can reduce the BW system size and also provide for a cleaner environment. Details on how to remove the temporary tables and entries in can be found in note number

16 BI Content Consistency Checks (Optional) If you suspect inconsistency in the BI content, or are planning to deploy new BI content, you can run the BI Content Analyzer. This can be as a transparent table or loaded to a DSO (tcode RSBICA) The automated BI Content Analyzer Checks include:  Inactive Transfer Structure checks  List of InfoObjects without an InfoObject Catalog  Inconsistent Roles check  Routines that refer to fixed, programmed structures  Query Elements with Duplicate GUIDs  Several Object Collection Errors  Several Object Status checks  Many checks for Inconsistent Naming Conventions To help plan any testing, get a list of where the objects in the system are used (SAP Note: 28022)

17 SAP HANA and SAP BW 7.4 For BW 7.4 on HANA, SAP has continued to move more of the process intensive functions from the application to the DB server The benefits of this approach are dramatically faster data activation, data transformations, and query executions This takes advantage of the performance improvements of an in-memory DB It also reduces the need for data transfers between application and DB server

18 Optimized Transformations and Data Activation Activating data for reporting against the BW NIRV table was a very slow process in the older BW systems. With some number range buffering (BW 7.0) and package fetch (BW 7.3) this was somewhat improved, but it was still not extremely fast To speed up activation in BW 7.3 on HANA, and transformations in BW 7.4 on HANA, more of this was moved to the database layer Currently, transformations include conversions, masterdata reads, formulas, data mapping, and SQL scripts using most InfoProviders and loading to a DSO

19 What We’ll Cover SAP BW 7.3 performance basics Housekeeping tasks in SAP BW 7.3 and 7.4 Demo: Optimal SAP BW 7.4 on HANA performance 12-step pre-HANA migration cleanup tasks Demo: SAP BW 7.4 performance monitoring BW optimization after HANA migration LSA vs. LSA++ optimization and SPOs Wrap-up

20 Pre-Steps — Cleaning Up Your BW System You can save significant amounts of work by doing a cleanup effort before you start your HANA migration or BW upgrade project For example, an international company had a BW system with over 108TB, with only 38TB in the production box and the remaining data on their Near-Line Storage (NLS) solution This cleaned BW system saved them potentially millions of dollars in hardware and HANA licensing costs It is not unusual to reduce a BW system size by 20-30% during a cleanup effort

21 The SAP_BW_HOUSEKEEPING Task List If you are on 7.0 SP32 or higher, you can generate an SAP BW Housekeeping task list and get automated help in cleaning the system weeks before upgrading it You first have to install the program from SAP Note before you can generate the SAP_BW_HOUSEKEEPING task list using tcode STC01 7.Re-assigns requests written into the incorrect PSA partition 8.Verifies DataSource segment assignment to PSA 9.Deletes the entries no longer required in table RSIXW 10.Clears all OLAP Cache parameters 11.Repairs InfoCube fact table indices at Data Dictionary level 12.Reorganizes and deletes bookmark IDs and view IDs 1.Checks BW metadata with DDIC 2.Deletes RSTT traces 3.Deletes BW statistical data 4.Deletes aggregate data via deactivation 5.Ensures partitioned tables are correctly indexed for PSA 6.Ensures request consistencies in the PSA

22 A Tool to Help Migrate and Clean Up SAP has created a cockpit to:  Clean up the SAP BW system  Reduce system size  Conduct pre-checks (readiness checks)  Size the system  Find sub-optimal code (i.e., transformations)  Look at table distributions and loads There are over 235 tests in this tool These tools are thanks to SAP’s Marc Bernard and his team at SAP Labs Canada

23 More Tips to Make the Database Smaller Use write-optimized DSOs as first level data stores. These can easily be off-loaded out of main memory in HANA and save you money. Keep your Persistent Staging Tables (PSA) clean. BTW: The PSA is often not needed at all in BW 7.4. If you are on BW 7.3 Service Pack 8 and HANA with at least Service Pack 5, the write-optimized DSOs and PSAs are flagged as “early unload” from the HANA memory. This will help you keep the system smaller and require less memory. In HANA 1.0 SP-10 we got dynamic tiring that automatically manages objects in memory/disk You can also flag other InfoCubes, DSOs, tables, and partition them as “not active.” If you do so, they will only be loaded into memory when actually required. The sizing program in SAP Note takes these size savings settings into account when sizing your HANA system

24 What We’ll Cover SAP BW 7.3 performance basics Housekeeping tasks in SAP BW 7.3 and 7.4 Demo: Optimal SAP BW 7.4 on HANA performance 12-step pre-HANA migration cleanup tasks Demo: SAP BW 7.4 performance monitoring BW optimization after HANA migration LSA vs. LSA++ optimization and SPOs Wrap-up

25 Demo: Optimal SAP BW on HANA Performance

26 What We’ll Cover SAP BW 7.3 performance basics Housekeeping tasks in SAP BW 7.3 and 7.4 Demo: Optimal SAP BW 7.4 on HANA performance 12-step pre-HANA migration cleanup tasks Demo: SAP BW 7.4 performance monitoring BW optimization after HANA migration LSA vs. LSA++ optimization and SPOs Wrap-up

27 12 Pre-Steps — Cleaning Up Your BW System 1. Clean the Persistent Staging Area (PSA) for data already loaded to DSOs. 2. Delete the Aggregates (summary tables). They will not be needed again. 3. Compress the E and F tables in all InfoCubes. This will make InfoCubes much smaller. 4. Remove data from the statistical cubes (they start with the technical name of 0CTC_xxx). These contain performance information for the BW system running on the relational database. You can do this using the transaction RSDDSTAT or the program RSDDSTAT_DATA_DELETE to help you. 5. Look at the log files, bookmarks, and unused BEx queries and templates (transaction RSZDELETE). 6. Remove as much as possible of the DTP temporary storage, DTP error logs, and temporary database objects. Help and programs to do this are found in SAP Notes and

28 12 Pre-Steps — Cleaning Up Your BW System (cont.) 7. For write-optimized DSOs that push data to reportable DSOs (LSA approach), remove data in the write-optimized DSOs. It is already available in higher-level objects. 8. Migrate old data to Near-Line Storage (NLS) on a small server. This will still provide access to the data for the few users who infrequently need to see this old data. You will also be able to query it when BW is on HANA, but it does not need to be in-memory. 9. Remove data in unused DSOs, InfoCubes, and files used for staging in the BW system. This includes possible reorganization of master data text and attributes using process type in RSPC.

29 12 Pre-Steps — Cleaning Up Your BW System (cont.) 10. You may also want to clean up background information stored in the table RSBATCHDATA. This table can get very big if not managed. You should also consider archiving any IDocs and clean the tRFC queues. All of this will reduce the size of the HANA system and help you fit the system tables on the master node. 11. In SAP Note , SAP provides some ideas on how to keep the Basis tables from growing too fast in the future; if you are on Service Pack 23 on BW 7.0 or higher, you can also delete unwanted master data directly (see SAP Note ). 12. Finally, you can use the program RSDDCVER_DIM_UNUSED to delete any unused dimension entries in your InfoCubes to reduce the overall system size.

30 Deep-Dive: Cleaning Up Your BW System

31 What We’ll Cover SAP BW 7.3 performance basics Housekeeping tasks in SAP BW 7.3 and 7.4 Demo: Optimal SAP BW 7.4 on HANA performance 12-step pre-HANA migration cleanup tasks Demo: SAP BW 7.4 performance monitoring BW optimization after HANA migration LSA vs. LSA++ optimization and SPOs Wrap-up

32 Demo: SAP BW 7.4 Performance Monitoring In this demo we will explore the BW 7.4 on HANA DBA Cockpit Features

33 Demo: SAP BW 7.4 Performance Monitoring (cont.)

34 What We’ll Cover SAP BW 7.3 performance basics Housekeeping tasks in SAP BW 7.3 and 7.4 Demo: Optimal SAP BW 7.4 on HANA performance 12-step pre-HANA migration cleanup tasks Demo: SAP BW 7.4 performance monitoring BW optimization after HANA migration LSA vs. LSA++ optimization and SPOs Wrap-up

35 Converting InfoProviders and/or Data Flows While not required, InfoCubes can be optimized further for HANA performance This basically means “flattening” the data structures and removing the dimensions in BW from the physical layer (they still look as if they exist) Many refer to this optional step as a “functional migration” and do this after the HANA migration has been completed, often as a separate initiative (see SAP Note )

36 HANA Optimized BW 7.4 DSOs and Performance Improvements BW optimized DSOs are now created by “default” in HANA This means that data activations are done much faster at the HANA database layer The change log is kept in a calculation view resulting in smaller DSOs HANA optimized DSOs are also available for BW 7.3, but now they are created by default, so do not convert DSOs to HANA-optimized. Fast activation is available for all standard DSOs without conversion to HANA-optimized.

37 Converting InfoProviders and/or Data Flows To help you, the SAP Migration Cockpit also allows you to migrate your data flows from 3.x to Data Transfer Processes (DTPs) as used in versions 7.0 and higher If you convert the data flows, you get better automated data package DTP optimization, which loads data faster into HANA You can also simulate the data flow before you do the real conversion. When doing so, data is loaded for both versions (3.x and 7.x) of the dataflows and the results are stored in cluster tables. The data is then compared to verify that the dataflow after migration calculates the same data as it did before migration. Since the differences are displayed separately, you can analyze the results and changes in details

38 What We’ll Cover SAP BW 7.3 performance basics Housekeeping tasks in SAP BW 7.3 and 7.4 Demo: Optimal SAP BW 7.4 on HANA performance 12-step pre-HANA migration cleanup tasks Demo: SAP BW 7.4 performance monitoring BW optimization after HANA migration LSA vs. LSA++ optimization and SPOs Wrap-up

39 EDW Design vs. Evolution An organization has two fundamental choices: 1. Build a new well architected EDW 2. Evolve the old EDW or reporting system Both solutions are feasible, but organizations that select an evolutionary approach should be self-aware and monitor undesirable add-ons and “workarounds” Failure to break with the past can be detrimental to an EDW’s long-term success …

40 Data Design The Use of Layered Scalable Architecture (LSA) The LSA consists logically of:  Acquisition layer  Harmonization/quality layer  Propagation layer  Business transformation layer  Reporting layer  Virtualization layer Since SAP BW 7.3 SP3 we have had a set of 10 templates to help build a layered data architecture for large- scale data warehousing

41 Example: Current LSA Data Architecture in SAP BW BWERP Germany FLEXIBLE REPORTING Europe (excl. Germany) Europe 2 Europe 3 USA Americas 1 Americas 2 Asia Germany BUSINESS TRANS. Europe (excl.Germany) Europe 2 Europe 3 USA Americas 1 Americas 2 Asia Germany DATA PROPAGATION Europe (excl.Germany) Europe 2 Europe 3 USA Americas 1 Americas 2 Asia DATA ACQUISITION Data Acquisition Data Source Transfer Rule Info Source ERP Table Germany CORPORATE MEMORY Europe (excl.Germany) Europe 2 Europe 3 USA Americas 1 Americas 2 Asia Germany DIMENSIONAL REPORTING Europe (excl. Germany) Europe 2 Europe 3 USA Americas 1 Americas 2 Asia 8 semantic partitions 6 LSA Layers 41 total objects

42 Example: Simplified LSA++ Data Architecture BWERP FLEXIBLE REPORTING BUSINESS TRANS. Europe Americas Asia DATA PROPAGATION Europe Americas Asia DATA ACQUISITION Data Source Transfer Rule Info Source ERP Table CORPORATE MEMORY Europe Americas Asia DIMENSIONAL REPORTING Remove 5 semantic partitions Remove 3 LSA layers 41 shrinks to 9 total objects

43 Conformed Reportable DSO Write Optimized DSO EDW — Complex Layered Architectures This BPC on BW system was experiencing substantial load performance issues Some of this was due to underlying SAP BW configuration, while some was due to the technical configuration of the data store architecture and data flow inside SAP BW Production Issues included: 1)Dependent jobs not running sequentially, i.e., load from Summary cube to Staging cube is sometimes executed before the Summary cube data is loaded and activated, resulting in zero records in the Staging cube. 2)Long latency with 6 layers of PSA, DSOs, and InfoCubes before consolidation processes can be executed. FIGL_D15SFIGL_D10SFIGL_D08FIGL_D13SFIGL_D11S FIGL_D21FIGL_D17FIGL_D14FIGL_D20FIGL_D18 GL Summary Cube (FIGL_C03) BPC Staging Cube (BPC_C01) Consolidation Cube (OC_CON) ECC 6.0 Asia- Pacific ECC 6.0 North-America ECC 4.7 Latin-America R/3 3.1i EU ECC 4.7 ASIA Persistent Staging Area (PSA) Consolidation Processes: 1)Clearing 2)Load 3)Foreign Exchange 4)Eliminations 5)Optimizations Real Example

44 Write Optimized DSO Fixes to Complex EDW Architecture The fix to this system included removing the conformed DSO layer, with BEx flags for DataStores that are never reported on Also, the BPC Staging cube served little practical purpose since the data is already staged in the GL Summary cube and the logic can be maintained in the load from this cube directly to the consolidation cube FIGL_D15SFIGL_D10SFIGL_D08FIGL_D13SFIGL_D11S GL Summary Cube (FIGL_C03) Consolidation Cube (OC_CON) ECC 6.0 Asia- Pacific ECC 6.0 North-America ECC 4.7 Latin-America R/3 3.1i EU ECC 4.7 ASIA Persistent Staging Area (PSA) Consolidation Processes: 1)Clearing 2)Load 3)Foreign Exchange 4)Eliminations 5)Optimizations Long-term benefits included reduced data latency, faster data activation, less data replication, and smaller system backups as well as simplified system maintenance. Real Example

45 EDW Data Design — Classical Use of MultiProvider Hints in BW If a query has restrictions on this characteristic, the OLAP processor is already checked to see which part of the cubes can return data for the query. The data manager can then completely ignore the remaining cubes. Problem: To reduce data volume in each InfoCube, data is partitioned by Time period. A query must now search in all InfoProviders to find the data. This is very slow. Solution: We can add “hints” to guide the query execution. In the RRKMULTIPROVHINT table, you can specify one or several characteristics for each MultiProvider, which are then used to partition the MultiProvider into BasicCubes. An entry in RRKMULTIPROVHINT only makes sense if a few attributes of this characteristic (that is, only a few data slices) are affected in the majority of, or the most important, queries ( SAP Note See also and ).

46 BW 7.3 and Higher (non-HANA) — Semantic Partitioned Objects (SPO) When DataStores and InfoCubes are allowed to grow over time, the data load and query performance suffers Normally objects should be physically partitioned when the numbers of records exceed 100 – 200 million  However, this may be different depending on the size of your hardware and the type of database you use In SAP BW 7.3 we get an option to create a Semantic Partitioned Object (SPO) through wizards  You can partition based on fields such as calendar year, region, country, etc.

47 Data Design — Semantic Partitioned Objects When an SPO is created, a reference structure keeps track of the partitions. The structure is placed in the MultiProvider for querying. SPO Wizards create all Data Transfer Processes (DTP), transformations, filters for each data store, and a process chain

48 What We’ll Cover SAP BW 7.3 performance basics Housekeeping tasks in SAP BW 7.3 and 7.4 Demo: Optimal SAP BW 7.4 on HANA performance 12-step pre-HANA migration cleanup tasks Demo: SAP BW 7.4 performance monitoring BW optimization after HANA migration LSA vs. LSA++ optimization and SPOs Wrap-up

49 Where to Find More Information Bjarne Berg and Penny Silvia, Introduction to SAP HANA (3rd Edition) (SAP PRESS, 2014).  Michaela Pastor, “SAP Business Warehouse 7.4” (SCN, November 2014).  SAP NetWeaver 7.4 on the SAP Help Portal  SAP BI content release note for BW 7.4

50 7 Key Points to Take Home SAP BW 7.4 is the first release to take full advantage of SAP HANA Some of the functions in 7.4 are also available to non-HANA customers The new CompositeProviders and the Open ODS View make HANA and BW tightly integrated and capable to support EDWs better You should break from the past and start designing with the new BW 7.4 features in mind The new monitoring features in the BW DBA Cockpit and the HANA systems make it much easier to see what is occurring from a database level for the non-basis team Before you size your system, clean it up and save hardware costs All customers should consider the BW move to HANA in 2016!

51 Your Turn! How to contact me: Dr. Berg Please remember to complete your session evaluation

52 Disclaimer SAP and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP SE (or an SAP affiliate company) in Germany and other countries. All other product and service names mentioned are the trademarks of their respective companies. Wellesley Information Services is neither owned nor controlled by SAP SE.