Presentation is loading. Please wait.

Presentation is loading. Please wait.

AL_Tables Jim Kirk August, 2011. ©2011 SAP AG. All rights reserved.2 Agenda – metadata tables 1) AL_Statistics 2) AL_History 3) AL_Lang & AL_Langxmlltext.

Similar presentations


Presentation on theme: "AL_Tables Jim Kirk August, 2011. ©2011 SAP AG. All rights reserved.2 Agenda – metadata tables 1) AL_Statistics 2) AL_History 3) AL_Lang & AL_Langxmlltext."— Presentation transcript:

1 AL_Tables Jim Kirk August, 2011

2 ©2011 SAP AG. All rights reserved.2 Agenda – metadata tables 1) AL_Statistics 2) AL_History 3) AL_Lang & AL_Langxmlltext 4) AL_Cache & AL_Cache_Stats 5) AL_Audit & AL_Audit_Info The intent is to present the logic behind some of the resolutions you have all used or seen in the past. This content is for primary support engineers and not to be posted to case notes, external KBA or wiki content. You have no doubt heard these before: My logs aren't being deleted, why? Uncheck Use Collected Statistics and rerun the job. Change the Cache Type to In-Memory. I get Virtual Memory warnings or errors. My AL_Statistics table is huge. My Management Console Admin/Batch is very slow, why? Y

3 Data Integrator Background

4 ©2011 SAP AG. All rights reserved.4 Acta Roots: Business ObjectsBusiness Objects's Data Integrator is a data integration and ETL tool that was previously known as ActaWorks.data integrationETL Acta Technology_ posted in April 2002 Acta provides the industry's first and only batch and real-time data integration platform expanding extraction, transformation, and loading (ETL) technology beyond its traditional data warehousing role to more effectively integrate business intelligence (BI), enterprise resource planning (ERP), customer relationship management (CRM), supply chain management (SCM), and eCommerce solutions. With Acta's highly scalable data integration software, enterprises can rapidly design and deploy data flows that provide customers, suppliers, employees, and partners with the critical information they need, faster and at lower cost.

5 ©2011 SAP AG. All rights reserved.5 Metadata AL Tables ReferenceGuide\Metadata in Repository Tables and Views This will contain a small sampling of the tables and their purpose. It will include sample sql to query against the content for reporting.

6 AL_Cache & AL & Cache_Stats

7 ©2011 SAP AG. All rights reserved.7 References Wiki DI caching -> http://wiki.sdn.sap.com/wiki/x/aYJCEhttp://wiki.sdn.sap.com/wiki/x/aYJCE Wiki Cache transforms -> http://wiki.sdn.sap.com/wiki/x/94NCEhttp://wiki.sdn.sap.com/wiki/x/94NCE

8 ©2011 SAP AG. All rights reserved.8 Can I determine the size of the cache (say one table)? table size = (in bytes) # of rows * # of columns * 20 bytes (average column size) * 1.3 (30% overhead) Example: a modest transactional table with manageable data types. 250000 * 20 * 20 * 1.3 = 169mb

9 ©2011 SAP AG. All rights reserved.9 What do we know up front uses cache? http://wiki.sdn.sap.com/wiki/display/BOBJ/Cached+Transforms http://wiki.sdn.sap.com/wiki/display/BOBJ/Cached+Transforms

10 ©2011 SAP AG. All rights reserved.10 Definition - Collect statistics for optimization Select this check box if you want to collect statistics that the Data Services optimizer will use to choose an optimal cache type (in-memory or pageable). This option is not selected by default. For more information, see Caching sources in the Performance Optimization Guide. For more information, see the Using Caches chapter in the Performance Optimization Guide

11 ©2011 SAP AG. All rights reserved.11 Caching Data Reference Using Caches in Performance Opt Guide You can improve the performance of data transformations that occur in memory by caching as much data as possible. By caching data, you limit the number of times the system must access the database. SAP BusinessObjects Data Services provides the following types of caches that your data flow can use for all of the operations it contains: In-memory Use in-memory cache when your data flow processes a small amount of data that fits in memory. Pageable cache Use pageable cache when your data flow processes a very large amount of data that does not fit in memory. When memory-intensive operations (such as Group By and Order By) exceed available memory, the software uses pageable cache to complete the operation. Pageable cache is the default cache type. To change the cache type, use the Cache type option on the data flow Properties window. Note: If your data fits in memory, it is recommended that you use in-memory cache because pageable cache incurs an overhead cost. SAP BusinessObjects Data Services uses cache statistics collected from previous job runs to automatically determine which cache type to use for a data flow. Cache statistics include the number of rows processed. The default cache type is pageable. the software can switch to in-memory cache when it determines that your data flow processes a small amount of data that fits in memory. A cache (pronounced CASH) is a place to store something temporarily

12 ©2011 SAP AG. All rights reserved.12 Definitons (on Oracle) AL_Cache_Stats AL_Cache When the collect statistics for optimization is checked these two tables populate based on design and cache used while running in pageable mode.

13 ©2011 SAP AG. All rights reserved.13 Simple test case illustrating collect stats Row gen for 20mil rows; two lookups on a table with addresses, a groupby on region + zip code and orderby region; map op to discard all and xml template for the target. This design will leverage paging. On a fresh repo or after truncating the al_cache and al_cache_statistics tables run the job with Collect statistics for Optimization and for monitoring checked. NOTE: Whenever the job is run to collect stats it will run in pageable. Then run this sql on the repo: select s.STATNAME,s.statvalue, c.text_value from al_cache_stats s, al_cache c where s.object_key = c.parent_objid and c.SEQNUM <> '-1' and s.STATNAME = 'cachesize Statvalue in bits. The following illustrations are for #1 the sql results, and #2 the activity on Pcache directory.

14 ©2011 SAP AG. All rights reserved.14 Al_cache_stats will have four rows per transformation using cache With four objects\functions requiring cache add four the stats up.12,309,256 bytes or 12.3mb

15 ©2011 SAP AG. All rights reserved.15 Now run using collected stats Trace Log: (12.2) 12-15-11 10:28:28 (20296:18372) DATAFLOW: Data flow is started. (12.2) 12-15-11 10:28:28 (20296:18372) DATAFLOW: Cache statistics determined that data flow uses caches with a total size of bytes. This is less than(or equal to) the virtual memory bytes available for caches. Statistics is switching the cache type to IN MEMORY.(12.2) 12-15-11 10:28:28 (20296:18372) DATAFLOW: Data flow using IN MEMORY Cache. NOTE: DONT confuse the cache types here with the OS resources. On a Windows machine with virtual memory, the computer can look for areas of RAM that have not been used recently and copy them onto the hard disk. The area of the hard disk that stores the RAM image is called a page filehard disk The virtual memory manager might issue the message: "Your system is low on virtual memory. Windows is increasing the size of your virtual memory paging file." This happens if it is required to swap out a page from RAM to the pagefile while all pages in the pagefile are already taken. With that message, it will allocate more space to the pagefile and use the added space to store the newly- swapped-out page (and subsequent pages).

16 ©2011 SAP AG. All rights reserved.16 Force to run in-memory On the Data Flow set cache type to in-memory, and job execution options check only collect stats for monitoring. This forces the job to run in-memory. 11.7mb total cache memory required here, the system saw 24mb total increase while the job was running. The content of the two cache tables was truncated. If there are stats and the DF is set to default pageable and the run is set without checking either collect or use stats the rows for this job, data flow are deleted. Trace on MS SQL contains DELETE FROM AL_CACHE_STATS WHERE OBJECT_KEY in (1)

17 ©2011 SAP AG. All rights reserved.17 Start over, or take a deep breath. Leave DF as in-memory but collect stats for optimization and monitoring. Trace: 1675618060DATAFLOW12/15/2011 12:09:01 PMSwitching cache type from IN MEMORY to PAGEABLE for data flow to allow collection of statistics for1675618060DATAFLOW12/15/2011 12:09:01 PM optimization. PAGEABLE cache is chosen while collecting statistics to allow the data flow to finish any amount of data being1675618060DATAFLOW12/15/2011 12:09:01 PMcached.1675618060DATAFLOW12/15/2011 12:09:01 PMData flow using PAGEABLE Cache with buffer pool. Run again using collected stats and the job runs in pageable.

18 ©2011 SAP AG. All rights reserved.18 Trace log examples First run on new job, with use collected stats checked and DF set to default cache type == pageable. 1840820152DATAFLOW12/16/2011 2:03:07 PMCache statistics for data flow are not available to be used for optimization and need to be1840820152DATAFLOW12/16/2011 2:03:07 PMcollected before they can be used.1840820152DATAFLOW12/16/2011 2:03:07 PMData flow using PAGEABLE Cache with buffer pool.

19 ©2011 SAP AG. All rights reserved.19 Customer Example – cache, bad join, 119mil rows

20 ©2011 SAP AG. All rights reserved.20 Lots of threads

21 Thank You! Contact information: In Exile.

22 ©2011 SAP AG. All rights reserved.22 No part of this publication may be reproduced or transmitted in any form or for any purpose without the express permission of SAP AG. The information contained herein may be changed without prior notice. Some software products marketed by SAP AG and its distributors contain proprietary software components of other software vendors. Microsoft, Windows, Excel, Outlook, and PowerPoint are registered trademarks of Microsoft Corporation. IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, System z9, z10, z9, iSeries, pSeries, xSeries, zSeries, eServer, z/VM, z/OS, i5/OS, S/390, OS/390, OS/400, AS/400, S/390 Parallel Enterprise Server, PowerVM, Power Architecture, POWER6+, POWER6, POWER5+, POWER5, POWER, OpenPower, PowerPC, BatchPipes, BladeCenter, System Storage, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, Parallel Sysplex, MVS/ESA, AIX, Intelligent Miner, WebSphere, Netfinity, Tivoli and Informix are trademarks or registered trademarks of IBM Corporation. Linux is the registered trademark of Linus Torvalds in the U.S. and other countries. Adobe, the Adobe logo, Acrobat, PostScript, and Reader are either trademarks or registered trademarks of Adobe Systems Incorporated in the United States and/or other countries. Oracle and Java are registered trademarks of Oracle and/or its affiliates. UNIX, X/Open, OSF/1, and Motif are registered trademarks of the Open Group. Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame, and MultiWin are trademarks or registered trademarks of Citrix Systems, Inc. HTML, XML, XHTML and W3C are trademarks or registered trademarks of W3C ®, World Wide Web Consortium, Massachusetts Institute of Technology. © 2011 SAP AG. All rights reserved. SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer, StreamWork, and other SAP products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of SAP AG in Germany and other countries. Business Objects and the Business Objects logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius, and other Business Objects products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Business Objects Software Ltd. Business Objects is an SAP company. Sybase and Adaptive Server, iAnywhere, Sybase 365, SQL Anywhere, and other Sybase products and services mentioned herein as well as their respective logos are trademarks or registered trademarks of Sybase, Inc. Sybase is an SAP company. All other product and service names mentioned are the trademarks of their respective companies. Data contained in this document serves informational purposes only. National product specifications may vary. The information in this document is proprietary to SAP. No part of this document may be reproduced, copied, or transmitted in any form or for any purpose without the express prior written permission of SAP AG.

23 ©2011 SAP AG. All rights reserved.23 © 2011 SAP AG. Alle Rechte vorbehalten. Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, ohne die ausdrückliche schriftliche Genehmigung durch SAP AG nicht gestattet. In dieser Publikation enthaltene Informationen können ohne vorherige Ankündigung geändert werden. Die von SAP AG oder deren Vertriebsfirmen angebotenen Softwareprodukte können Softwarekomponenten auch anderer Softwarehersteller enthalten. Microsoft, Windows, Excel, Outlook, und PowerPoint sind eingetragene Marken der Microsoft Corporation. IBM, DB2, DB2 Universal Database, System i, System i5, System p, System p5, System x, System z, System z10, System z9, z10, z9, iSeries, pSeries, xSeries, zSeries, eServer, z/VM, z/OS, i5/OS, S/390, OS/390, OS/400, AS/400, S/390 Parallel Enterprise Server, PowerVM, Power Architecture, POWER6+, POWER6, POWER5+, POWER5, POWER, OpenPower, PowerPC, BatchPipes, BladeCenter, System Storage, GPFS, HACMP, RETAIN, DB2 Connect, RACF, Redbooks, OS/2, Parallel Sysplex, MVS/ESA, AIX, Intelligent Miner, WebSphere, Netfinity, Tivoli und Informix sind Marken oder eingetragene Marken der IBM Corporation. Linux ist eine eingetragene Marke von Linus Torvalds in den USA und anderen Ländern. Adobe, das Adobe-Logo, Acrobat, PostScript und Reader sind Marken oder eingetragene Marken von Adobe Systems Incorporated in den USA und/oder anderen Ländern. Oracle und Java sind eingetragene Marken von Oracle und/oder ihrer Tochtergesellschaften. UNIX, X/Open, OSF/1 und Motif sind eingetragene Marken der Open Group. Citrix, ICA, Program Neighborhood, MetaFrame, WinFrame, VideoFrame und MultiWin sind Marken oder eingetragene Marken von Citrix Systems, Inc. HTML, XML, XHTML und W3C sind Marken oder eingetragene Marken des W3C ®, World Wide Web Consortium, Massachusetts Institute of Technology. SAP, R/3, SAP NetWeaver, Duet, PartnerEdge, ByDesign, SAP BusinessObjects Explorer, StreamWork und weitere im Text erwähnte SAP-Produkte und ­ Dienstleistungen sowie die entsprechenden Logos sind Marken oder eingetragene Marken der SAP AG in Deutschland und anderen Ländern. Business Objects und das Business-Objects-Logo, BusinessObjects, Crystal Reports, Crystal Decisions, Web Intelligence, Xcelsius und andere im Text erwähnte Business-Objects-Produkte und ­Dienstleistungen sowie die entsprechenden Logos sind Marken oder eingetragene Marken der Business Objects Software Ltd. Business Objects ist ein Unternehmen der SAP AG. Sybase und Adaptive Server, iAnywhere, Sybase 365, SQL Anywhere und weitere im Text erwähnte Sybase-Produkte und -Dienstleistungen sowie die entsprechenden Logos sind Marken oder eingetragene Marken der Sybase Inc. Sybase ist ein Unternehmen der SAP AG. Alle anderen Namen von Produkten und Dienstleistungen sind Marken der jeweiligen Firmen. Die Angaben im Text sind unverbindlich und dienen lediglich zu Informationszwecken. Produkte können länderspezifische Unterschiede aufweisen. Die in dieser Publikation enthaltene Information ist Eigentum der SAP. Weitergabe und Vervielfältigung dieser Publikation oder von Teilen daraus sind, zu welchem Zweck und in welcher Form auch immer, nur mit ausdrücklicher schriftlicher Genehmigung durch SAP AG gestattet.


Download ppt "AL_Tables Jim Kirk August, 2011. ©2011 SAP AG. All rights reserved.2 Agenda – metadata tables 1) AL_Statistics 2) AL_History 3) AL_Lang & AL_Langxmlltext."

Similar presentations


Ads by Google