2. SSIS Exploring Scalability, Performance and Deployment Vinod Kumar & Srinivas Sampath MVP – SQL Server

3. Presentation Scope A high level view Design considerations How to measure performance Performance implications of architecture Manageability aspects of SSIS Deployment tips Out of scope Prescriptive guidance for specific situations

4. Agenda Buffers and Memory OVAL Concept Detailed Component Specific Notes Manageability Features Deployment Considerations

5. Introduction

6. SSIS Life Cycle tools Design the SSIS Package Business Intelligence Studio (visual Studio) Migration wizard for pre SQL 2005 packages Version Control Integration (VSS) Deployment/Execution Deployment Utility to copy packages Command Line execution (dtexec.exe and dtexecui.exe) Flexible Configuration Options Supportability Rich per package Logging SQL Management Studio for monitoring running packages and organizing stored packages Checkpoint - Restartability

7. SSIS Tools SSIS packages packages BI Studio SSIS Service Mgt Studio Import Export WizardDeploymentInstaller File set Dtexec.exeDtexecui.exe Dtutil.exe execution View running and import\export deploy

8. Deep dive into Performance

9. Buffers and Memory Buffers based on design time metadata The width of a row determines the size of the buffer Smaller rows = more rows in memory = greater efficiency Memory copies are expensive! A buffer might have placeholder columns filled by downstream components Pointer magic where possible

10. Component Types Logically works at a row level Buffer Reused Data Convert, Derived Column Row based (synchronousoutputs) Partially Blocking (asynchronousoutputs) Blocking(asynchronousoutputs) May logically work at a row level Data copied to new buffers Merge, Merge Join, Union All Needs all input buffers before producing any output rows Data copied to new buffers Aggregate, Sort

11. CPU Utilization Execution Tree Starts from a source or an async output Ends at a destination or an input that has no sync outputs Each Execution Tree can get a worker thread MaxEngineThreads to control parallelism

12. Performance Strategy Use OVAL to identify the factors affecting data integration performance… O perations Which app is best suited to these operations on this volume of data? For example, use SQL Server or SSIS for sorting data? V olume A pplication L ocation How much data must be processed? What logic should be applied to the data? Where should the app run? For example, on a shared server, or on a standalone machine?

13. An OVAL Example— Loading a Text File Simple scenario… Interesting performance considerations! Text file on Server 1 SQL Server on Server 2

14. Understand all operations performed Operations Beware of hidden operations Data conversion in either step 3 or 4 1. Open a transaction on SQL Server 2. Read data from the text file 3. Load data into the SSIS data flow 4. Load the data into SQL Server 5. Commit the transaction

15. File Source Unnecessary data type conversions ‘FastParse’ in Flat File Source Unnecessary operations: E.g., converting from text to datetime, then from datetime to date Reduce database operations Database logging Commit size Fast Load Table lock Operations - Sharpen

16. Volume Reduce where possible Don’t push unneeded columns Conditional split for filtering rows Do not parse or convert columns unnecessarily In a fixed-width format you can combine adjacent unneeded columns into one Leave unneeded columns as strings

17. Volume - Sharpen Use appropriate data types An integer in the range 1-999 takes 2 bytes as an integer, 3 bytes as a string, but 4 bytes as a real Suggest Types in the flat file connection manager UI Use parallelism If loading multiple files, can they be loaded in parallel?

18. Application Is SSIS right for this? Overhead of starting up an SSIS package may offset any performance gain over BCP for small data sets. Is BCP good enough? Is the greater manageability and control of SSIS needed? Bulk Import Task vs. Data Flow

19. Location Consider the following configuration … Text file on Server 1 SQL Server on Server 2 Where should SSIS run? (Licensing issues aside)

20. Location Considerations SSIS on Server 1 Competes with apps for resources Will data conversion on Server 1 reduce or increase the volume of data transferred across the network? Can not use the fast SSIS SQL Server Destination SSIS on Server 2 Competes with SQL Server for resources Will pulling text over conversion be expensive? Also consider transferring the file unparsed to Server 2 and read it locally from there Can use the fast SSIS SQL Server Destination

21. Measuring Performance OVAL does not provide prescriptive guidance Too many variables Improve performance by applying OVAL and measuring SSIS Logging Performance counters SQL Server Profiler For extract queries, lookups and loading

22. Parallelism Focus on critical path Utilize available resources Memory Constrained Reader and CPU Constrained Let it rip! Optimize the slowest

23. Moving Ahead

24. Manageability Features Logging and Log Providers Checkpoint Restartability Precedence Constraints Configurations SSIS Service

25. Logging and Log Providers Log entries are a blend of status and result messages Can select what ‘details’ per control flow object within each package (e.g. OnError, OnWarning, OnPreExecute) Can select what fields (e.g.computer, operator, ExecutionID…) Can define multiple log providers (SQL, text file, Windows Event..) per package

26. Checkpointing Checkpoint File Created Write Checkpoint Checkpoint File deleted Package LoadsPackage Completes Data Flow Task Send Mail Task

27. Configurations ‘Feed’ changes into a package and alter execution without editing the package directly (e.g. file name to load) The ‘feed’ can be sourced from a SQL table, XML file, Registry key, OS environment var, a Parent package. You can apply 1-many configuration sets per package and from a mix of sources

28. Configuration Scenario Dev DB Multiple Configurations Dev Test Production Test DBProd DB Machines where packages are being designed /tested /executed Configuration updates package on load with DB locations (and mail server, file share locations….) Package Handoff

29. Precedence constraints Directs Flow from object to object… Basically, ‘when do I move on’ Success, Failure, Completion or one of those plus an expression (condition) Dataflow Task SendMail Task Success Completion Failure Success & expression

30. Manageability Demo

31. Deployment Flow Tools to organize and ‘copy’ packages and supporting files Design Package Add Configurations Add Miscellaneous files Set Project Deployment properties Build Choose Destination (SQL File System) Modify protection level Choose location of supporting files Change configurations Execute Installation Wizard Bi Studio Copy/Move Deployment folder\files User Create desired agent jobs SQL Agent Copy/Move Deployment folder\filesUser

32. SQL Management Studio Utilizes the SSIS service Allows Monitoring of currently Executing packages Maintain stored package structure Ad hoc Package execution

33. Deployment Demo

34. Some more Tips LookupAggregateSortSwapping

35. Performance of Lookups The reference set Restrict to only those columns you actually use Restrict rows with WHERE if possible The lookup cache Caching can improve performance Full cache When the reference set will fit comfortably in memory Partial Build a cache as the input records are matched Useful for duplicate keys in the input, such as SKUs None Reference set doesn’t fit in memory and partial cache has no advantage

36. Performance of Aggregate Majority of work happens in ProcessInput call. This is on the thread in the previous execution tree! Memory requirements depend on how ‘deep’ the aggregations are Can reuse buckets if one agg can be derived from another Use when memory is limited, single threaded operation

37. Performance of Sort ProcessInput hangs on to the incoming data PrimeOutput does the sort and is the expensive part Sort needs all data to be in memory Sort can have unpredictable CPU requirements Merging is single threaded Stock Sort component will be good enough for most users Third party (“fastest sort in the world”) available if you really need it

38. Swapping buffers When physical memory is not available Each buffer gets written out to one file Multiple paths can be specified for swapping buffers BufferTempStoragePath property on the Pipeline Do everything in your power to avoid swapping Else, performance is really unpredictable Options: 64 bits, out of process execution, serializing operations

39. SSIS: Summary Fast ! Data flows process large volumes of data efficiently - even through complex operations Exceptional price / performance on multi-core Feature Rich Many pre-built adapters and transformations reduce hand coding Extensible object model enables specialized custom or scripted components Highly productive visual environment speeds development and debugging Integral part of a complete BI stack (IS-AS-RS) Beyond ETL Enables integration of XML, RSS and Web Services data Data cleansing features enable “difficult” data to be handled during loading Data and Text mining allow “smart” handling of data for imputation of incomplete data, conditional processing of potential problems, or smart escalation of issues such as fraud detection

40. Your Feedback is Important! Please Fill Out the feedback form

41. Questions !!!

42. Links & Resources Vinod Kumar, MVP-SQL Server, www.ExtremeExperts.com Intel Technology India Pvt. Ltd. vinod@ExtremeExperts.com SQL Server Integration Services public site http://msdn.microsoft.com/SQL/sqlwar ehouse/SSIS/default.aspx http://msdn.microsoft.com/SQL/sqlwar ehouse/SSIS/default.aspx SQL Server Business Intelligence public site http://www.microsoft.com/sql/evaluati on/bi/default.asp http://www.microsoft.com/sql/evaluati on/bi/default.asp SSIS MVPs community site http://www.sqlis.com Newsgroupsmicrosoft.private.sqlserver2005.dts Srinivas Sampath, MVP-SQL Server www32.brinkster.com/srisamp SCT Software Solutions srisamp@

43. © 2005 Microsoft Corporation. All rights reserved. This presentation is for informational purposes only. Microsoft makes no warranties, express or implied, in this summary.