1. Performance Tuning SSIS

2. HR Departments are no fun. Don’t mention the stalking incident with Clay Aiken What happened in Vegas My prom date with a puppet Most unfortunate incident with a turtle My fear of bounce houses How to sexually harass the HR rep What I did to a fish when I was 8 Any talk about my college years The surgery I had last summer The stint I had as a traveling gypsy Why am I still not allowed back Texas How what I did in Vegas truly can’t stay in Vegas

3. About Brian Wasn’t very good with girls Even Kermit the Frog founded a company All 15 still awaiting a publisher. Where he writes about his miniature donkey collection. SQL Server MVP Founder of Pragmatic Works Author of 15 books Blogs at BIDN.com Twitter: @BrianKnight

4. 4 GeoSpatial Data: Semi structured Legacy data: binary files Application database Integration is a seamless, manageable operation Source, prepare, & load data in single, auditable process Scale to handle heavy and complex data requirements SQL Server Integration Services GeoSpatial Components Custom source Standard sources Data-cleansing components Merges Data mining components Warehouse Reports Mobile data Integration Services in Action Cube

5. 5 Advanced Session

6. 6 Today’s Problems with Integration Integration today Increasing data volumes Increasingly diverse sources Requirements reached the Tipping Point Low-impact source extraction Efficient transformation Bulk loading techniques

7. 7 Tuning Decisions Choose the right tool for the job Don’t be afraid to use T-SQL Will parallelism work?

8. 8 Source Optimization Flat files – When available, use Fast Parse OLE DB sources – Change network packet size Use T-SQL whenever possible in the OLE DB Source Joining NULL handling Where clauses

9. SQL Server Network Traffic Connection Settings Packet size defaults to 4096 Increase to 32767 on large data sets Database SSIS Package SWITCHSWITCH LAN

10. 10 Impact of Compression on ETL * Not official Microsoft results.

12. 12 Transform Components The Pipeline presents the buffer to each downstream component

13. 13 SSIS Data Flow Architecture Synchronous vs. Non Synchronous

14. 14 Case Study: Patterns 105 seconds 83 seconds

15. 15 Source Data Extraction Extracting data from the source is expensive Efficient extraction is key to improving ETL performance Involves bulk loading data into staging areas or warehouse Time consuming & resource intensive Triggers (synchronous IO penalty) Timestamp columns (Schema changes) Complex queries (delayed IO penalty) Custom (ISV, mirror, snapshot, …) Incremental data load is key to efficient extraction Need to know what changed at source since a point in time Expensive lookups to determine changed columns Providing information up front about which columns changed will improve efficiency

16. 16 SQL Server 2008: Change Data Capture (CDC) Information about what changed at the source Changes captured from the log asynchronously Enabled per table CDC APIs provide access to change data Change Tables OLTP Data Warehouse

18. 18 Lookup Component Three modes of operation Full Cache: for small lookup datasets No Cache: for volatile lookup datasets Partial Cache: for large lookup datasets Tradeoff memory vs. performance Use Cascaded Lookups Merge Join may be alternative

19. 19 SQL Server 2008: Lookup Transform Hydrate cache files for large data sets Can reuse cache Can load cache during day and use in nightly ETL

21. 21 Data Destinations Use “Fast Load” or SQL Server Destination Table Lock on insert operations Trace flags for improvement Old principles still apply

23. 23 Managing Resources Logging events to watch pipeline internals PipelineExecutionPlan, PipelineExecutionTree, BufferSizeTuning System Monitor to track I/O issues Buffers In Use tracks how many buffers are presently being used Buffers Spooled tracks how many 10 mb buffers have been spooled to disk

25. 25 Location Consider the following configuration… Where should SSIS run? (Licensing issues aside) SQL Server 1 SQL Server 2 SSIS Server

26. 26 WSRM Windows System Resource Manager (WSRM) can throttle CPU and memory Creates a soft throttle Can be scheduled so SSIS gets priority on weekends and nights Only activates policy if resources begin to become constrained (about 70%) WSRM is free with Windows Server 2003 Enterprise Edition and included in Windows Server 2008

28. Building a Work Queue System Create a work queue table. Create a loop to shift over the work queue constantly checking out work Spawn x times with a batch file

29. Demo Results

34. Summary Planning Don’t underestimate the power of the whiteboard! Use the right tool for the right job Leverage the power of the engine Patterns and Practices Understand best practices But don’t be afraid to experiment

35. 35 The End Already? Questions http://www.bidn.com/people/brianknight @BrianKnight bknight@pragmaticworks.com http://www.youtube.com/pragmaticworks