1. Designing and Implementing an ETL Framework
Mike Litchfield
Assistant Vice President – IT
Architect & Sr. Developer
Income Research & Management

2. Agenda Architecture Considerations Reference Architecture
Why build an ETL framework ?
Architecture Considerations
Reference Architecture
What features to include ?
Walk through sql and ssis components
Walk through web-based components

3. Why Build a Framework ? Reusability \ Efficiency \ Time-To-Market
Definition
Reusability \ Efficiency \ Time-To-Market
Code Standardization \ Ease-Of-Support \ Minimized Troubleshooting
Adherance to best practices
Functional Flexibility and Scalability
Accessability to Data Flow Status
End User Notification

4. IRM Goals Reliability and Accuracy
Implement many routines quickly, with limited developer resources
Reliability and Accuracy
Ability to introduce \ modify \ remove transformation rules
Ability to maintain and apply logical business rules on data
Support for scheduled and user-initiated package execution
Visibility into ETL Status
Error Notification and Ability to Debug Quickly
End User Confirmation
Support for logical dependencies between Sql Agent steps and SSIS
packages
Package instance concurrency handling
Historical persistence of data, errors, overrides, thresholds

6. Architecture Considerations
Organization of database servers and databases
Sql Server 32-bit vs 64-bit
Database access methodology
How ssis packages will be invoked
Checkpoint restart \ database transactions
Compartmentalizing ssis functionality
Linked Server
Excel vs tables

7. What features to Include ?
Ssis package templates – parent and child
Multiple ways to invoke the package
Package instance concurrency handling
Run date \ Override date handling
Tracking package progress
Error handling and notification
Variables definition
Standardized flow
Concurrency Check
Registration
Extract
Transform \ re-process corrections
Rule execution
Final destination posting
Historical archiving
Notification

9. Sql Components Framework Tables Etl_pkg Etl_pkg_auth Etl_pkg_req
Etl_control
ETL_batch_override_dates
ETL_data_supplier_vendor
Ssis_task_failures
Ssis_errors
Transact_sql_errors
sysdtslog90
Framework procs
Flow-Specific Tables
Work
Staging
Transformation_Error
Etl_control
Override
Thresholds
Logical_Errors
Staging_Hist
Override_Hist
Threshold_Hist
Corresponding procs

10. SSIS Parent and Child Templates Package naming convention
Variable namespaces
Data access methodology - procs
Handling Task failures
Handling detail error messages
Handling proc failures
Logging execution
Pre-compiling script
Unique system::id
Compartmentalizing into many small packages vs one big
Common flow
Control the flow, let Sql operate on the data

11. Web Components - Operations

12. Web Components – Threshold Maintenance

13. ETL Success Notification

14. ETL Failure Notification

15. Web Components - Reporting

16. Thank You ! Mike Litchfield Assistant Vice President – IT
Architect & Sr. Developer
Income Research & Management