ETL Overview February 24, 2004

DS User Group - ETL - February ETL Overview “ETL is the heart and soul of business intelligence (BI).” -- TDWI ETL (Extract, Transform, Load) is the process of extracting data from the source system (i.e., Banner), transforming it (i.e., applying business rules) and loading it to the target system (i.e., Data Warehouse).

DS User Group - ETL - February Data Warehouse Environment Data Marts –Designed for particular use –Subset of data –Combines data in simpler structure –Apply business rules –Fast and easy to use –Slice and dice counts Banner Reporting Copy Other Systems Legacy Data Data Mart Business Objects Universe(s) Enterprise Data Warehouse Extract Transform and Load EDW –Core repository of data –Multiple subject areas –Very flexible but complex structure –Track change history –Day old data UI2 Standard Reports Custom (User) Reports EDDIE (InfoView) Portal Users Custom (User) Reports

DS User Group - ETL - February Questions Which tools are used for ETL processing? Informatica is used to develop and execute the ETL maps. The maps are grouped together by subject area (e.g., HR) and scheduled by Appworx. When do the ETL maps run in production? The ETL process begins at 12:05 AM every day, 7 days a week and generally finishes before 8:00 AM. What happens if an ETL map fails at night? Decision Support has on-call support for the ETL processes to respond to production failures. Trivia question: How large is the Data Warehouse? Decision Support currently maintains about 550 production tables and approximately 10,000 columns. Over 1,000 ETL maps are used to populate these tables every night! Once Records and Registration is in place, the table count will exceed 650 tables and over 12,000 columns!

DS User Group - ETL - February Questions Why not develop customized program for ETL processing? Tools like Informatica permit developers to quickly and accurately define ETL maps via the graphical user interface. The tools are also efficient in executing the maps. How does data get from Banner to the EDW? Relevant data is first copied from the Banner production source database to Decision Support’s staging database. Next, the data is transformed to target tables in the staging database. Finally, the data is loaded to the production Data Warehouse for use by Business Objects and other users. What is the difference between the EDW and Data Marts? The EDW contains detailed target data. Data Marts are customized for specific reporting needs and may include aggregated data to streamline the business intelligence process.

DS User Group - ETL - February For More Information Check these helpful links for more information about ETL: Decision Support’s web site ( The Data Warehousing Institute ( DM Review ( Informatica (