Get your Oracle data into SQL Server faster! Phlippie Smit #83 Johannesburg

Phlippie Smit I have over 15 years’ experience with SQL Server. I spent most of my career in a MIS/BI environment and have used SQL Server 6.5 – 2008 in my endeavours. I am passionate about performance and optimisation, and improving out-dated processes and thinking.

Objectives Explore when to use the Microsoft Connector for Oracle by Attunity to extract data from Oracle sources. Learn about the requirements and installation of the Microsoft Connector for Oracle by Attunity. Understand how to use the Microsoft Connector for Oracle by Attunity effectively to extract data. Learn why the Microsoft Connector for Oracle by Attunity is so much faster than OLEDB, ODBC and .Net Providers to extract data from Oracle sources. Compare the different technologies – Extract speeds.

When to use Data extracts from operational systems running Oracle that carry high volumes of data, e.g. customer orders, billing data, payment data. Data extracts from Oracle data warehouses to further analyse in the Microsoft BI stack (SSIS/SSRS/SSAS) Any other high volume data extracts - Data cleansing, integration projects, test data, reference data, etc.

Requirements Oracle client software version 10.x or 11.x. Use 32-bit client on 32 bit platforms, 64-bit on 64-bit platform. On 64-bit platforms running BIDS, both the 64 and 32-bit Oracle clients should be installed. SQL Server 2008 and above - Enterprise or Developer Edition. Windows 2003,2008 and above.

Installation Install Oracle client(s). Check Oracle naming methods and ensure you have the latest data (e.g. tnsnames.ora) for your environment. Bing “microsoft provider for oracle by attunity download” to get the applicable providers on Microsoft’s website – get version 1.1. Install Microsoft Provider for Oracle (Attunity). Start BIDS, new SSIS project and check for the Oracle Source and Destination Data Flow Items in the Toolbox. If not there, go to “Tools”, “Choose Toolbox Items…” See Additional Resources (3)

Add the toolbox items

How to use the connector effectively Run BIDS as Administrator! Convert the extracted Oracle columns to string type by using “TO_CHAR” and a query, not table. Add type conversion after Oracle Source in Data Flow. Set Oracle Row BatchSize to 10000 (Data Flow Properties) Set DefaultBufferMaxRows to 20000 (Data Flow Properties)

Test/Demo configuration Oracle VirtualBox (Oracle Enterprise Linux) 2 VCPU, 2GB RAM Oracle 11G Dell Precision M6500 Laptop, Intel I7, 8 core, 16GB RAM Windows 7 x64 SQL Server 2008R2 Developer Edition Oracle 11G client (32 & 64 bit)

Test/Demo configuration CREATE TABLE "T_PEOPLE" ( "ID" NUMBER(18,0) NOT NULL ENABLE, "NAME" VARCHAR2(10 BYTE), "SURNAME" VARCHAR2(10 BYTE), "MARRIED" CHAR(1 BYTE), "CARS" CHAR(1 BYTE), "SALARY" NUMBER(18,2), "CHILDREN" CHAR(1 BYTE), "BANKNAME" VARCHAR2(50 BYTE), "CITY" VARCHAR2(50 BYTE), "DATEJOINED" DATE, "ACCOUNT_NUMBER" CHAR(8 BYTE), "EMPLOYER" VARCHAR2(100 BYTE), "DEPT" VARCHAR2(100 BYTE), "SECTOR" VARCHAR2(100 BYTE), "EMAIL_ADDRESS" VARCHAR2(100 BYTE) ) (10 million records) CREATE TABLE [dbo].[t_People]( [id] [bigint] NOT NULL, [name] [nchar](10) NOT NULL, [surname] [nchar](10) NOT NULL, [married] [bit] NOT NULL, [cars] [smallint] NOT NULL, [salary] [money] NOT NULL, [children] [smallint] NOT NULL, [bankname] [varchar](50) NOT NULL, [city] [varchar](50) NOT NULL, [datejoined] [datetime] NULL, [account_number] [char](8) NULL, [employer] [varchar](100) NULL, [dept] [varchar](100) NULL, [sector] [varchar](100) NULL, [email_address] [varchar](100) NULL)

Demo

Why is it faster? It speaks “Oracle” – The Oracle Call Interface (OCI) is used to communicate with the Oracle database instance. Customers are even using this technology to move data between Oracle instances (See Additional Resources for link). Our method removes the complexities of data type conversions from the extraction process while still handling the type conversion in SSIS. Control over row batch sizes. Did I mention it speaks “Oracle”?

Oracle Call Interface (OCI) The most comprehensive, high performance, native 'C' language based interface to the Oracle Database that exposes the full power of the Oracle Database. OCI provides the foundation on which various language specific interfaces such as Oracle JDBC-OCI, ODP.Net, Oracle Precompilers, Oracle ODBC and Oracle C++ Call Interface (OCCI) drivers are built.

Architecture comparison OLEDB, .Net, ODBC Microsoft Connector for Oracle vs

Microsoft Connector Version 1 Microsoft Connector Version 1.1 for Oracle by Attunity supported platforms & versions SQL Server 2008 9.2.0.4 and higher SSIS Oracle OCI DB Oracle Client Windows XP x86, x64 Windows Vista x86, x64 Windows Server 2003 x86, x64, IA64 Windows Server 2008 x86, x64, IA64 Any platform

Comparisons (Extract) Provider Time (seconds) Rows per second No Type Conversion (Attunity) 343 29154.51895 Conversion to String (Attunity) 327 30581.03976 ADO.Net Provider for Oracle 411 24330.90024 OLEDB Provider for Oracle 476 21008.40336

Mixed Type vs. String Conversion

Convert to string and set the row batch size

Additional resources Microsoft Provider for Oracle by Attunity v 1.1 download SSIS ETL Record – Loaded 1TB in 30 Minutes! Using the Microsoft Connector for Oracle by Attunity with SQL Server 2008 Integration Services Customers are moving data from Oracle to Oracle using the Microsoft Provider for Oracle Codeplex SSIS Community Tasks and Components SSIS Product Behaviour (MSDN)

? Q & A That big question mark is editable – use whatever character you want!

SQL Saturday #83 Johannesburg @ SQL Saturday #83 Johannesburg Get your Oracle data into SQL Server faster! Phlippie Smit (http://www.linkedin.com/in/pfsmit)