Presentation is loading. Please wait.

Presentation is loading. Please wait.

DataMigrator 7.7 in Real Time Chris Bevilacqua iWay Solutions Architect.

Similar presentations


Presentation on theme: "DataMigrator 7.7 in Real Time Chris Bevilacqua iWay Solutions Architect."— Presentation transcript:

1 DataMigrator 7.7 in Real Time Chris Bevilacqua iWay Solutions Architect

2 Real Time ETL with DataMigator Agenda  Web Services  Change Data Capture  File Listener  IWAF Adapters

3 Web Services What are Web Services for?

4 What is a Web Service? A software system designed to support interoperable machine- to-machine interaction over a network. It has an interface described in a machine-processable format (specifically Web Services Description Language WSDL). – World Wide Web Consortium

5 .net Publicly Available Commercial Web Services Some sources of Public Web Services Free and commercial

6 WSDLL.COM Web Services – Free and Commercial

7 Xmethods.net List of public web services

8 Seekda.com Find WSDL for PhoneVerify Service from cdyne.com

9 Add adapter for Web Services

10 Add Connection for a Web Service

11 Add Connection for Web Service

12 Create Synonym

13 Open Synonym

14 Sample Data Supply values for PHONENUMBER and LICENSEKEY

15 Sample Data Show all fields returned by Web Service

16 Create Data Flow Add input file

17 User Defined Function Cleans up phone number

18 Add transformation Clean up phone number and assign name

19 Add synonym for Web Service

20 Join input file to Web Service On PHONENUMBER and LICENSEKEY

21 Select Columns from Web Service

22 Test SQL Shows “join” from input file to Web Service

23 Change Data Capture When would you want to capture changes?

24 Why Change Data Capture?  Large source databases, small batch window  Want to only copy the changes  Want near real-time updates  Need some way to detect what has changed…

25 Change Data Capture How to detect changes to database tables  Change the application?  Track timestamp of changes, flag deletes  Write changes to a separate table  Compare source to target?  Effective for small tables  Poll the database for changes periodically?  Detects additions only  Use database triggers?  Developer adds to database…. or automatically generated  Requires staging table

26 Change Data Capture Data Adapter to read from database logs  Read the database log…  Proprietary formats  Every database is different  New Data Adapter for database log tables  No actual staging table required  Synonym describes a view of database log for one table  Provides automatic checkpoint processing

27 Change Data Capture Database configuration  Platforms and databases supported  ORACLE 10g and11g on UNIX and Windows  UDB on UNIX and Windows  DB2 on IBM zSeries and iSeries  MS SQL Server 2008  DBA issues database specific commands to  Configure database to use archiving or enable logging  Add logging or enable capture mode  Create additional synonym for table log records

28 Create Synonym For Table Log Records

29 Create Synonym for Table Log Records

30 Synonym created for Log Table Three CDC columns added

31 Synonym for Log Table Layout same as source table… plus three columns  CDC_OPER – Operation Type  I = Insert  U = Update  D = Delete  CDC_TID – Transaction ID  CDC_TIMES – Timestamp

32 Create Synonym for Log Table CDC Configuration  Polling interval  Timeout  Checkpoint processing

33 CDC Configuration  POLLING – Log polling interval for LUW  TIMEOUT – Timeout interval polling for LUW  START – CHKPT after last checkpoint saved  CHECKPT_SAVE [YES|NO] – save last checkpoint in file  CHECKPT_FILE – Location and name of file  MAX_LUWS – Maximum number of LUWs processed

34 UDB Synonym for Log Table

35 Data Flow Log table as source

36 Column Selection

37 Target Transformations

38 Enabled for IUD Processing IUD control column identified

39 UDB CDC Start with Source and Target the same Run program to update source

40 CDC Flow Log shows rows inserted / updated / deleted

41 Change Data Capture Source and Target now have same rows

42 Change Data Capture Reprocessing  For testing or reprocessing… Start at specified timestamp

43 File Listener Process files as they arrive in a directory… posted or copied or program output

44 Refactoring  7.6 Listener based  Required creating and managing Listener  Only Direct Load flows  Only fixed length files, only character data  Flow never completed… could not view log could not send or run process flow  Could not schedule flows  7.7 Adapter based  All specifications in synonym, simpler to set up  Either SQL or Direct Load flows  Any file type, any data type  Flow completes when file or time limit reached… Can send or run flow or procedure on completion  Can schedule flows as needed

45 File Listener Create Sample Data

46 File Listener Parameters Synonym - dmordfl

47 Data Allocation Parameters  Connection – Name of a connection for an FTP Server  Data Origin  FILE – one time read  LISTENER – process files as they arrive  TAIL – future development  Identify File(s)  Directory – Application Directory to poll for files  Name – file name, wildcards allowed  Extension – file type or extension

48 Listening Parameters and Read Limits  Listening  Polling – interval in seconds  Timeout – time in seconds. If no new files arrive end job  Read Limits  Maxfiles – Maximum number of data files  Maxrecs – Maximum number of records (only for TAIL)

49 Pre and Post Processing  Pre Processing - Pickup  IMMEDIATE – As soon as files arrives  TRIGGER – Wait for trigger file with  EXTENSION  Post Processing - Discard  DELETE  ARCHIVE – Copy files to  DIRECTORY  KEEP – Delete trigger only and keep file

50 File Name Field with Alias of INSTANCE

51 SQL or Direct Load Flow … looks like any other flow

52 File Listener Create directories listen and archive

53 File Listener Submit the Flow – Waits for TIMEOUT interval

54 File Listener Drop the Files

55 File Listener Drop the Files - within timeout interval (60 seconds)

56 File Listener Sample Data

57 Log shows number of rows processed For all files

58 IWAF - iWay Adapter Framework

59 iWay Adapter Framework Application Adapters  Data Adapters for some ERP systems  New support for Application Adapters including  Salesforce.com  MySAP  Seibel  Peoplesoft  So that DataMigrator can use  Services  Events

60 Configure IWAF Adapter

61 Add IWAF Adapter

62 Select IWAF Adapter

63 Add Connection for IWAF Salesforce Service

64 Salesforce Adapter Create Synonym

65

66

67 Create Synonym for Salesforce Service

68 Salesforce Service Sample Data

69 Salesforce Event Synonym

70 Salesforce Data Flow

71 Salesforce.com

72 Salesforce Output table and log


Download ppt "DataMigrator 7.7 in Real Time Chris Bevilacqua iWay Solutions Architect."

Similar presentations


Ads by Google