Presentation is loading. Please wait.

Presentation is loading. Please wait.

IoNode A Framework for Interoperability Shell Project - ETL Activities October 15 th 2003 John Bigerstaff Wynne Rees.

Similar presentations


Presentation on theme: "IoNode A Framework for Interoperability Shell Project - ETL Activities October 15 th 2003 John Bigerstaff Wynne Rees."— Presentation transcript:

1 ioNode A Framework for Interoperability Shell Project - ETL Activities October 15 th 2003 John Bigerstaff Wynne Rees

2 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ETL Solutions Ltd Overview MBO of Prismtech April 2002 Data Transformation Team and Product Based in Blaenau Ffestiniog Wales, UK 10 member team (8 technical) Sales and Marketing Office in North East Standards based solutions POSC, PPDM, W3C standards (XML, XSLT) Product and services company >$4m investment in development of Extract Transform Load (ETL) tools Principal market is large enterprises in Finance, Oil and Gas, Education and Health...

3 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ETL Solutions Strengths ETL product, Transformation Manager (TM) System Integrators of Messaging and ETL solutions Highly qualified and experienced 10 member company with IT and Transformation Skills Requirements Capture and Analysis Data Modelling using EXPRESS, UML, Many years of Java and C++ Programming RDBMS design, development and tuning Extensive knowledge of W3C standards XML, XSD, XSLT Project Management Transformation Manager ETL Tool Meta Data Code Generator Java components for easy deployment in client architectures

4 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioNode A framework for interoperability to meet the interoperability challenges of today Flexible Hardware and Software configuration Developed in Java to run anywhere Software based on Open Source middleware software Low cost transformation component Standards based SOAP [Simple Object Access Protocol] XML [eXtensible Mark up Language] ebXML [electronic business XML] JMS [Java Messaging Service]

5 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioNode Software

6 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ETL Role in ioNode Responsible for the design and implementation of ioNode software components: ioAgent – messaging component ioHub – message routing ioTransform – data transformation ioDB – XML persistent data store Utilise Transformation Manager for design of transforms For SHELL project CSV to IMS-LIP LSC to HESA code lookup and vice versa

7 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Shell Hub and ioAgents

8 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Scalability of ioNode and SHELL Architecture

9 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. SHELL Data Flow and Transforms

10 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Messaging Requirements Once and once only message delivery ebXML uniquely identified messages are acknowledged upon delivery checks for duplication are carried out Send and re-send ebXML messages not successfully acknowledged are re-sent a fixed number of times Persistent storage Provides a record of the message exchanges and allows messages to be re-sent if required

11 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioAgent at College Messages posted using one of the following: Direct to outbound message queue A file drop directory A HTTP Post interface Send Message Respond to messages posted in outbound message queue (ionodepost). Initiate transforms that have been configured e.g. CSV to IMS-LIP and LSC to HESA code conversions Retry messages preset number of times if synchronous reply not received Receive Message Respond to messages in ionodereceive Initiate transforms that have been configured e.g. IMS-LIP to CSV and HESA to LSC to code conversions Place message in inbound message queue Persistence provided by PostgreSQL

12 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioAgent

13 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Synchronous Message Response

14 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioAgent – ioNodePost indexField nametypeDescription 1 indexvarchar(80) Unique identifier for entry (Primary Key) 2 rowcountbigint, Index of number of rows 3 messageIdvarchar(80) Unique message ID, set by ioNode, used for message identification 4 timestampbigint, Timestamp set when table is read by ioNode 5 status_clientvarchar(20), Flag for success of failure of message use by application 6 status_messagevarchar(20), Flag for success or failure of message delivery 7 sourcedatatypevarchar(20), Type of source data in data field eg. CSV 8 sourcedatatypeversionvarchar(20), Version of source data in data field eg. v1 9 targetdatatypevarchar(20), Type required of target data eg. LIP 10 targetdatatypeversionvarchar(20), Version required of target data eg v1 11 senders_authorisationvarchar(80), Authorization field for sender 12 function_typevarchar(80), Action type of post eg. Add 13 targetvarchar(80), Intended recipient of message eg. UoP. 14 domainrefvarchar(80), The identity of the sender eg. Cornwall 15 localidentifiervarchar(80), Security identity field for the sender 16 altdomainrefvarchar(80), Alternative identity for the sender 17 altlocalidentifervarchar(80), Alternative security field for the sender 18 databytea The data to be sent as payload eg. CSV

15 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. IoAgent - ioNodeReceive indexField nametypeDescription 1 rowcountbigint, Index of number of rows 2 messageIdvarchar(80) Unique message ID, set by ioNode, used for message identification 3 timestampbigint, Timestamp set when table is read by ioNode 4 message_typevarchar(80) Type of data eg. CSV 5 versionvarchar(20) Version of data eg v1 6 senders_authorisation varchar(80), Authorization field for sender 7 function_typevarchar(80), Action type of post eg. Add 8 fromTargetvarchar(80), URL of sender of message 9 domainrefvarchar(80), The identity of the sender eg. Cornwall 10 localidentifiervarchar(80), Security identity field for the sender 11 altdomainrefvarchar(80), Alternative identity for the sender 12 altlocalidentifervarchar(80), Alternative security field for the sender 13 databytea The data received

16 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioHub at University Implements IoAgent send and receive of messages Sends synchronous and asynchronous messages replies May act as message router Incoming messages received in Message Receive Queue (ionodereceive) Acts as router to other ioAgents if required In this case makes synchronous reply only when successfully forwarded May transform incoming messages before placing in inbound queue Outgoing messages Responds to asynchronous messages placed in outbound message queue May transform outgoing messages before placing in message delivery queue Acts as message router for asynchronous replies

17 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Hub as Router

18 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Hub as a Router – Seamless Connection

19 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Messaging Implementation ioAgent to ioAgent

20 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioTransform Ability to deploy java transforms as an integral part of messaging Transforms automatically generated using ETL Integrated Development Environment, Transform Manager Open Interfaces allow transforms to and from virtually any format

21 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Transformation Manager Rapid Development Process Transformation Manager supports the whole development process Design Testing Deployment The heart of the system is Transformation Builder, a state-of-the-art interactive transform designer Close integration with Transformation Tester to quickly verify transforms Flexible Deployment via mixed Java and XSLT run time environments

22 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Step By Step Approach Load data models into repository In-built support for RDBMS, XML, and Java Open Interface to access virtually any kind of data Create Transforms Define transforms by drag and drop Add more complex mapping and complete business logic Viewing instance data to provide assistance Test transforms Viewing instance data Rework transforms Deploy Simple Java deployment Update and maintain Transforms generate HTML documentation Transforms are easily updated as models change

23 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Transformation Manager Design Environment

24 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Open Interfaces – Extends TMs Ability to Any Scenario Access to User Specified Models Supports Associated Instance Data Fits Client Architectures Supports Push and Pull modes Pull Model supports Propagation of Selection Example Uses Express\Epicentre Part21 ASCII LIS, LAS, Binary RP66 Edifact Messages

25 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Truly Open Interfaces Access to all forms of models and instance data Allows user models to be loaded with full constraints, relationship cardinality, etc. Easily create user-defined read and write adapters for accessing instance data Transforms called by client architecture with parameter passing and callbacks as required Java interface description published for users to implement Models and instance data can be accessed and viewed as any other data in TM Numerous uses and examples available Edifact, CSV files, EXPRESS Models and Part 21

26 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioTransform Interface ioNode is designed to support a very simple and generic transformation interface public ArrayList transform(String configFile,ArrayList dataIn)throws Exception; Carry out a transform on a list of source data, dataIn, and return a list of target data entities. The configuration of the transform is contained in the configuration file supplied by the caller. The configuration file is read using methods contained by the transform provider so can be of any schema and format

27 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioDB Requirements Storage of student learner records as native XML Generation of a consortium and LRDB unique identifier Update student learner records Roll back log of present number of transactions or messages Generation of Asynchronous messages and responses to business transactions such as new record. The ability to switch and configure the number of transactions stored per consortium and LRDB unique identifier. The ability to switch the facility for extending and building individual master records using xpath updates and a unique identifier. An admin interface for record retrieval and reporting is provided. A standard API for third party interfaces such as the Learner record portal is to be made available and documented.

28 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Learner Record Database

29 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioDB

30 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioDB Interface MethodDescription setDatabaseConnection Connect to the database that hosts the learner records repository createCollection Create a new collection in the learner records setLearnerRecordCollection Set the name of the collection that holds the learner records setArchiveLearnerRecordCollection Set the name of the collection that holds the archived learner records getCollectionList Get a list of collection names in the database removeCollection Remove a collection from the learner records database addRecord Add a new student to the learner record collection updateRecord Update an existing student in the learner record collection getRecord Get a particular learner record in the database queryCollection Query the learner record database using the supplied XPath deleteRecord Delete a student in the learner record collection resetUniqueIDSequence Reset the unique ioDB sequence number setNumberofArchivedRecords Set maximum number of archived entries for each learner record rollbackRecord Reinstate archived learner record as the current learner record

31 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. ioDB Interface Sequence Diagram

32 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Current Activities Example CSV files provided by Phosphorix Create, Update, and Withdraw student records now implemented Transform CSV files to IMS-LIP LSC to HESA Lookup Secure messaging between ioAgents and ioHub Stored in ioDB with unique Global ID Asynchronous replies generated Transform IMS-LIP files to CSV files on round-trip completion HESA to LSC Lookup

33 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Next Steps - Some Issues Web Browser access to all software interfaces Provide easy to use access to data via existing interfaces Resolve a few incompatibilities between CSV and IMS-LIP formats Agree format for asynchronous replies Implement Publish / Subscribe messaging Implement rollback an archive logic in ioDB Implement full query logic in ioDB Testing of routing logic Discuss logic for multi-hop messages Complete full JMS messaging (currently used only as a persistent messaging mechanism) Obtain full set of messages and transform descriptions Create and test full set use TM HTML documentation for review Maintain as regression test suite Determine timescales for rollout

34 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Q&A ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ? ?

35 Commercial in Confidence © 2003 ETL Solutions Ltd. All rights reserved. Further Information John Bigerstaff +44 (0) (0) or


Download ppt "IoNode A Framework for Interoperability Shell Project - ETL Activities October 15 th 2003 John Bigerstaff Wynne Rees."

Similar presentations


Ads by Google