1. TotalETL:infoServer Chris Fournier Nathan Clark Scott Longley Cyril Shilnikov MQP Project 2005 Sponsored by TotalETL inc.

2. TotalETL Small ETL Company ETL (Extract Transform Load) –Used in large companies –Multimillion dollar business Existing Product is infoSight-- desktop solution

3. infoSight

4. infoSight Current Features GUI Project creation Library of Transformers Works with multiple input types Single machine Single user One project at a time

5. MQP Project goals Prototype the client-server version of infoSight –Distributed –Multi-user –Database-centric –Extensible –Alpha-level code –Focus on back-end design

6. Project Methodology Met with TotalETL team on-site Design requirements Refine and discuss requirements as needed Build core modules, demo end first term Build additional modules, final demo.

7. General design overview Thin Clients Thick Clients Repository Distributed Server System

8. Actual design overview Security Manager Session Manager Repository Manager Event & Log Manager Project Manager Scheduling Manager Job Manager DB Client Version Manager

9. Repository Manager System core Store all information about –System operation –Security –Projects XML Parser to store Projects JDBC to connect to DB’s

10. Repository Table Design

11. Project Manager & Version Control Storage and Retrieval of Projects –In-memory Object -> XML File -> Repository Version Control –Per user locking –Version tracking

12. Job Manager Combine Projects into Jobs Set interdependencies Running Jobs

13. Schedule Manager Schedule Jobs –On request –Per schedule Multiple scheduling strategies

14. Session Manager Establish and maintain client connections RMI –Simple, robust, built-in to Java Front end for all functions in server Security checking –Authentication of users –Authorization of commands

15. Security Manager Determine user’s privileges Control access to Projects, Jobs, etc. Custom Security Model –Role-based ACLs –Read, Write, Execute (Projects and Jobs) –Read, Create, Modify (System Configuration)

16. Event Manager & Logger Useful for future expansion Complex Hierarchy of Events All Events Logged –Log4J format

17. Event Hierarchy InfoserverEventUserEvent UserLoginFailedEvent UserLoginEvent UserLogoutEvent ProjectEvent (Other level-2 events) (Other level-3 events) Listeners

18. Saving and Loading Projects Security Manager Session Manager Repository Manager Event & Log Manager Project Manager DB Client Version Manager

19. Creating Jobs from Projects Security Manager Session Manager Repository Manager Event & Log Manager Project ManagerJob Manager DB Client

20. Scheduling Jobs to Run Security Manager Session Manager Repository Manager Event & Log Manager Scheduling Manager Job Manager DB Client

21. Project Summary Relational Database storage –Projects –Operational Information Job Scheduling Tailored Security Model Version control Logging

22. Future work Distributed servers Clients, thick and thin Support for more databases More advanced scheduling algorithms

23. Thanks Professor E. A. Rundensteiner Arun Shastry Greg Goldberg Rest of the TotalETL Team

24. Questions?