Presentation is loading. Please wait.

Presentation is loading. Please wait.

A Java File Transfer Framework (and More) Using Quartz Jared Lynem Amway Corporation

Similar presentations


Presentation on theme: "A Java File Transfer Framework (and More) Using Quartz Jared Lynem Amway Corporation"— Presentation transcript:

1 A Java File Transfer Framework (and More) Using Quartz Jared Lynem Amway Corporation jlynem@gmail.com

2 Hello  I’m Jared  We’ll talk about: File transfers: requirements, reasons, concepts A framework for file transfers A way to schedule them Other quick victories  Links to resources at the end of this document  Feel free to contact me jlynem@gmail.com

3 Requirements  The first business data “integrations” Moving paper files from one office to another Ditto machines! Crazy. Offices like on Mad Men  Fast forward to early digital integrations Moving data files from one server to another  Bread and butter, reliable, easy

4 Requirements – What?  What – select and move files based on location, file name  When – based on a schedule or as a process step  Where – multiple targets, multiple sources, archiving, PCI/PII considerations  How – Protocols: FTP, SFTP, FTPS, CIFS, VPN; Encryption: PGP

5 Requirements – Why?  Why indeed.  Not our problem, just do as the customer requests.  Seriously though, fight for the best design.

6 Reasons  Why should we use file transfers? To communicate securely with external entities To interact with legacy applications Because they’re simple and easy to support When you don’t need data transformation When someone need to review the data  Beware of PCI/PII requirements!

7 Reasons  Why develop a framework? Oracle SOA Suite “FTP Adapter” doesn’t do what you might think it does. Oracle Data Integrator FTP/SFTP setup is clunky, not easily extensible Often times, many very similar integrations are required

8 Concepts - Java  Object oriented programming  Native language for Weblogic Application Server  Relatively easy, JDeveloper is a great IDE  Much, much faster than SOA tools  Widely used, plenty of knowledgeable developers  Oracle: “Resistance is futile”

9 Concepts – Transfer Protocols  FTP – Old ‘n reliable Credential based login, or anonymous Not encrypted Easy to implement, widely supported  SFTP – “FTP” over SSH Encrypted transfer using Secure Shell Allows for additional authentication measures

10 Concepts – Transfer Protocols  FTPS – FTP with encryption over SSL Support for multiple authentication methods Support for public key certificates (SSL)  CIFS (aka SMB) – Used by Windows Allows for Active Directory or Kerberos authentication Handy when FTP isn’t set up on Windows servers

11 Concepts – Security  Transport Level Security (See above)  Data Level Security – PGP (Pretty Good Privacy) Offers encryption in a wide variety of algorithms and strengths Offers data compression, integrity checking, and message authentication Unbreakable!

12 The Framework  Most file transfers follow a similar design: 1.Trigger condition (sometimes) 2.Source(s) for files 3.Encryption/decryption/renaming (sometimes) 4.Target(s) for files 5.Archiving (sometimes)

13 The Framework - Goal  To make these common tasks configurable via XML elements  To allow custom tasks to work seamlessly with common tasks to minimize development time

14 The Framework – JAXB  The XML part is easy: Create an XML schema with elements for each step’s configuration Use JAXB (built-in to JDeveloper) to create matching Java objects Use these objects as “properties” for your step implementations Create a mapping between XML element types and Java classes to execute

15 The Framework – Trigger Condition  A trigger condition decides whether the transfer should occur  Typically a “trigger file” is used, placed in a directory near where the actual file will be  Useful for preventing accidental transfer during file write  Configurations: FTP location, authentication, file name

16 The Framework – Source config  FTP location, authentication, remote directory, file names or file patterns  Java supports regular expressions, so use them  Not all FTP servers support glob expressions

17 The Framework – local work  Encryption config Public key Whether to delete unencrypted file afterwards (yes)  Decryption config Private key Passphrase

18 The Framework – Target Config  Multiple target elements, each with: FTP location, authentication Remote directory

19 The Framwork - Archiving  Considerations Local vs. remote Append timestamps, other clerical tasks Archiving sensitive data

20 The Framework – Custom task  Create your own Java class to do some complicated work (i.e., building a file from a database query)  XML configuration: Class name Name/Value properties used by the class

21 Putting it All Together – Task Interface  A “Task” needs: A definition: settings, configurations A context: what happened before? A result: something that can be interpreted and acted upon

22 Task Interface – Task Definition  Say, one of the XML elements you just defined  The Task knows what to do with it

23 Task Interface - Context  A way to keep track of what happened in previous steps  Most obvious example is, what files are we working with?  Doesn’t need to be too complex

24 Task Interface – Result  Three values Success/Failure Whether to continue Message for grateful support person

25 One More Interface - Plan  Just a way to build a list of tasks from a given XML configuration  Execute them in order, checking the result after each task  Bonus points: have “Plan” implement “Task”

26 phew  Hard part is over  Lunch soon! I feel ya, bro

27 Quartz  Well known (among Java nerds) library for scheduling jobs  Free as in speech, beer  Well documented, tons of features  Runs in a clustered environment  Free web-app UIs exist  You can pay for support if you want

28 Quartz – Runs stuff  Use Quartz if: You want to run something every five minutes You want to run something every day You want to run something on demand You want to run something every 15 minutes from noon to 2pm on the fourth day of each month and every second Tuesday

29 An Alternative to Quartz  Weblogic Job Scheduler Built-in to Weblogic =) Not portable =( Not widely used =( More complicated setup =( Documented by Oracle =(

30 Quartz – Concepts  Job – interface for Java classes, with only and “execute” method. Quartz can run any class that implements Job.  Trigger – Tells the schedule when to run  Groups – Group Jobs for better control  Listener – Your own optional class that gets notified when things run (for logging purposes)  Job Data Map – Configurations or data passed in when the Job is run.

31 Putting the FTP Framework in Quartz  Create a Java class that implements Job (so Quartz can run it)  Schedule the Job, put your XML file in the Job Data Map so it is accessible by your new class.  Have the class set up a Plan of Tasks based on the XML file, and execute it.

32 Bonus - Quartz/SOA Interaction  Quartz -> SOA Use the javax.xml.soap libraries to create a simple web service Job Now you can schedule your SOA composites!  SOA -> Quartz Create a simple class in your scheduler project that triggers a job on-demand in Quartz based on an input job name. In JDeveloper, right-click and select “Create Web Service” Now you can kick-off scheduler jobs from SOA!

33 You’ve been a great audience  Takeaways: Java is your best bet for file transfers if you’re using Oracle products for integration Abstract Source/Target/Encryption tasks since they’re all pretty much the same XML is a good format for configurations since it’s extensible and easy to import to Java

34 Thanks OWMUG  Takeaways: Quartz is a powerful, free scheduling tool used by many large organizations Quartz can interact with SOA via web services, too (or anything Java can interact with- ODI, AS400, SQL, etc.)

35 select * from QUESTIONS

36 Libraries and Links (all free!)  Quartz - http://quartz-scheduler.org/http://quartz-scheduler.org/  Apache Java FTP/FTPS client library - http://commons.apache.org/net/ http://commons.apache.org/net/  JSCH Java SFTP client library – http://www.jcraft.com/jsch/ http://www.jcraft.com/jsch/  CIFS/SMB client library - http://www.samba.org/http://www.samba.org/  Bouncy Castle Java PGP library - http://www.bouncycastle.org/ http://www.bouncycastle.org/  JWatch Quartz Manager UI - http://code.google.com/p/jwatch/ http://code.google.com/p/jwatch/


Download ppt "A Java File Transfer Framework (and More) Using Quartz Jared Lynem Amway Corporation"

Similar presentations


Ads by Google