1. Introduction to WebSphere MQ File Transfer Edition

2. WebSphere MQ: Connectivity to, from and within an Enterprise
A Universal Message Bus for access to data wherever it exists to support your business. MQ moves:
Data
Messages
Files
Events
Services
Provides a comprehensive range of Messaging capabilities to support your Business requirements for data integration
Reliability and availability QoS
Messaging integration patterns
Managed File Transfer
SOA foundation
Provides appropriate data access and data privacy controls to help meet audit and regulatory requirements
MQ 7.5 is GAed on 6/15/2012, single, integrated offering for all messaging functions
Regional Office
Sensor
e.g. RFID
Branch
Outlet
Retail Store
MQ has been around for about 20 years now. It solves the fundamental problem of moving information to or from and within an enterprise between heterogeneous applications and platforms. Not only that, MQ has evolved to adopt new standards and requirements for connectivity over the years.
It can move all kinds of data, messages, files, events, and services from applications to applications, file systems, databases and exchange data with mobile devices and sensors too.
It provides various QoS based on reliability, network status, performance, and messaging patterns such as fire and forget, request and reply, store and forward, publish and subscribe, managed file transfer and SOA.
In addition, it has built in security and access control to ensure end to end data privacy no matter the data is in transit or at rest.
MQ 7.5 is now available and has offered a single package to enable all these capabilities in MQ, instead of separate installations of different products from different vendors.
Today we’ll not be able to cover all capabilities of MQ 7.5, if you are interested in MQ for mobile devices or sensors, and IBM’s mobile solution, please tune in for other upcoming webinars.
Pervasive
Device
Refinery
Petrol Forecourt
Mobile
Phone 2
CSS: F S 2

3. What problems does Managed File Transfer Solution solve
Reliability
Do you experience lost file, partially written file and duplicate due to inconsistent network or system being not available?
What is the business impact when files aren’t properly transferred?
How much time and energy do you spend trying to figure out what happened when a file went astray?
How much time and effort does it take to recover/restore things after a failure?
Performance
Do you need to transfer large file or many files at the same time? Are you able to meet the SLA?
Every time when a file transfer fails, does it require to be restarted from the beginning?
Security
Do you have any security concerns, such as plain file in transit which contains sensitive data?
How about access control, such as who can transfer what file(s) from source to destination?
Visibility and Traceability
Do you have any visibility of the file transfer status and progress, whether the file is partially sent, completes successfully or not, in a timely fashion?
Do you want to track the entire journey of a file across the enterprise?
Are you able to show in an audit where your last 10 transferred files and documents came from and went?
Automated and Integrated
Do you need various ways to invoke file transfer such as ad-hoc, scheduled, event triggered, or through programming API?
Do you need to integrate file transfer with the rest of your IT infrastructure and business processes?

4. FTP – the lowest common denominator for file transfer
FTP is 30 years old, and showing its age. It is no meets business needs
Limited Reliability
Limited Security
Partially written files (often only detected after substantial process and time is wasted)
Cost of re-transmitting big files (no check-point restart)
Lack of Character Set conversion (file data can be unusable after transfer
Username and password are often in plain text
Lack of Privacy, authentication and encryption often not available
Little visibility or traceability
FTP, although is commonly used for file transfer and free, doesn’t meet these business needs for enterprise file transfer solution, especially when facing network inconsistency, large file or high volume of data, security, and the need to manage, integrate and automate file transfer process.
Limited Flexibility
No notifications when a transfer fails
No central management and progress monitoring
Cannot track the entire journey of files
All resources have to be available at the same time
ftp Scripts are typically scattered across machines, and require platform specific skills
Often only one ftp transfer can run at a time

5. WebSphere MQ File Transfer Edition
Adds managed file transfer capabilities to WebSphere MQ ……
WebSphere MQ File Transfer Edition A B C X Y Z
Simple
Centralized file transfer configuration, management and monitoring via graphic tool
Automated
Enables scheduled and event-driven file transfers
Reliable
Solid reliability of WebSphere MQ + automated retry, error recovery and checkpoint restart
Any file size
Multi threading, efficiently handles anything from bytes to terabytes, and data bombardment
Secure
Provides role based access control. Protects file data using SSL + MQ AMS.
Auditable
Full logging and auditing of file transfers + archive
Integrated
Programming API, Conversion between file and messages
Cost Effective
Exploits investment in WebSphere MQ. Wide range of support (inc. z/OS and IBM i)
MQ FTE on the other hand, leveraging MQ as the under pining transport, provides you with a simple to use, no coding, reliable, secure and complete file transfer solution.
MQ FTE builds on MQ’s centralized GUI management tool for file transfer configurations and progress monitoring.
It can handle any file size by streaming, supports automatic retry and checkpoint restart to the last failure point, and easily scaled to meet high data volume requirements.
For security, It uses SSL for file transfer, allows end to end data encryption and digital signature by MQ Advanced Message Security, and you can set up authorities for file transfer operations such as who can transfer what files from which source to which destination.
File transfers are auditable too, can easily be integrated with MQ network and other file transfer networks such as FTP, SFTP, HTTP, etc.
Last, it saves you money.

6. WebSphere MQ - A consolidated transport for both files and messages
Maximizes the ROI
Traditional approaches to file transfer result in parallel infrastructures
One for files – typically built on FTP
One for application messaging – based on WebSphere MQ, or similar
High degree of duplication in creating and maintaining the two infrastructures
MQ File Transfer Edition reuses the MQ network for managed file transfer and yields:
Operational savings and simplification
Reduced administration effort
Reduced skills requirements and maintenance
Application
Messaging
File Transfers
Consolidated Transport
for Messages & Files

7. How does WebSphere MQ FTE work?
MQ Server
1. Send file data
Agent QM
2. Receive file data
MQ FTE Agent
MQ FTE Agent
MQ network
MQ network

8. How does WebSphere MQ FTE work?

9. Components of a typical WMQ File Transfer Edition Network
WebSphere MQ
Provides the transport for all communication between agents, commands and the coordination hub
Agents
The endpoints for file transfers
Long running, multi-threaded MQ applications,
Associated with one particular queue manager (V6 or V7)
Efficient transfer protocol for files
Commands
Send instructions to agents and display information about agent configuration
Many implementations of commands: MQ Explorer, Command Line, Scripting, API
Coordination Hub
Stores configuration information about the WMQ FTE network
Collects events for the file transfers
Keeps a historical account of transfers that have taken place

10. Ways to configure and control managed file transfers
Graphical Eclipse-based
MQ Explorer
Command Line Interface
Job Control Language (JCL)
XML Scripts using Apache Ant

11. Centralized configuration using MQ Explorer
Eclipse GUI integrated into WebSphere MQ Explorer
Shows progress of current transfers and outcome of recent transfers
3rd party and bespoke applications can also subscribe to these events

12. Managed file transfers using MQ Explorer
Graphical user interface that allows:
Operators to create ad-hoc file transfers to satisfy a particular business need
File transfers can:
Specify code page conversion
Group together multiple files into a single transfer
Recursively move directory structures

13. Scheduled transfers
Transfers can be scheduled to take place at a specific time
Scheduled transfers can be used to initiate a managed file transfer during a processing window
Scheduled transfers can repeat:
Every day, week, year etc.
For a specified number of repetitions
Until an end date/time

14. Directory Monitoring
Monitor file system for presence of “trigger files” and start a file transfer operation
Directory monitoring can be used to integrate with an application which produce files as its output – without needing changes to the existing application
“Trigger files” can be:
Specified using wildcards
Found by recursively searching directory trees
Attributes of file transfer can be inferred from trigger file, for example:
Transfer all files from same directory as trigger file
Select destination agent name from parent directory of trigger file
Name destination file with timestamp of trigger file
Etc.
Supports extension via user exit routines

15. Via the Command Line...
Equivalent function to the GUI is also scriptable from the command line
Administrative commands:
Define and delete agents
Configure agents
List and show details about agents
Start and stop agents
Create and delete scheduled transfers
Create and delete directory monitors
Operational commands:
Start transfers
Cancel transfers
List in-progress transfers

16. … and also Job Control Language (JCL)

17. Scripting
Script together complex sequences of transfer operations which execute conditionally
E.g. If step1 successful do step2 else administrator
Use scripting to call out to other programs to perform pre/post processing of file data
E.g. Run a program prior to the transfer to generate the file, or after the transfer to process the file
The output from any programs run is captured in the transfer audit information
Based on the Apache Ant language which is:
Open… (so you can get a book on it…)
..and extensible (so you can use Ant tasks developed elsewhere)

18. Example of a file transfer script
FTE can also start another application to process the file
FTE transports file to destination
Existing Application
WMQ FTE Agent
WMQ FTE Agent
*tap*
Existing Application
Application writes file to file system
Agent monitors file system, spots arrival of file and based on rules, transfers the file
At destination MQ FTE writes file to file system

19. Other Examples
Transfer one or multiple files from source to destination
Transfer file(s) from source to multiple destinations
Unzip, Combine and Zip files to send from source to destination
Concatenate files from multiple sources into a single file
Split a single file into multiple files and send to different destinations
Validate the file data before file transfer
Send notification if file transfer fails

20. Integration with event driven applications
MQ FTE Helps:
Deliver files as message payloads and vice versa
Monitor queues and transfer message payloads to files
Bridge between batch based applications and event driven applications
WebSphere MQ & File Transfer Edition 20 20

21. MQ queue monitoring
FTE agent can monitor for files arriving…
The WMQ FTE agent can monitor queues for the arrival of messages, then perform an action, such as transferring the payload from the message to a file
Conditions that can be monitored for:
Queue not empty
Complete group of messages
Existing Application
WMQ FTE Agent
It can also monitor for
messages arriving on a queue…
Existing Application
WMQ FTE Agent

22. Example Usage of converting data between files and messages
One file becomes one message
One file to one message
WMQ FTE
One file to a group of messages
The file can be split based on:
Size
Binary delimiter
Regular expression
WMQ FTE
One message becomes one file
One message to one file
WMQ FTE
Optionally, a delimiter can be inserted between each message used to compose the file
A group of messages (or all messages on the queue) to one file
WMQ FTE

23. API for Invoking File Transfer
1. Send request
File Transfers can be invoked
by ESB as a service
ESB
2. Start file transfer
3. Send file data
MQ Server
4. Receive file data
MQ FTE Agent
MQ FTE Agent
MQ network
MQ network
Database
Logger
Central Management Tool

24. File Broker – integration with ESB
File to Message(s) or
Message(s) to File
conversion
2. Process file data
if required such as
data transformation
MQ FTE Agent
1. Send file data
3. Receive file data
MQ FTE Agent
MQ Server
MQ FTE Agent
MQ network
MQ network
File Transfers can be initiated via
- Scheduler
- File Directory Monitor
Ad Hoc
Script
Database
Logger
Central Management Tool

25. Web Gateway - Integration with Web
Pain-points:
Difficult to mix human imitated file transfers with existing infrastructure for machine-to-machine managed file transfer
Managed file transfers to zero-install, small-footprint devices
File Transfer Edition Helps:
A RESTful API for exchange files with an WMQ FTE network
Example web 2.0 applications to use as a starting point
Web Gateway
Agent
Agent
Agent
HTTPS
WebSphere MQ
HTTP
HTTPS
Agent
Agent

26. Protocol Bridging Agents
Support for transferring files located on FTP and SFTP servers
The source or destination for a transfer can be an FTP or an SFTP server
Enables incremental modernization of FTP-based home-grown solutions
Provides auditability of transfers across FTP/SFTP to central audit log
Ensures reliability of transfers across FTP/SFTP with checkpoint restart
Fully integrated into graphical, command line and XML scripting interfaces
Just looks like another FTE agent…
Files exchanged between FTE and FTP/SFTP
FTP/ SFTP Client
FTP/ SFTP
Agent
Agent
Agent
Protocol Bridge Agent
FTP/ SFTP Server
FTP/ SFTP Client
WebSphere MQ
FTP/ SFTP Client
Audit information

27. Security - File Transfer Access Control
WebSphere MQ already provides access control that can be used to prevent unauthorized users from accessing MQ objects (such as queues)
File Transfer Edition extends this to include authorities that relate to file transfer operations (e.g. should this user be able to transfer files from the source system, and to the destination system)

28. Securing file data transfer with SSL and WMQ AMS
WMQ FTE supports transport level encryption using SSL
Data is encrypted before it is sent over a channel and decrypted when it is received
WebSphere MQ Queue Manager
WebSphere MQ Queue Manager
svrconn channel
sndr/rcvr channels
Agent
Agent
When combined with WMQ Advanced Message Security
Allows file data to be encrypted at the source system and only decrypted when it reaches the destination system
Data is secure even when at rest on a queue
WebSphere MQ Queue Manager
WebSphere MQ Queue Manager
svrconn channel
sndr/rcvr channels
Agent
Agent

29. File Transfer Solution Design Considerations
Who and How to invoke file transfer - scheduled, event driven, ad-hoc, or via API?
How to transfer the file reliably and securely - HTTP, FTP, MQ or in combination?
How to access the file data at source and destination – File directory, MQ, HTTP, etc.?
Is any file data processing required such as transformation?
Is there any pre and post file transfer processing steps required at source and destination, such as notification?
How to handle file transfer failure?
Is there any file transfer tasks which need to be executed in sequence?
Is there any integration points with other applications/systems which are not based on files?
What’s monitoring and auditing requirement if any?
Is security a concern?
Other considerations
How big is the file, and how often are the file transfers required?
How many systems (source and destination) are involved and what system? Same location or not?
What’s the topology for file transfers, hub/spoke or point to point?

30. Resources Information Center: Redbooks / Redguides / Redpapers:
Redbooks / Redguides / Redpapers:
Getting Started with WebSphere MQ File Transfer Edition V7
IBM WebSphere MQ File Transfer Edition Solution Overview
Managed File Transfer for SOA using IBM WebSphere MQ File Transfer Edition
B2B Enabled Managed File Transfer using WebSphere DataPower B2B Appliance XB60 and WebSphere MQ File Transfer Edition
IBM Sterling Managed File Transfer Integration and WebSphere Connectivity for a Multi-Enterprise Solution
Multi-Enterprise File Transfer with WebSphere Connectivity
Trial Download:
Early Design Program
Interested in participating in the development of future versions of FTE?
Ask your local IBM representative to nominate you for the FTE EDP program

31. Any Questions?
I’ll put this diagram up as the end for our session on MQ capabilities, any questions?

32. Back-up material

33. MQ Advanced Message Security
Is a MQ plug-in, which pre-req MQ
Adds additional security services to WebSphere MQ
Enables secure message transfers at application level
Replaces costly, home-grown solutions that lack management controls
Extra Security capabilities
Policy-based authentication for each
application
Protects message contents end-to-end
including when residing in queues
Centralized control of fine-grained MQ security
policies
Audit logs of data and queue access
No changes needed to applications or queues

34. WMQ vs WMQ AMS WebSphere MQ
Authentication (local OS for server apps or peer authenticated SSL for client apps)
Authorisation (OAM on distributed, RACF on z/OS)
Integrity (SSL for channels)
Privacy (SSL for channels)
WebSphere MQ AMS
As above, additionally:
Integrity (Digital signature of message content)
Privacy (Message content encryption) 34

35. Securing file transfer with MQ AMS
File data is decrypted and signature verified after transfer
File data is encrypted and
signed before transfer
1. Send file data encrypted and signed
2. Receive file data encrypted and signed
MQ FTE Agent
MQ Server
MQ FTE Agent
MQ network
MQ AMS Client
MQ AMS Server
MQ network
MQ AMS Client
Database
Logger
Central Management Tool

36. WebSphere Message Broker Nodes for MQ FTE
Part of WMB
Message Broker
Execution Group
WMQ FTE Agent
Message Flow
WMQ FTE Agent
WMQ FTE Agent
WMQ FTE Agent
FTEInput
FTEOutput
FTEInput node
Build flows that accepts file transfers from the WMQ FTE network
FTEOutput node
Build flows that are designed to send a file across a WMQ FTE network
When WMQ FTE nodes are used in a flow an FTE agent is automatically started in the Message Broker Execution Group 36 36

37. Integration with Message Broker (1)
data transformation,
Enrichment, convert
file data to
message(s), etc.
Integrate with
non file oriented systems
File Transfers can be initiated via
- Scheduler
- File Directory Monitor
Ad Hoc
Script/API
WMB
3. Receive file data
Message Flow
Web Services, MQ, DB, etc.
MQ FTE Agent
2. Process file data
if required
1. Send file data
3. Receive file data
MQ FTE Agent
MQ Server
MQ FTE Agent
MQ network
MQ network
Database
Logger
Central Management Tool

38. Integration with Message Broker (2)
File Transfers can be invoked
By Message Broker as a service
WMB
Message Broker
MQ FTE Agent
1. Start file transfer
2. Send file data
3. Receive file data
MQ Server
MQ FTE Agent
MQ FTE Agent
MQ network
MQ network
Database
Logger
Central Management Tool

39. Integration with IBM Sterling Connect:Direct
C:D Node
Agent
Agent
Agent
C:D Bridge Agent
C:D Node
C:D Node
WebSphere MQ
Reference
C:D Node
Trading Partner
FTE Audit
C:D Audit
Inside the FTE audit trail…
The Connect:Direct Bridge capability supports managed file transfers that span FTE and C:D with a joined up audit trail
The audit information for each FTE transfer references related C:D audit information