1. Streams – DataStage Integration InfoSphere Streams Version 3.0
Mike Koranda
Release Architect

2. Agenda What is InfoSphere Information Server and DataStage?
Integration use cases
Architecture of the integration solution
Tooling

3. Information Integration Vision
Transform Enterprise Business Processes & Applications with Trusted Information
Deliver Trusted Information for Data Warehousing and Business Analytics
Address information integration in context of broad and changing environment Simplify & accelerate: Design once and leverage anywhere
Secure Enterprise Data & Ensure Compliance
Build and Manage a Single View
Key Speaking Points
InfoSphere is the first true IIG platform in the market
Comprehensive
It has the most mature and comprehensive set of capabilities in the market. Each component is a recognized market leader in its respective technology
Integrated
The components share a common foundation of meta data, data discovery, and a business glossary.
The components are integrated to work with one another to address the enterprise use cases.
Intelligent
The objective of InfoSphere is to offer the functionality required to actually integrate and govern data – its objective is to be far more than a tool, or a collection of tools.
It includes automation of repeatable tasks, and business logic to manage proactive alerts and notifications
Consolidate and Retire Applications
Integrate & Govern Big Data
Make Enterprise Applications more Efficient 3

4. Structured Repeatable Linear Unstructured Exploratory Iterative
IBM Comprehensive Vision
Traditional Approach Structured, analytical, logical
New Approach Creative, holistic thought, intuition
Data Warehouse
Hadoop Streams
Data Warehouse
Hadoop Streams
Transaction Data
Web Logs
Internal App Data
Social Data
Structured Repeatable Linear
Unstructured Exploratory Iterative
Information Integration & Governance
Mainframe Data
Text & Images
Key Points
Traditional technologies are very well suited to structured, repeatable tasks – when you do something many times it makes sense to structure it
Also have controls in place for the accuracy and quality of the data
Historical data – trend analysis
New technologies are complementary – they address speed and flexibility
Very good an one-time or ad-hoc analysis
Also good at exploration – determining new questions to ask
The point is organizations need both sides – and data growth (or big data) is a challenge for both sides. A big data platform has to address both sides to truly address enterprise needs.
OLTP System Data
Sensor Data
Traditional Sources
New Sources
ERP data
Traditional Sources
New Sources
RFID

5. 1 4 2 5 3 6 IBM InfoSphere DataStage
Industry Leading Data Integration for the Enterprise
Simple to design - Powerful to deploy
Rich capabilities spanning six critical dimensions
Developer Productivity Rich user interface features that simplify the design process and metadata management requirements
Runtime Scalability & Flexibility Performant engine providing unlimited scalability through all objects tasks in both batch and real-time 1 4
Transformation Components Extensive set of pre-built objects that act on data to satisfy both simple & complex data integration tasks
Operational Management Simple management of the operational environment lending analytics for understanding and investigation. 2 5
Connectivity Objects Native access to common industry databases and applications exploiting key features of each
Enterprise Class Administration Intuitive and robust features for installation, maintenance, and configuration 3 6

6. Use Cases - Parallel real-time analytics
An enterprise may wish to send data from DataStage to Streams to perform near real-time analytic processing (RTAP) on the data stream. By sending data to Streams from the DataStage flow, Streams can perform RTAP in parallel to data being loaded into a warehouse by DataStage.

7. Use Cases - Streams feeding DataStage
Alternatively, an enterprise may wish to send data from Streams to DataStage. A typical use-case might be processing telco call details: the Streams job performs RTAP processing, and then forwards the data to Data Stage to enrich, transform and store the call details for archival and lineage purposes.
A Streams application may require a data source that is not provided by Streams but has a first-class connector in DataStage (e.g. SAP)

8. Use Cases – Data Enrichment
In enrichment deeper analytics are offloaded from the main DataStage flow performed in streams either because of scalability or stream specific analytic capabilities like text analytics, video analytics time series analysis, SPSS model scoring, etc.
Care should be taken in this scenario as DataStage can provide a transactional flow while the interaction to steams can lose tuples unless the applications is properly architected.

9. Runtime Integration High Level View
DataStage
Streams
Job
Job
DSSource / DSSink Operator
Streams Connector
TCP/IP
Composite operators that wrap existing TCPSource/TCPSink operators

10. Streams Application (SPL)
use com.ibm.streams.etl.datastage.adapters::*; composite SendStrings {
type RecordSchema =
rstring a, ustring b;
graph
stream<RecordSchema> Data = Beacon() {
param iterations : 100u;
initDelay:1.0;
output Data : a="This is single byte chars"r,
b="This is unicode"u; }
() as Sink = DSSink(Data) {
param name : "SendStrings";
config applicationScope : "MyDataStage";
When the job starts, the DSSink/DSStage stage registers its name with the SWS nameserver

11. DataStage Job
User adds a Streams Connector and configures properties and columns

12. DataStage Streams Runtime Connector
Uses nameserver lookup to establish connection (“name” + “application scope”) via HTTPS/REST
Uses TCPSource/TCPSink binary format
Has initial handshaking to verify the metadata
Supports runtime column propagation
Connection retry (both initial & in process)
Supports all Streams types
Collection types (List, Set, Map) are represented as a single XML column
Nested tuples are flattened
Schema reconciliation options (unmatched columns, RCP, etc)
Wave to punctuation mapping on input and output
Null value mapping

13. Tooling Scenarios
User creates both DataStage job and Streams application from scratch
Create DataStage job in IBM Infosphere DataStage and QualityStage Designer
Create Streams Application in Streams Studio
User wishes to add Streams analysis to existing DataStage jobs
From Streams Studio create Streams application from DataStage Metadata
User wishes to add DataStage processing to existing Streams application
From Streams Studio create Endpoint Definition File and import into DataStage

14. Streams to DataStage Import
On Streams side, user runs ‘generate-ds-endpoint-defs’ command to generate an ‘Endpoint Definition File’ (EDF) from one or more ADL files
User transfers file to DataStage domain or client machine
User runs new Streams importer in IMAM to import EDF to StreamsEndPoint model
Job Designer selects end point metadata from stage. The connection name and columns are populated accordingly.
IMAM
Streams
command line or Studio menu
ADL
EDF
EDF
Xmeta
ADL
FTP

15. Stage Editor

16. Stage Editor

17. DataStage to Streams Import
On Streams side, user runs ‘generate-ds-spl-code’ command to generate a template application that from a DataStage job definition
The command uses a Java API that uses REST to query DataStage jobs in the repository
The tool provides commands to identify jobs that use the Streams Connector, and to extract the connection name and column information
The template job includes a DSSink or DSSource stage with tuples defined according to the DataStage link definition
Streams
command line or Studio menu
Java API
Xmeta
SPL
REST API
HTTP

18. DataStage to Streams Import

19. Availability
Streams Connector available in InfoSphere Information Server 9.1
The Streams components available in InfoSphere Streams Version 3.0 in the IBM InfoSphere DataStage Integration Toolkit