1. Data Virtualization and Visualization
SQL access and visualization of mainframe data within IBM technologies
Rob Parker
Virtualizing mainframe data sources take advantage of zIIP engines and parallel processing which can ease the process of data access and eventually assembly. Using extracted SMF Log information and DevOps Experience data, we will explore how IBM technologies can be used to easily visualize data that would otherwise be difficult to consume.
Virtualization technology is not new and has been in the market for many years. If you attend last years conference a coworker of mine discussed data virtualization.

2. Agenda Virtualizing VSAM, DBMS, IDMS, IMS and other data sources
Accessing virtualized mainframe data via IBM Db2 technologies
SQL Access to Mainframe data
Review Log Analysis and DevOps dashboards
Dashboard Auto generation
I want to take you on a journey that starts with data virtualization and wraps up with data visualization.
IBM has a mainframe virtualization technology that treats non relational mainframe data sources, essentially the same way that Db2 treats a table when it creates a view.
Virtualization capabilities have been available for many years and IBM has put this technology into several of it’s product offerings, IDAA Loader, QDS, DVM (Any data source any platform) and IBM Open Data Analytics for z/OS (Machine Learning).
Allowing these virtualized technologies to connect to a vast amount of mainframe and non mainframe data sources: DB2, IMS (IBM Information Management System), VSAM (Virtual Storage Access Method) files, Adabas (Adaptable Database System), SMF (System Management Facilities) records, system logs, and operation logs.
IBM has also developed a dashboard technology that provides an easy to use drag and drop interface, without requiring knowledge of a programming language. All of this technology is available to Db2.
Virtualization capabilities have been available for many years, but was announced within IBM dB2 Analytics Accelerator Loader for z/OS in October 2013 and was made available for visualization capabilities in QMF in May 2016 and then followed by IZODA in July 2017 and DVM in October 2017.
Provides SQL API to keep the data fresh. Presents the data in a virtual format.

3. Considerations for virtualization
Why move the data off the mainframe or restructure it?
What does it cost for MSU? z13 or z14 (Tailored fit pricing)
Are there security risks? Does the DBA own the security credentials once it’s moved? Who’s ensuring that the data is secure?
No matter how fast the data is collected and moved, it will never be as fast as real-time, in place virtual access.
Are you moving data off the mainframe at the request of the Line of Business owners?
Moving this data tends to be slow, rigid and lacks the enterprise reach needed to support new cognitive analytics for both cloud and mobile.
Moving data off the mainframe costs money. The additional MSU used contributes to millions of dollars spent on software, hardware, labor and storage.
A recent IBM analysis found that ETL processes can consume large amount of MSU and core consumption.
Tailored Fit Pricing – Consumption and Capacity solutions-
Uses in place z/OS security
_____________________________________________________________________________________________________
With QMF version came the enhancements that you are seeing in this presentation.
The QMF Data Service Studio allows you to integrate mainframe resident data with other data sources and applications.
Uses server-side processing and heavily uses z Systems Integrated Information Processors (zIIP) processors
(up to 99%) and parallelism to deliver information from its sources with extreme efficiency.
MQ – queue manager
A VSAM example will be shown
Installation documentation can be found at:
DB2 QMF Data Service Getting Started Guide, GC
DB2 QMF Data Service Customization Guide, GC
DB2 QMF Data Service Solutions Guide, GC
DB2 QMF Data Service Studio User’s Guide, SC

4. SQL Access to Mainframe data using IBM Db2 technologies
Mainframe-resident data virtualization happens in real-time and data remains under z/OS control.
Utilizing server-side processing and z Systems Integrated Information Processors (zIIP up to 99%) making the queries cheaper to run (Minimizing MSU).
Only one set of credentials required.
Virtualization of ADABAS, DBMS, IDMS, IMS, MQ, Sequential, VSAM and zFS
Only stores the metadata definitions and retrieves the data on demand.
On the z System, the data is loaded into memory and then the results are returned to the client.
Multiple versions of the same table across different Db2’s in parallel.
Moving data off the mainframe is slow, rigid and lacks the enterprise reach needed to support new cognitive analytics for both cloud and mobile.
MSU – Millions of Service Units
With QMF version came the enhancements that you are seeing in this presentation.
The QMF Data Service Studio allows you to integrate mainframe resident data with other data sources and applications.
Uses server-side processing and heavily uses z Systems Integrated Information Processors (zIIP) processors (up to 99%) and parallelism to deliver information from its sources with extreme efficiency to minimize MSU/MIPS impact.
MQ – queue manager
A VSAM example will be shown
Installation documentation can be found at:
DB2 QMF Data Service Getting Started Guide, GC
DB2 QMF Data Service Customization Guide, GC
DB2 QMF Data Service Solutions Guide, GC
DB2 QMF Data Service Studio User’s Guide, SC

5. IBM Study - Four Year Cost Estimate
Estimate is for moving data – What about transforming and analyzing it?
Run analytic workloads as close to the data as possible.
Estimate is not utilizing zIIP, data virtualization, parallel processing or IDAA.
ODS = Operation data source
Something to consider: If data is moved off the mainframe what % of data is not static? What if we could make it look relational?
IBM Redbook:

6. The results are returned to the client
Simplified Workflow
QDS SQL engine, takes the table name(s) to the metadata to find the location of the base data
A client sends SQL
The results are returned to the client
The data is loaded into memory and the SQL is executed by QDS
Sequential
SMF
IMS
Adabas
System logs
DVM
QDS
IDAA Loader
IBM Open Data Analytics for z/OS (IZODA)
Db2
IDMS
VSAM

7. Connecting to Data Sources
Connect to the server where the DVM engine is installed.
To access the mainframe data we require server settings. These settings are the Host, Port, Userid and User Password.
QMF for z/OS has the DVM engine embedded within

8. Virtualizing Data Sources
Expand SQL, Data, Server and then Virtual Tables
Right mouse click and select Create Virtual Table(s)
The server side program in this example is named CQDS. This name is assigned upon installation. After connecting expand SQL, Data, Server and then Virtual Tables
Right mouse click and choose Create Virtual Tables

9. Db2 Virtualized VSAM Select the source type for the virtual table.
Assign a Name the end user will see for the source library
Select an Available Source Library
Select and download the source library
Expand the Source to view the fields defined in the copybook
Provide the name of the cluster

10. Accessing Virtualized Table and Data assembly
Multiples of ways to access virtualized tables and views.
Connect to the QMF Data Service.
Enter a Data Source name and connection parameters.
Filtered to only show GLOSALES%
Prep data in IBM technologies
Generate queries and utilize Analytic Queries and calculated columns to assemble the data.
Save the query and construct to make available to users.
SO Measurements

11. Publishing to Other Applications
Publish queries to QMF Vision

12. Dashboard Data Connection
Connect to QDS via IBM technologies: QMF for WebSphere, QMF for Workstation, QMF Vision
Directly connect to Db2.
Create a query and enter the server side user credentials
Retrieve data live

13. Viewing the virtualized data set

14. Dashboards for DBA’s
Data for the Lines of Business are not of interest. Why do I want to see Profit or Sales of product?
I want to see what’s occurring on my server, what process are happening and utilization.
How is my system performing? Inserts, updates, deletes and commits per second.
How are log files being consumed?
What dataset is important to capture?

15. Daily Mission Control Observing CPU utilization
Times of the day with most activity
Dashboard contains multiples charts
Data sourced from log files
Queries are run every minute

16. Log File Reports
Option to load data into Db2 in a Db2 Load format. The Log Analysis Tool generates the necessary DDL and LOAD format JCL. Loads the data into the target table and saves it in a file named LOADFILE. The LAT is an analytical tool which is used to view dB2 structures…can also perform undo
Should use Execute dynamic SQL in batch

17. Log Analysis - updates, inserts, deletes and commits
Before delving into the dashboard creation process, let’s review a previously designed Log Analysis dashboard.
The data for this dashboard was generated by the Log Analysis Tool and loaded into Db2.
The data identifies the processes that are performing different tasks on the Db2. Primarily these tasks are inserts, updates, merges and deletes. The LAT basically provides a bible to Db2 changes.

18. DevOps Processes .yaml returned as JSON and then loaded into Db2
How do you perform traditional Db2 changes? How would you manage in an agile environment? In the process of a 2 week sprint, changes need to be made quickly. Relieve the bottleneck is time during a sprint.

19. DevOps Process Dashboard
By looking at a log file, how can I visually see how many subsystems and objects are under DevOps control?
Two indicators showing that nothing is being changed in these subsystems.
Drilldown on Column Chart and see all objects being affect.

20. DevOps Process Dashboard
User applications

21. Dashboard Auto generation
Using GLOSALESVSAMSS
Create a query and enter the user credentials
Analysis Target can be either a measure or dimension
Since I do not have an active connection to my mainframe, I have extracted the VSAM content into a .csv file.
Open – csv file in Excel
Drag and drop the csv into the QMF Vision drop zone.

22. Dashboard Auto Generation

23. IBM QMF Vision Reposition and modify the KPI’s.
Change the Geo Map to contain Profit %
Change the Box Plot chart to a Pie – Remove legend, show Percent Value and Show Detail

24. Resources
QDS Installation:
IBM Redbooks:
Youtube IBM QMF Channel:
Digital Technical Engagement Site (DTE):
IBM Data Virtualization Manager for z/OS: