1. Virtual Exchange Services VES
11/01/2017

2. Background
The VES is a cloud-based platform designed to minimize the burden of building data exchanges by using standard templates that define much of the processing
It has been in service for more than two years now
Partners can use VES to supplement their existing node or to replace their node
The exchanges (e.g. SDWIS) are configured not installed
Partners can get started quickly using the VES web application wizards and powerful dataflow import functions
All new components and features such as dataflow definitions and services are shareable (from VES or between partners)

3. Why VES? Provides a low cost option for partners to exchange data
Eliminates the need for partner node HW/SW
Streamlines development and maintenance of dataflows
Updates to templates are made centrally
Minimizes the support staff needed for a node
You do still need a database person

4. What are the Virtual Exchange Services?
A “data exchange as a service” platform that hosts partner services for State, Tribe, Region, Agency, Industry, etc.
This means that all of the functionality of an application server is provided as standard reusable components.
Partners no longer deal with the complexities of development, data formatting , dataflow logistics, notifications…
Partners import exchanges (pre-filled forms) from a template for each dataflow they deploy
The system generates REST and SOAP services based on the imported templates/forms
Data submissions and data publishing flows are supported

5. What’s New in VES?
Upgraded cloud infrastructure - Tripled the processing power
Added new Q/A service to speed testing of dataflows
Optimized XML creation processes to speed throughput of large submissions
Upgraded to .Net workflow v4.5
Added support for ORACLE stored procedures in the workflow process
             

6. What’s New in VES ? Added Custom Header storage support
Implemented Owner Messaging (notifications/messages to all owners)
Enhanced security policy module (access control managed by owners)
Added supported for tasks with dynamic parameters (rolling dates or date ranges)
 Implemented workflow persistence and pending transaction recovery
 Added automatic REST URL generation for Query services
             

7. Loads Staging data to either
VES
Partner
Environment
Azure Cloud
Exchange Network/
CDX/ Other Users
REST/
SOAP/ FTP ..
Virtual
Exchange
Services
Admin
Configures dataflow based on Template
Submissions
Establishes connectivity
Local Staging Tables
Internet Service Bus
Or VPN
Reports
Notifications
Loads Staging data to either
Local or cloud DB
Cloud-based
Staging Tables
Production Database

8. SDWIS Dataflow Tasks Owner: VES SDWIS Prime State
Create SDWIS Workflow in VES
Establish SS Database Connectivity and Data source (if using local SS database) or
VES establishes Cloud SS Database
Scheduled VES workflow calls SDWIS Service for a specific state with parameters
State
Date Range
Stylesheet Name(assuming you are supporting state specific data)
SDWIS Service
Extract data for the state (SQL query )
Transform raw data from SQL query into 1 SDWIS XML Schema (Stylesheet mapping)
Return SDWIS State XML document as a Blob in the response to VES service call
VES
Workflow continues – Logs, Notifies, Processes Response
Connect to SDWIS State Db in state (with established VPN or Service Bus) and inserts Blob or
Connect to Cloud SDWIS State Db and inserts Blob
Execute Load Process on SDWIS State Db
Reviews processing reports

9. Potential SDWIS Dataflow Design
Azure Cloud
Partner
Environment
SDWIS Prime NCC
Virtual
Exchange
Services
State Admin
Tasks
Configures Data Source
PRIME
Run your DB scripts
to establish
Local database
(pointer to your database)
Create standard dataflow
Create schedulers
Create State Dataflows
Establishes VPN or
Service Bus Connection
(if necessary)
Extract Single
States Data
Service Parameters:
Service,
State,
Date Range
Stylesheet …
Stylesheet
SS XML Doc
Local State
Specific Database
Blob
State blob data
inserted to either
Local or cloud DB
Solicit
Execute State specific Stored
Procedure
Download
Standard EN
Asynchronous
SOAP Request
Response
Exchange Network Node
SDWIS Prime creates custom
DBs and procedures to
load data
You create a SOAP service
On your node to support
Asynchronous extracts
from SDWIS Prime
Execute State
Specific Stored
Procedure
Cloud
State Specific Database
Note: Supports any file size
without timeout issues
Potential SDWIS Dataflow Design

10. Home | About | FAQs | Help
Configuring a Service in the VES Admin Tool
Test User | Profile | Logout
Home | About | FAQs | Help
Virtual Node Administration Platform (VNAP)
Node Dashboard
Services Setup Wizard
Steps
Service Settings
Database
Service
Template
Finish
Method Type
Explanation of Fields
Review these requirements to ensure your node is working.
Solicit V
Workflow
Select a workflow… V
Node ID / Description:
CWANJ - Clean Water Act NJ Waterways
Dataflow:
CWANJ
Service Status
Enabled V
Service Description
Display all water transaction data
Data Source
Please select an existing data source and provide SQL statement to test
cgi.oracle.test V
SQL Statement
SELECT * FROM {TABLE} WHERE {FIELD}.{TABLE}
TEST QUERY

11. REST Data Services
Virtual Exchange Services support REST type services for publishing data
GetCountyCode – actual REST service on my Chris Clark Production VES:
This is what the REST service URLs look like for my VES REST services
JSON format:
XML format:

12. Partners and Dataflows
Hualapai Tribe:  WQX production flow
San Joaquin Valley Air District:  ICIS AIR
Jefferson County Health Dept: ICIS Air
Klamath Tribe: WQX
Alabama: ICIS-AIR
Georgia: ICIS-AIR dataflows , EIS
Region 1: Storm 3 Buoy Flow, SPCC Tanks
St. Regis Mohawk Tribe
Gila River Tribe - AQS
Oregon
Tennessee: ICIS-AIR, ICIS NPDES
Texas: AQS, Facility FRS
Maine: ICIS data access (Export)
Gold Systems: WQX, ICIS-AIR, AQWMS
New Hampshire: Facility, AQS, etc..
South Dakota: TRI-OUT, ICIS-AIR, EIS
Pacific States Marine Fisheries Commission: Fishery information publishing service
Washington - Radon
Other Organizations
OR,CT,VT,Chesapeake Bay Program

13. Data Exchanges Supported
AQS 3.0
ICIS-AIR
FACID v3.0 (FRS)
E-Beaches
EIS
EMTS v3.0
ENDS v2.0
ICIS-NPDES
SDWIS
SPCC  v1.1 (Region 1)
Buoy Data (Region 1)
TRI (OUT)
UIC
WQX v2.1
CROMERR
OWIR_ATT
RCRA 5.2,5.3
Radon

15. Questions
Contact Information:

16. More Advantages Decreased Plug-in development time
No Plug-in deployment
Streamlined setup
No administrative staff
Simplified testing and QA