Download presentation
Presentation is loading. Please wait.
1
Top 10 Tips and Techniques to Jumpstart
Your iWay Data Quality and Mastering Projects Jeff Paoletti
2
10 Installation Options
3
Installation DQS and MDS are available as
Standalone alone installations Integrated into the iWay Service Manager (ISM) runtime and as Plug-ins to the Integration Tools (iIT) designtime
4
Standalone alone installations simple unzip
Useful for “loosely coupled” integration topography DQ plan runs as an online webservice runs in its own memory & address space (usually on a different server) No version issues No “updates” - new version requires complete install
5
Installation Integrated into the iWay Service Manager (ISM) runtime
and as Plug-ins to the Integration Tools (iIT) designtime DQ library is loaded into the ISM runtime process + faster - scalability
6
Check For Updates Installation To install DQ/MD plugins into iiT
iIT menu > Help > Install New Software Current version is Check For Updates
7
SYNCHRONICITY IIT PLUGINS IIT RUNTIME \lib This is the DQ engine
8
ISM installation ISM RUNTIME
Installed as an option when ISM was installed
9
SYNCHRONICITY Standalone DQS installation Runtime
Can be ahead of what’s in ISM runtime Moral of the story: be careful! Synchrionicity issues
10
9 Licensing
11
Licensing Changes Online Registration
Temporary licenses no longer available on download site
12
Loqation Loqation Loqation Address Cleansing
13
Loqate
14
Loqate Coverage of 240 Countries
Validate against Global Knowledge Repository (GKR) Provide Geocodes Parse unstructured text into labeled address components Correct misspellings Add missing components and postcodes Standardize to the correct local format Certified option ( CASS for US)
15
Install GKR data using a key provided via iWay
Updates available quarterly
16
Loqate Loqate steps have been integrated into the DQ GUI
17
Configuring Loqate Steps
In DQ, identify the Data Folder Map Input and Output fields
18
Loqation, Loqation, Loqation
Address Verification Code (AVC) An 11 character code giving all the details of the address analysis Evaluate in this order: Verification Level (“4”) Verification Status (“V”) Matchscore (“100”) Postal code status (“6”) Parsing Status
19
Verification Level Delivery Point (PostBox , Suite, SubBuilding) Premise 3 - Thoroughfare (street) 2 - Locality (city) 1 - AdministrativeArea (State) 0 - None
20
Verification Status V - Verified
a complete match was made between the input data and a single record from the GKR P - Partially Verified a partial match was made U - Unverified Unable to verify A - Ambiguous More than one close reference data match
21
Matchscore Range : 0 – 100 Gives no indication of the level of verification (that is given by the post-processed verification match level) Indicates how much (or little) the input data has been changed i.e “Trustability” General Rules of thumb 100% means no changes have been made. > only a character or two have been changed < 80 might be considered unacceptable (it’s guessing too much !)
22
Processing the AVC
23
CASS CASS US Postal Certified Greater accuracy
Discounts for volume users
24
CRAIGS LIST
25
Create and test an online service from a component
Now the result is in the Response window
26
Create and test an online service from a component
27
Create and test an online service from a component
28
Create and test an online service from a component
A component can be reused as an online server and quickly tested with just a few clicks of the mouse. Here we have a simple component to verify an address
29
Create and test an online service from a component
Now just right click on the component and select ‘Publish as Online Service’ Optionally change the style/type SOAP,JSON,JMS,XML Single vs multiple record As well as name and location Click ‘Finish’
30
Create and test an online service from a component
We now have the configuration for our service!
31
Create and test an online service from a component
To start the service just click the Run button in the toolbar And watch in the console as your service starts By default your service is listening on port 8888
32
Create and test an online service from a component
To test a service just click the Test Editor in the Console Here you can change the Request then click ‘Send Request’
33
Determine and configuring threads for online services
34
Determine and configuring threads for online services
Online services run in their own JRE and threads can be specified for the container as well as for each service The total number of threads for the JRE should be large enough for all concurrently running services as well as threads for all people that may be using the console If your service can receive concurrent calls the threads must be modified! JRE level threads are set in [dqs_home]/runtime/server/etc Service threads are set in the .online file for the service via either the GUI or directly editing the file
35
Determine and configuring threads for online services
There are two ways to modify the service threads Via the GUI: Directly editing the .online file The JRE should be restarted to pick up changes
36
Determine and configuring threads for online services
The threads for the JRE should be the maximum number of threads for concurrent services plus threads for users accessing the console Edit [dqs_home]/runtime/server/etc/default.serverConfig Scroll to around line 76 – look for the <listeners> section For each of the listeners you are using – default unsecure port of 8888 and any other listeners defined to change
37
Building a custom function
38
Building a custom function
Create a new Java project Make sure to add the [dqs_install]/runtime/lib to your User Library!
39
Building a custom function
Create a directory named ‘META-INF’ and then a text file named ‘cif-plugin.xml’ in the newly created directory. Open ‘cif-plugin.xml’ and add the following, making sure to set the buildString to your version of DQS: <?xml version='1.0' encoding='UTF-8'?> <cif-plugin name="DQC Functions"> <version-info buildString="9.0.0.ga "/> <startup-class>com.ibi.dqs.functions.PluginStartupClass</startup-class> </cif-plugin>
40
Building a custom function
Right click on the ‘src’ directory in your project and create a package named ‘com.ibi.dqs.functions’ – this was part of the ‘StartupClass’ defined on the previous slide Create a new class in the package named ‘PluginStartupClass.java’ and paste the following lines: package com.ibi.dqs.functions; public class PluginStartupClass implements IPluginStartupClass { public void pluginStart(PluginConfiguration config) { UserFunctionRegister.registerUserFunctionClass(DQCFunctions.class); }
41
Building a custom function
Resolve the dependencies by hovering over IPluginStartupClass, PluginConfiguration and UserFunctionRegister and selecting the Import option In the package create another class named ‘DQSFunctions.java’
42
Building a custom function
Add the following code to the new java file package com.ibi.dqs.functions; public class DQSFunctions { public static String peek(String string) { System.out.println("[peek] " + string); return string; } Make sure everything is saved Right click on the project and select ‘Export..’
43
Building a custom function
For the export type, select ‘Jar’ Click Next Make sure your project is selected! Give the JAR a name Click Finish Copy the jar to [dqs_install]/runtime/lib
44
Building a custom function
Start DQS and create a test plan In the column assigned, create a dummy attribute to use the new function
45
Building a custom function
Save and run to see your new custom function in action!
46
Dynamic Expression Assigner
47
Dynamic Expression Assigner
Do you need to use dynamic formulas during the execution of your plan or component? Dynamic Expression Assigner may be your answer! For example, if you have an input file such as the following
48
Dynamic Expression Assigner
Create a plan with this file as the input, add a Dynamic Expression assigner and a text file writer In the Dynamic Expression Assigner set the expression to the formula
49
Dynamic Expression Assigner
When we execute the plan we see the result of the dynamic calculation!
50
New JSON Reader and JSON Call
51
New JSON Reader and JSON Call
New with the v9 release we now have a means to read from a JSON file or execute a call and parse a JSON response. Similar to XML Reader, but different.. JSON Reader Allows reading from files JSON Call Ability execute a HTTP GET or POST with optional JSON template on call Both JSON Reader and JSON Call have the same configuration for parsing the output
52
New JSON Reader and JSON Call
JSON Reader example A sample input file with an array of names, company and address
53
New JSON Reader and JSON Call
JSON Reader example Drag in a JSON Reader, add a Data Stream reflecting your data structure. The root of a JSON file is ‘$.’!
54
New JSON Reader and JSON Call
JSON Call example Similar to JSON Reader, the JSON Call parses a JSON results but the response is from a HTTP GET or POST JSON Call can also pass parameters as part of the request
55
New JSON Reader and JSON Call
JSON Call example Just set the URL, Method and optionally the Input Template The Reader section is the same as in the JSON Reader step
56
New Sample Component Projects
57
New Sample Component Projects
New with v9 sample component projects ship with the product OOTB functionality various functionality including: Country code lookup Basic credit card number validation Basic EAN code validation Phone number formatting for several countries validation SSN / SIN validation Including documentation for each component
58
New Sample Component Projects
Add a new Project and select from the template list
59
New Sample Component Projects
Drag component into a plan or comp
Similar presentations
© 2025 SlidePlayer.com Inc.
All rights reserved.