2. Data Consistency Verification with Veridata
Lorena Lobato Pardavila - CERN
GoldenGate captures and delivers replicated data but it won´t check the data for you unless you add some custom codes to the validations. Veridata can do a comparison that will report the data discrepancies on the bad data and any data that GG can skip.
Replication Technology Evolution for ATLAS Data Workshop

3. Veridata Overview
Is a high-performance cross-platform data comparison tool that supports high-volume compares
Allows data consistency validation on “hot” data sets
CLIENTS
REPOSITORY
OGG Veridata Agents
SOURCE
OGG Veridata CLI
OGG Veridata Server
TARGET
Oracle GoldenGate Veridata is a high-speed data comparison solution that identifies and reports on data discrepancies between heterogeneous databases without interrupting ongoing operations.
First at all, we have Oracle GoldenGate Veridata Server which is the ley component of the entire Veridata System. It performs several functions:
-Coordinates all aspect of the Oracle GoldenGate Veridata execution tasks with the agents
-Compares de data collected from GoldenGate Veridata Agents
-Conducts the confirmation comparisons of out-of-sync data
-Produces out-of-sync, confirmation and performance reports.
This server connects with the repository, which stores all the configurations for all the comparison objects and rules. The metadata is stored as clear text in the repository database so the reports can be configured by the custom. It also stores the job status for integrating our ETL and monitoring seamlessly.
Vericom is a command-line interface to schedule the compare jobs. It can override some profile settings, partition sql-predicates, shut down Veridata Server and generate/review the out-of-sync reports. However, it cannot configure the comparison settings and rules that can be done on the Veridata Web Client.
This is the tool that we´ll use to configure most of the Veridata tasks and that i´ll explain deeply later on. It performs functions as
-Configure all comparison objects and rules, including connections, groups, jobs, etc
-Starts and stops the comparison jobs
-Provides real-time status on the job status
-Reviews the comparisons and out-of-sync data reports
You might wonder how can be databases being monitored from Veridata. There will be many agents as databases (one each one). These agents executes the SQL to fetch and return blocks of data to the Server to comparte. They also extract the out-of-sync rows detail to the Server for reporting purposes. They have their own config.properties file which will be manipulated in order to monitor the databases.
n : n
DATABASES
OGG Veridata Web
Replication Technology Evolution for ATLAS Data Workshop

4. Veridata Overview
When Veridata components are installed, we have to connect to the Server and…this is what we get.
This is a overview from web home where you can see a summary of the jobs that I was running. You can see jobs running in real time or when they have already finished. No worries if you don´t know what´s a job. For the moment, this is to show how Veridata works once everything is installed. But to have Veridata installed, which are the system requirements?
Replication Technology Evolution for ATLAS Data Workshop

5. System Requirements
DYNAMICPORTLIST in each GoldenGateManager process (Might use 20 ports)
For Veridata Agent
Every VA must be installed for each database instance that contains data that is to be compared
Possibility of running multiple agents from a single install
1GB of RAM
Java or later is required for release
Ojdbc6.jar
For Veridata Server : 200 MB of fixed virtual memory for basic task
Privileges: Create a user and password and database role named VERIDATA_ROLE
CREATE SESSION, CREATE TABLE, CREATE VIEW, CREATE PROCEDURE, CREATE SYNONYM
Grant VERIDATA_ROLE to the user
Specify QUOTA UNLIMITED on the user's default tablespace 1.
2.One Oracle GG Veridata Agent must be installed for each database instance that contains data that is to be compared. At minimum, therefor, there will be installed two agents, one to retrieve source rows and one to retrieve target rows . One agent can retrieve rows from different databases or schemas within a given database instance but one agent cannot retrieve rows from different database instance.
What it´s true is can have possibility of running multiple agents from a single install, running each agent with its own config.properties. 3.
Replication Technology Evolution for ATLAS Data Workshop

6. Setting up the Veridata Compares
Create database connections, source and target
Create group, a group of comparing pairs
Create compare pairs, what tables to compare
Create a job, to run one or more groups
Run a job
Compare results
1. Connection : Specify how OGG Veridata Server connects to an agent(each machine) that connects to an Oracle database.
2. Group : A logical container of object (for example in this case the tables)to be compared from a source and target database along with connections to the databases. 3.
4. Job : Contains Groups. Job is associated with a comparison. After a job is run the result of the comparison become available.
Replication Technology Evolution for ATLAS Data Workshop

7. Step 1: Connection
Create the connection to the corresponding databases where Veridata Agents are displayed
Click the connection Configuration on the left panel as shown, then fill the information for both databases. Make sure GoldenGate Veridata agents are installed on both databases onstances, otherwise you won´t be able to connect. The ports to specify is the same port you have specified on config.properties from the agent folder installed.
Replication Technology Evolution for ATLAS Data Workshop

8. Step 2: Groups Creation
Next step it would be adding source and target connections to the group. One group can have many compare pairs and one job has many groups. For instance, you can break down the group by subject areas, or by schedules.
The group envolves compare pair, which you can create click on this button.
Replication Technology Evolution for ATLAS Data Workshop

9. Step 3: Compare pair
But, what’s a comparison? Veridata is based in this parameter and is what we need to compare )for instance, compare tables, materialized views and views). The compare pair is a set of corresponding source and target tables. T
You can use Manual Mapping to add the compare pairs. After picking up the compare pair, make sure the compare pair names is in the Compare Pair detail section and save.
Replication Technology Evolution for ATLAS Data Workshop

10. Step 4: Job Creation
Jobs are like shell scripts, they are used to run a series of compare groups. There is not built-in scheduler in Veridata, so you can use OS schedulers as crontab, Windows schedulers ..etc to run “vericom” command-line jobs. I prefer to use Veridata web so I´ll show you how you can create a job. Click on Jobs configuration and fill the Job and group name field. There are the only required parameters.
This is an overview of list of jobs already created. And in this case, we are seeing IN2P3 job parameters.
Note:If you click on Tier1 name, you go to the previous slide (Groups configuration)
Replication Technology Evolution for ATLAS Data Workshop

11. Step 5: Run / Execute Job
To compare job , click on Run/Execute job tab in the left panel on the Veridata Main screen.
Or you can use the command as indicated on vericom command line. Anyway asap you click on the button Run job, Veridata will start to compare the databases.
Replication Technology Evolution for ATLAS Data Workshop

12. Step 6: Results
When it finishes, you can check the results filtering as you wish.
Replication Technology Evolution for ATLAS Data Workshop

13. Reports
Replication Technology Evolution for ATLAS Data Workshop

14. Veridata Key Points
Powerful tool for the data missing-synchronization identification
Along with GoldenGate, allows data real-time integration and continuous availability solutions validated data consistency
Stores OOS(Out-of-Sync) reports in binary, XML or both
Agents can connect remotely, not needed installation in target databases
200GB production data have been compared in an ATLAS environment with a speed of MB/sec
Veridata coming soon
It will have the ability to repair/fix out of sync data
It will require WLS
Summer Student to investigate deeply Veridata
Replication Technology Evolution for ATLAS Data Workshop

15. Live Demo itrac910:8830/veridata/
Replication Technology Evolution for ATLAS Data Workshop

16. THANK YOU! lorena.lobato@cern.ch
Replication Technology Evolution for ATLAS Data Workshop