GoldenGate Monitoring and Troubleshooting Lorena Lobato Pardavila - CERN Replication Technology Evolution for ATLAS Data Workshop

Outline Database level Monitoring GoldenGate Troubleshooting Questions Oracle GoldenGate Director and Oracle GoldenGate Enterprise Manager Plug-in Database level Monitoring GoldenGate Troubleshooting Questions Director no demo because of web is not completed… Replication Technology Evolution for ATLAS Data Workshop

Outline Database level Monitoring GoldenGate Troubleshooting Questions Oracle GoldenGate Director and Oracle GoldenGate Enterprise Manager Plug-in Database level Monitoring GoldenGate Troubleshooting Questions Replication Technology Evolution for ATLAS Data Workshop

Oracle GG Director: Overview Multi-tiered, client-server application that enables the configuration and management of Oracle GoldenGate instances from a remote client OGG Director Server Domain GGSCI OGG Director Web OGG Director Server Application GGSCI OGG Director Client GGSCI OGG Director Administrator In order to administer OGG, you have to open many SSH windows for each server on which GG is running. But there is a monitoring tool, which helps you to administer all processes from one instance.OGG Director is a multi-tiered GUI application that manages the GG processes without the need to login to the GG server and easier than ggsci command line, as it show a view the near real-time lag on each procss and a overview of the whole system. Director consists of several components: We have an OGG Director server domain which is composed of OGG Director server application, and a monitor agent and this Server Domain is running WebLogic Server. Is not needed running it but it is needed having all their components installed. At the same time, this Domain is connected with a Repository database which stores all the configurations and the statistics in clear text format. Then, on one hand, we have that Director can link to many manager processes running on OGG instances. The Manager IP address and port number are the only required parameters for the Server to connect with them. On the other hand, we have set of clients OGG Director Administrator which is used to set up all the GGSI instance configurations. All of this information is stored in the Director database. From this client is when database connection are established. Then we have OGG Director Client where we can create change or delete the parameter files and trail files. It also displays online the system overview and it´s the main part of this system considering that you can control processes from here. You can control and monitors them on Director web too but not manipulate them, indeed . So after this overview, let´s go to check the system requirements needed for installing OGG Director Monitor Agent Clients OGG Instances OGG DIRECTOR DATABASE Replication Technology Evolution for ATLAS Data Workshop

System Requirements Oracle GoldenGate Director Server 1 GB of RAM up to 1.5 GB (recommended) of free disk space. Dedicated http port (default port is 7001) DYNAMICPORTLIST in each Manager process Oracle WebLogic Server 11g Standard Edition 10.3.1, 10.3.2, 10.3.3, 10.3.4, or 10.3.5 or 12c (12.1.1) Oracle GoldenGate Director Web Supports Microsoft Internet Explorer v6.0 or later and Mozilla Firefox v2.0 or later No software is installed on the client system for the OGG Web Oracle Director Desktop Client Same version as that Oracle GoldenGate Director Server Install Java Runtime Environment (JRE) on each cliente host Security requirements Oracle GoldenGate Director Server connects to remote Oracle GoldenGate instances through each of their Manager ports. If a firewall is present within the Oracle GoldenGate network, do the following: 1. Use the DYNAMICPORTLIST parameter in each Manager parameter file to specify a list of available ports. For usage and syntax, see the Oracle GoldenGate Windows and UNIX Reference Guide or the Oracle GoldenGate HP NonStop Reference Guide, as appropriate for the platform. 2. Open those ports, plus the Manager port and the Oracle GoldenGate Director Server http port, through the firewall. Replication Technology Evolution for ATLAS Data Workshop

Oracle GG Director: Overview OGG Director Administrator Tool Replication Technology Evolution for ATLAS Data Workshop

Oracle GG Director: Overview OGG Director Desktop Client Replication Technology Evolution for ATLAS Data Workshop

Oracle GG Director: Overview OGG log Replication Technology Evolution for ATLAS Data Workshop

Oracle GG Director: Overview Alerts configuration Replication Technology Evolution for ATLAS Data Workshop

Oracle GG Director: Web Server_domain/acon Replication Technology Evolution for ATLAS Data Workshop

OGG Enterprise Manager Plug-in (Live) For installing the plug-in: Enterprise Manager Cloud Control 12c Bundle Patch 1 (12.1.0.1) and later Oracle GoldenGate 11g Release 2 (11.2.1.0.1) and later Management features: Monitor Oracle GoldenGate instances. Gather configuration data and track configuration changes for Oracle GoldenGate instances. Raise alerts and violations based on thresholds set on monitored targets and configuration data. Support monitoring by a remote Agent. A Local Agent is an agent running on the same host as the Oracle GoldenGate instance. www.oem-test.cern.ch Nice login Replication Technology Evolution for ATLAS Data Workshop

OGG Enterprise Manager Plug-in Summary Replication Technology Evolution for ATLAS Data Workshop

OGG Enterprise Manager Plug-in (Live Demo) https://oem-test.cern.ch/ Replication Technology Evolution for ATLAS Data Workshop

Outline Oracle GoldenGate Director and Oracle GoldenGate Enterprise Manager Plug-in Database level Monitoring GoldenGate Troubleshooting Questions Replication Technology Evolution for ATLAS Data Workshop

Outline Database level Monitoring GoldenGate Troubleshooting Questions Oracle GoldenGate Director and Oracle GoldenGate Enterprise Manager Plug-in Database level Monitoring GoldenGate Troubleshooting Questions As databases professionals, one of the most difficult aspects of working with OGG is how to troubleshoot and resolve errors that occur throughout the lifecycle of the implementation. Due to the nature of OGG functionality and operations as enterprise software, it involves all layers of the technology stack from source to target database to network issues and more. The goal of this chapter is to provide the best possible overall framework to identify and resolve problems that occur within the OGG. We are going to review the most common problems by using a holistic approach to troubleshooting. Because OGG touches multiple areas including database, network, storage and operating systems, we’ll focus on these areas as related Replication Technology Evolution for ATLAS Data Workshop

Troubleshooting Network Hardware Storage/Disk OGG REPORTS Operating System Database Application Server Let´s go to take a visual approach to OGG root-cause analysis. What does GoldenGate have to trave and identify the source of problems from this analysis. Storage/Disk OGG REPORTS ORACLE DATABASES LOGDUMP Replication Technology Evolution for ATLAS Data Workshop

OGG Reports Oracle GoldenGate Process Failures GGSCI> info all GGSCI> info <process_name>, detail GGSCI> view report <process_name> Oracle GoldenGate Trail Files Problems Trail files that don’t roll over: info exttrail * Trail files purging issues: PURGEOLDEXTRACTS The first type of problem within OGG environments are process failures. These may occur on the source, target or on both environments for one or more processes. Replication Technology Evolution for ATLAS Data Workshop

OGG Reports Oracle GoldenGate Error Log Analysis : ggserr.log GGSCI> view ggsevt Configuration issues Incorrect sofware version installed for OGG Database configuration issues on the source and/or target with OGG OGG parameter file configuration issues Operating system configuration issues Network configuration issues Configuration issues: From second points *Database configuration issues on the source and/or target with OGG, like can be -No DDL script installed -No checkpoint table specified for replpicat -No minimum supplemental loggin is enabled… *OGG parameter file configuration issues -Unable to open exttrail file -Bad written *Operating system configuration issues -LD_LIBRARY problems -ORACLE_HOME *Network configuration issues -Network access and connectivy -Network latency -Network data-transfer issues -tnsnames.ora -firewall closed Replication Technology Evolution for ATLAS Data Workshop

Oracle Database issues with GoldenGate Extract errors = logminer problems Replicat issues = streams apply problems Data-synchronization issues Missing column and rows errors Procedures the same like for Streams! Repeat: there are may problems but we don´t have time to envolve all of them so let´s try to focus in the headers Extract Can’t access Oracle database archive and redo logs Failure condition due to Oracle source database issues Data Pump errors Replicat issues Errors on the Oracle database target system Hangs on the target system Experiences an abends failure on the target system Fails on large transactions Incompatible record errors with a replicat Data-synchronization issues Missing column errors Fetch failures Replication Technology Evolution for ATLAS Data Workshop

LogDump GroupID Tokens Trail Info Signature, Compatibility, Character Set, Creation Time, URI, URI History, File Name, Multi-part, Trail Sequence Number, File Size, First CSN, Last CSN, First IO Time, Last IO Time, Log BSN Machine Info Operating System Name, Node Name, Release, Version, Hardware Database Info Vendor, Name, Character Set, Major Version, Minor Version, Version String, Client Character Set, Client Version String, Client NCharacter Set, Db Locale, Db NCharacter Set, Db Object Metadata, Db Time Zone Producer Info Name, Data Source, Major Version, Minor Version, Maintenance Level, Bug Fix Level, Build Number, Version String Continuity Info Recovery Mode OGG LogDump utility is the Swiss Army knife of analysis and troubleshooting. It´s complicated to understand it but it´s a potent tool in this field. By using it. We can view data in transactions present for a trail file, identify missing transactions, and troubleshoot replication issues (as duplicate and lost data transactions). As you can see here is the Trail File structure. To fully comprehend the data available via Logdump, one must first know something about the OGG Trail and its record layout. The following diagram provides a simplified view of an OGG Trail. The File Header contains a wealth of information about the physical environment that produced the Trail file and Trail file contents. This information includes (CLICK) Each source operation captured is encapsulated into a unique transaction record. This record consists of a GoldenGate Header (GHDR), variable length data area, and GGS and User Tokens. GHDR The GoldenGate Header, or GHDR, area details information about the source operation and transaction. This includes the source commit timestamp (IO Time token), record length, database operation type (IO Type), whether this record is a before or after data image, and source table name. The GHDR also contains information that denotes each record as the beginning record of a transaction, the ending record of a transaction, a record between the beginning and end of a transaction, or a single record transaction. Data Area The variable length data area contains the atomic data for the source database operation. This includes the column index number and data changed for an after image, or original column data for a before image. GGS Tokens GoldenGate Tokens written for each transaction record include the source database row id, source database change number (LOGCSN), and source database transaction id (TRANID). User Tokens User Tokens are data defined by the user and added to the Trail via the Extract @TOKEN column conversion function. Now that we know the basic layout for an OGG Trail, lets see what information we can view via Logdump. Replication Technology Evolution for ATLAS Data Workshop

LogDump ./logdump Open /trail_file_full_path count Standard utility that comes with Oracle GoldenGate Enables to the user to view and scan for transactions and records Obtains transaction statistics and save data contained within OGG Trails Identify missing transactions Troubleshoot replication issues Getting Started With Logdump Logdump is run from a terminal window. When the application starts, a banner is displayed showing the version and copyright information, followed by the Logdump prompt. The command open [trail] will create a shared read file handler to the designated trail file. The command count will return data for all records contain in the trail. This data includes, total data bytes, average data bytes per record, totals for each type of database operation, the number of before and after image records, average number of transactions in the Trail, average data bytes per transaction, and average number of records per transaction For the picture This is great as an overview, but it does not provide any level of real detail about the Extract’s workload. The command detail on tells Logdump to display the totals and averages shown above and then sort the data and display it for each unique table contained within the Trail. “next” Replication Technology Evolution for ATLAS Data Workshop

THANK YOU! lorena.lobato@cern.ch Replication Technology Evolution for ATLAS Data Workshop