Presentation on theme: "Tips & Tricks for Troubleshooting Geodatabase Issues at ArcGIS 10"— Presentation transcript:
1 Tips & Tricks for Troubleshooting Geodatabase Issues at ArcGIS 10 July 13th, 2011Tips & Tricks for Troubleshooting Geodatabase Issues at ArcGIS 10Chet DobbinsNana Y. DeiWelcome & introductions.
2 Topics: Upgrading the geodatabase (v10.0) Compressing with Replicas SDEINTERCEPTAdditional Troubleshooting Resources…how do I upgrade……we can’t compress…One of the main topics that we wanted to present today was upgrading the geodatabase to version 10.- How many here have already performed this task?After we finish with the upgrading portion we will discuss compressing with replicas, and provide some insight into the SDEINTERCEPT file which can be used for troubleshooting performance and connection issues.If there are no questions at this point we can get started…
4 Enterprise Geodatabase Upgrade - ArcGIS 10.0 As many of you may know, or have heard, upgrading an enterprise geodatabase with ArcGIS 10 is no longer performed using the ArcSDE post installation tools. Starting with ArcGIS 10 and beyond, enterprise geodatabases will be upgraded using tools provided with ArcGIS desktop. This process will be very similar to what has been available for personal & file geodatabases.
5 Before Upgrading… Backup DBMS and configuration files DBMS full backup & configuration filesDBTUNE, DBINIT, GIOMGRInstall an ArcGIS 10 clientUpgrade integrated into clientBackwards compatibleRequirementDirect connectProper credentialsFrom a support perspective we often see that upgrades are performed on a productions system and when the process fails, or results in an unwanted behavior, you must restore the old database from a backup. It is not possible to downgrade a geodatabase. In order to ensure that you are protected, it is best to be sure that you have backed up the database and any configuration files that were used to set up the system.Since the ArcSDE post install is no longer used for the upgrade, it will be necessary to have an ArcGIS Desktop 10 installed to perform this operation.Finally, be sure that you are using a direct connection with the proper credentials for upgrading. These can all be found in the online help or in the installation specific to the database management system being used.
6 Prerequisite Checks No active user connections InstallArcGIS 10ClientNo active user connectionsCorrect user to perform upgradeRequired privileges for upgradeDBMS XML supportGeodatabase schema integrityGeodatabase configuration checkST shape libraryMultiple user schema geodatabasesApplicationServer?UpgradeArcSDEApplicationyesnoPre-requisiteCheckUpgradeGeodatabaseSchemaEsri recommends using the prerequisite check prior to upgrading your geodatabase to determine if your geodatabase is ready for upgrading. If any of these prerequisites is not met, the tool logs an error in the Results window and does not upgrade the geodatabase. This saves you from starting the upgrade only to have it fail partway through due to one of these prerequisites not having been met. If any checks fail, you must correct the problem and restart the upgrade process.For enterprise geodatabases the prerequisite check will determine whether there are any other active connections to the geodatabase, as well as whether the connected user has the appropriate permissions required to perform the upgrade.XML is used for some data storage with version 10, and this process will also check for if the database is enabled for this support.The integrity of the geodatabase is checked and this will report warnings about feature classes and other geodatabase objects which may be orphaned or other similar issues.PostUpgradeConfiguration
7 Geodatabase Upgrade Options Two optionsUpgrade Geodatabase button in ArcCatalogUpgrade Geodatabase: GP Tool APIUpgradeGDB_management(…)Beginning with the ArcGIS 10 release, geodatabase upgrades must be performed using the Upgrade Geodatabase tool or Python script. Do not use the ArcSDE Post Installation wizard on Windows or the sdesetup command to upgrade the geodatabase. If you do, you will receive a message indicating the geodatabase already exists and the upgrade will not run.
8 Upgrade Resources ArcSDE Install Guides KB Articles Online Help -installation-guidesKB ArticlesOnline HelpArcSDE Geodatabase User ForumHere are a few additional resources that can be used when you begin preparing to upgrade to ArcGIS 10.
9 Compressing with Replicas Next we wanted to share some tips on getting the most out of a compress when replication has been implemented in an enterprise environment.Replication uses hidden internal system versions to manage the synchronization process between replicas. Understanding the details of how these system versions are used by ArcGIS is not necessary to successfully implement geodatabase replication. However, these versions can have an effect when attempting to compress an ArcSDE geodatabase.
10 Types of Replicas Check out/ Check in Parent Child One-way Parent Once onlyCheck out/ Check inParentChildMultiple timesOne-wayORParentChildTo get started it will be good to know what types of replication are available.The three types of geodatabase replication are: check out/check in, one way, and two way.For all types, data from an ArcSDE geodatabase must be used as the source for replica creation.Multiple timesTwo-wayParentChild
11 Compress with a One-way Replica Default1234123Project 1Replica 1123The first example that we are going to show is versioned editing and compress with a one-way replica.In this example Replica1 is created with the default version as the replica version.An editor begins editing and moves parcel 3 in the DEFAULT version.Another editor adds parcel 4 in the Project 1 version.And then a reconcile and post is performed between Project 1 and DEFAULT.If compress is called at this point the two edits referenced by each version cannot move to the base table. The reason is because the edits are not referenced by the Replica1 system version.When we synchronize Replica1 and changes are sent from this geodatabase to the relative replica geodatabase, the process will send the two edits to the relative replica. The relative replica should then send an acknowledgement message which automatically removes the existing system version, and a new one is created from Default.A compress executed at this point will be much more effective since the edits are referenced by all versions, and so that will be applied to the base tables, and removed from the delta tables.3434COMPRESS
12 Compress with a Two-way Replica ParentChildDefault123Default123434Replica 2Replica 2The next example starts the same way, except that in this case a two-way replica is created with the default version as the replica version. The replica creation process creates a system version from default for Replica2 in the parent geodatabase as well as the child geodatabase.An editor begins editing and moves parcel 3 in the DEFAULT version of the parent replica geodatabase.Another editor adds parcel 4 in the DEFAULT version of the child replica geodatabaseNext Replica2 is synchronized where changes are sent from the parent replica to the child replica, and the changes are acknowledged. Since an acknowledgement was received by the parent, the initial system version is deleted and a new one created. With the old system version removed, a compress called at this point on the parent replica geodatabase would remove the parcel 3 edit from the delta tables and apply it to the base table.In the child geodatabase the parcel 3 edit is applied to the replica version (DEFAULT), and the initial system version is left unchanged. Therefore a compress on the child will result in no changes removed from the delta tables.Replica2 is synchronized again, and changes are now pushed from the child replica to the parent replica. Again the changes are acknowledged, and this time the system version in the child geodatabase is deleted and a new one is created from default. This allows a compress on the child geodatabase to move the edits of parcels 3 & 4 from the delta tables to the base tables.A compress on the parent would still leave the edit to parcel 4 in the delta tables. In order to move the edit of parcel 4 to the base tables on the parent, another synchronize from parent to child needs to occur.COMPRESSCOMPRESS
13 Best Practices Remove unnecessary replicas Send changes regularly Reconcile & Post named versions before synchronizingFull compress is ideal but not always possibleLook to achieve an effective compress
14 SDEINTERCEPT TracingSometimes when troubleshooting certain connection and performance issues, you may need to dig deeper into the communication between the ArcGIS client application and the ArcSDE geodatabase. A SDEINTERCEPT file contains all the calls to ArcSDE along with their return values.
15 SDEINTERCEPT Built-in ArcSDE functionality Logs ArcSDE client - server callsHelps diagnose connection & performance issuesEnable SDEINTERCEPT by setting the following environment variables:set SDEINTERCEPT=crwTfset SDEINTERCEPTLOC=C:\temp\sde_interceptFlagDescriptioncIntercept the API command namerIntercept the Channel broadcasts read-onlywIntercept the Channel broadcasts write-onlytIntercept log time (minute:second)TIntercept log time (hour:minute:second)fIntercept flush immediateThe SDEINTERCEPT trace can be enabled on either the client or the server. To instruct ArcGIS to log information, and where to write the logs, you need to set certain system environment variables.SDEINTERCEPT says what to log.SDEINTERCEPTLOC says where to log.Client side: enabled by setting the environment variables before running the application.Server side (three-tier connections only): enable by adding definitions for the two environment variables to the SDEHOME\etc\dbinit.sde and restarting the ArcSDE service. Caution: this will log all commands for all connected users. Do not leave this on, because there is some overhead in processing time and disk space
16 Queried tables & columns SDEINTERCEPT Output========================================[W 14:30:54.582] Command: QueryWithInfo[W 14:30:54.582] Long:[W 14:30:54.582] Query Info:Num Columns: 1Columns: "Shape"SQL_Construct: Tables: "scratch.MAP.STATES"WhereClause: "STATE_NAME LIKE 'C%'"Query Type: 4ByClause: <null>Num Hints: 0Num Parameter markers: 0Logfile: <null>[R 14:30:54.582] Long:[R 14:30:54.582] Col_Defines: Command nameQueried tables & columnsWHERE & ORDER BYClauses, query typeThe SDEINTERCEPT log file includes commands and responses from ArcSDE server. Each item has a time stamp, preceded by and “R” (read) and “W” (write) to indicate the direction of the information flow.Breaking down the QueryWithInfo command, it consists of these parts.The command name.The table and columns to include in the query.Any WHERE or ORDER BY clauses to apply, if any.The query type, which helps the ArcSDE server determine how to optimize the query. The query type designates whether the business table, feature table or spatial index tables should be joined, and whether the attribute or spatial portion of the query should be executed first. For more information, see the documentation on the C-API function SE_queryinfo_set_query_type.The response, describing the columns that will be returned when the query is executed later. This information is needed by the client to establish appropriate storage for receiving the data from each column. Success from this command simply means that it was understood by the ArcSDE server. It does not mean that the query will ultimately succeed.Response: Descriptionof columns returned
18 Monitoring Tools SDEINTERCEPT SDETRACE ExecuteSQL Database SQL Traces Monitors commands generated by the ArcSDE client librarySDETRACELogs the calls from the client application to the ArcSDE C-APIExecuteSQLControl database SQL Traces from the ArcGIS clientDatabase SQL TracesRecords SQL received by the database engine, and what processing occurred in response to each request.SDEINTERCEPTWhen enabled, the ArcSDE built-in capability called SDEINTERCEPT monitors the commands that are generated by the ArcSDE client library for processing by the ArcSDE server.These commands are not SQL. They are database-agnostic commands which correspond to specific requests to ArcSDE from the client.SDEINTERCEPT records these commands, translating into human-readable form and placing in a text file.SDETRACElogs the calls from the client application to the ArcSDE C-API, providing a view of the interface between the client application code and the C-API.The log file contains the name of each function in the order called, the time the call was made and when it completed.This is useful to developers building applications and profiling its operation, or others familiar with the ArcSDE C-API who are diagnosing performance problems.ExecuteSQLThis command may be used to executeSQL against a connected database. It uses the ArcObjects IWorkspace.ExecuteSQL method. It is very useful in starting DBMS tracing from ArcMap.Database SQL Traces- Commonly referred to as SQL traces, the capability to monitor the command execution of one or more session (connection) is provided by the database vendors.Oracle: SQL traces ->TKPROF, or Enterprise ManagerSQL Server: The SQL Server ProfilerPostgreSQL: pglog -> pgfouineInformix: sqlidebugMajor uses:Show the SQL that was received by the database engine, and what processing occurred in response to each request.Useful for looking at raw database errors before they are returned to ArcSDE.Can trace some database events not directly caused by SQL commands can be logged (e.g. connections, DLL loading)The level of detail of the trace is configurable.
19 Diagnostic Tools ArcSDE Error Messages SDEVERBOSE ArcSDE errors/messages received on the client or in the logsSDEVERBOSELogs errors/messages to sde_<instance>.logGeodatabase Toolset (GDBT)Monitoring, investigating and reportingReplica Activity LogsRecords information for replica creation / synchronizationsdegdbrepairIdentifies and repairs any inconsistenciesArcSDE Error MessagesArcSDE errors/messages received on the client or in the logs.Error messages you see when working with ArcSDE geodatabases can come from the underlying database management system, ArcSDE, or the client application. The following documentation link provides a list of some of the messages you might see returned from ArcSDE in the client interface or log files.SDEVERBOSESDEVERBOSE reports internal messaging to the screen upon startup and writes gsrvr process startup and shutdown messages to sde_<instance>.log.set SDEVERBOSE=TRUE is added to your dbinit.sde file in the SDEHOME\etc directory and should be removed when not in use.The sde_<instance>.log is truncated each time the ArcSDE service is started.How this can be used when troubleshooting a performance issue:Spatial queries executed by each client are written to the ArcSDE instance's sde_<instance>.log. This allows the administrator or user to extract the SQL statement and execute the same query using SQL*Plus, which can help identify performance or resource contention issues.FAQ: What is the meaning of the 'Search SQL:' statement in the ArcSDE error log with Oracle?Geodatabase Toolset (GDBT) for ArcCatalogTools that can assist you in monitoring, investigating and reporting the performance of you multiuser geodatabase.Works with geodatabases stored on SQL Server, Oracle, Informix, DB2, or PostgreSQL databases.Information provided through using the toolset:The total number of rows in the add and delete tablesUsers connected to the geodatabase and locked schemasA graphic of the state tree lineageParent-child relationships between versions for version managementDetailed information of a feature class spatial indexTable and index statistics for geodatabases stored in an Oracle databaseCurrently available for 9.3 back to 9.1NOTE: For viewing purposes only. Do not update the repository with the GDB Tools.Replica Activity LogsRecords information for replica creation / synchronization:ERRORS — How many errors occurred while running a processWARNINGS — How many warnings occurred while running a processOperation Name — The name of the process that was runTime Completed — The date and time when the process finishedOperation Info — General information about the processThe log file is called ReplicaLog.dat and is found in the temp directory of the machine on which the operation was executed. If using ArcGIS Server, the information is stored in the Server Activity Log and can be accessed through the ArcGIS Server Manager.HowTo: Get a formatted view of the contents of the ReplicaLog.dat fileSDEGDBREPAIRWhen to use?The sdegdbrepair command is used to identify and repair any inconsistencies between the adds (A), the deletes (D) tables and versioning system tables of a versioned geodatabase.The diagnose_tables and diagnose_metadata operations are used to determine if your database contains orphaned or duplicate rows in the delta tables or versioned business table.The repair_tables and repair_metadata operations are used repair any inconsistencies identified in the delta tables or versioned business table.Results are returned on screen but are also written to the sde_repair.log located in the SDEHOME\etc directory.
20 Online Resources ArcGIS Resource Center ArcGIS Web-based Help Forums ArcGIS Web-based HelpForumsBlogs