Bryan Dickerson, Woolpert Jim Moening, ESRI

Bryan Dickerson, Woolpert Jim Moening, ESRI
If You Want to Have your Database Sing on Key, Then You Need to Get It Tuned Bryan Dickerson, Woolpert Jim Moening, ESRI

Database Tuning Water/Wastewater DB Design Conference February 2005
Agenda How Does ArcSDE Work? Database Tuning (Oracle and SQL) Best Practices for Geodatabase Design ArcSDE Management Roles & Responsibilities Database Tuning Water/Wastewater DB Design Conference February 2005

How Does ArcSDE Work? Database Tuning Water/Wastewater DB Design Conference February 2005

ArcSDE Data Storage A feature class (layer) is comprised of 3 tables in an ArcSDE database Business table Feature table Spatial index table Database Tuning Water/Wastewater DB Design Conference February 2005

Spatial Index table (S1)
Business Table Contains attributes and a spatial column Spatial column is a key used to relate to the feature and spatial index tables SHAPE OBJECTID NAME … 1 Main St Business table (Roads) FID … AREA LEN POINTS 1 926.45 Feature table (F1) Spatial Index table (S1) SP_FID GX GY EMINX EMINY EMAXX EMAXY 1 7 12 786 840 339 374 Database Tuning Water/Wastewater DB Design Conference February 2005

Feature Table Stores geometry including annotation and CAD features Also stores attributes describing the geometry such as Area and Length SHAPE OBJECTID NAME … 1 Main St Business table (Roads) FID … AREA LEN POINTS 1 926.45 Feature table (F1) SP_FID GX GY EMINX EMINY EMAXX EMAXY 1 7 12 786 840 339 374 Spatial Index table (S1) Database Tuning Water/Wastewater DB Design Conference February 2005

Defines the grid range for each feature Defines the shape envelope for each feature SHAPE OBJECTID NAME … 1 Main St Business table (Roads) FID … AREA LEN POINTS 1 926.45 Feature table (F1) SP_FID GX GY EMINX EMINY EMAXX EMAXY 1 7 12 786 840 339 374 Spatial Index table (S1) Database Tuning Water/Wastewater DB Design Conference February 2005

ArcSDE Storage Model Coordinate Information CENSUS_TRACTS SHAPE Spatial column 1 Business table 2 3 2 1 F34 2 FID POINTS 3 1 xy,xy,xy S34 1 2 SP_FID xy,xy,xy GX GY 3 1 xy,xy,xy 1 1 2 1 1 3 1 1 Feature table stores geometry Spatial index table stores grid tiles and envelopes Tables join on integer Feature ID column 2 1 2 ArcSDE binary storage Feature classes are stored in tables in the RDBMS. The business table, contains the attributes of the feature class. Each row in the business table represents one spatial feature (single or multi-part), with a unique shape identifier. The unique shape identifier is stored in the geometry column of the B table. The name of the business table is the feature class name. Each record is related to a corresponding row in the feature table, or F table, through the unique feature identifier. The F table physically stores the geometric shape in a binary column. There is a one to one relationship between the business table and the F table. The final table stored in the database for the layer is the spatial index table, or S table. The S table stores the grid tiles and layer envelopes for the layer. Only grid tiles that reference a feature envelope have an entry in the table. Feature envelopes that cross a tile are stored with the grid. The feature identifiers are also stored within the S table to provide a one to many relationship from the business or F table to the S table. The rows in the business, F and S tables are connected through a unique feature identifier. ArcSDE will maintain the integrity of the feature class tables when features are added, deleted or modified. Optionally, related attribute tables could be stored in the database. These tables would be linked to the business table through a key. Each feature class loaded into the database is given a unique ID by ArcSDE. This ID is used to identify which F and S tables relate to the business table. 3 1 2 1 2 1 2 2 1 Database Tuning Water/Wastewater DB Design Conference February 2005

Spatial index Spatial index is used to improve performance of data retrieval IBM DB2, Oracle, and SQL Server use grid tiles Goal Optimize the size of the grid tiles Balance Number of features referenced by each tile (features per tiles) Number of tiles referencing each feature (tiles per feature) Informix and Oracle Spatial use R-tree indexing R-tree index constructs the index using the feature envelope Do not need to add any parameters for building R-tree (built automatically) Spatial index Use the spatial index to improve query performance. Oracle, SQL Server and IBM DB2 These databases use types of grid tile spatial indexes (discussed in detail in Lesson 3). When creating a new feature class you must identify the grid tile size dimensions so the database can construct the spatial index. You must optimize the size of the grid tile by balancing the number of tiles per feature and the number of features per tile. Informix and Oracle Spatial Informix and optionally Oracle Spatial, use the R-TREE index schema (discussed in Lesson 3). This type of index does not require any user defined parameters for construction. The R-tree index is built by the database using the feature envelopes. Database Tuning Water/Wastewater DB Design Conference February 2005

Reducing Work With a Spatial Index
Clients use a spatial filter to reduce query results and speed up data retrieval Example: Only fetch visible features for display ArcSDE uses spatial index to reduce I/O Eliminates need for full table scan Same principle as RDBMS column index Spatial filter ArcSDE clients request information the server. The action of retrieving rows from database files may result in high disk I/O if the files are not cached in the server memory. Clients can apply filters on data requests to reduce the amount of data searched and returned by the server. Attribute filters are applied in the WHERE clause of the SQL statement and passed to the RDBMS. Spatial filters are used by the server to retrieve information based on a spatial condition. WHERE clause An attribute or spatial filter limits the number of rows returned by the SELECT statement, by identifying the rows to return to the client. Attribute filters use the attributes of the records to retrieve the results, while spatial filters use the spatial properties of the features to return the results. When performing an attribute filter, the WHERE clause of the SELECT statement is used to define the criteria. Attribute filters can use conditions in related tables to retrieve the selected features. Within the WHERE clause tables are joined based on a common item allowing attributes to be stored in related tables instead of entirely on the ArcSDE business table Spatial filter Retrieving features based on a spatial condition is performed by applying a spatial filter to the SELECT statement. The server builds a WHERE clause in the statement which applies the spatial filter to the rows in the feature table. The database administrator builds indexes on attribute columns to speed up attribute queries, while ArcSDE builds a spatial index to speed up spatial queries. Spatial index ArcSDE builds a spatial index on feature classes to reduce disk I/O and to speed up spatial queries. The spatial index avoids a full table scan by reducing the number of rows evaluated against the spatial constraint. Database Tuning Water/Wastewater DB Design Conference February 2005

Spatial Index Spatial index uses the envelope (bounding box) of the features for fast searching 38,68 53,28 49.5,53.6 Spatial index Depending on the RDBMS being used the spatial index will be created differently. The goal of any spatial index is to facilitate fast data retrieval. Linear (one dimensional) indexing methods are not effective for feature data. Features have two-dimensions so the spatial index must have be two dimensional. R-tree structure R-tree spatial indexes arrange the pages of the index in a hierarchy. The features are grouped into subsequent branches of the tree, until the actual feature is found in the bottom-most page (leaf). Variations of the R-tree index are used by Oracle Spatial and Informix. Grid tile structure Some spatial indexes use a series of grid tiles to reference features. The spatial extent of the feature class is divided into regular grid tiles. The spatial index is an ordered list of the grid tiles with each feature referenced by one or many grid tiles depending on the feature envelope. Variations of this structure are used by Oracle, SQL Server and IBM DB2. Feature envelope One of the most important aspects of the spatial index is the feature envelope (minimum bounding box surrounding a shape). This information is used by the RDBMS to quickly search through the coordinates to find those shape envelopes that fall within the spatial filter returned from the client. 49,25 32,64 Envelopes for area and line features use min x,y and max x,y coordinates. The envelope for a point is the actual x,y coordinate. Database Tuning Water/Wastewater DB Design Conference February 2005

Grid Tile Spatial Index Components
49 Parcels Example 48 73 74 75 76 77 Grid tiles Two numbers in indexed columns Shape envelope Two coordinates (four numbers) in indexed columns Shape Many coordinates in binary column The grid tile spatial index structure is made up of grid tiles, the envelopes of the features and the feature ID. As spatial queries are processed the spatial index reduces the number of features which will be evaluated in the final stage of the query and returned to the client. Grid Tiles The grid tiles act as fictional squares across the spatial extent of the layer. The grid tiles are the first stage of the spatial search, and are used for the initial feature selection. The grid tiles are created when the data is loaded and can be modified at any time. Shape envelope Every feature in a layer has an envelope, which is the lower left and upper right coordinate of the minimum bounding box of the shape. Feature envelopes are organized in the table according to the grid tile that the envelope crosses. The shape envelope is used in the second stage of the search to further reduce the number of features selected. Shape The feature ID is stored with the grid tiles and shape envelopes because every spatial query will eventually perform a binary comparison on the shape. The actual shape is stored in a binary column called a BLOB (Binary Large Object), which is not part of the spatial index. The feature ID is used to quickly find the shape to send back to the client. The grid tiles and envelopes reduce the number of rows which are retrieved for the shape comparison stage. A Database Tuning Water/Wastewater DB Design Conference February 2005

How the Grid Tile Spatial Index Works
Goal: Avoid reading shapes for entire layer (full table scan) Four stages of elimination The goal of the spatial index is to reduce the number of shape on shape comparisons performed by ArcSDE and speed up data retrieval. An ArcSDE client applies a spatial filter to a request, and the server must process the rows in the feature class to determine which shapes meet the spatial constraint. With a three-tier connection, the ArcSDE server (gsrvr) performs stages 3 and 4. With a two-tier connection, the gsrvr functionality is built into the client, therefore the client performs stages 3 and 4. GRID TILE SEARCH The client or the ArcSDE server (gsrvr) receives the spatial filter and determines the grid tile(s) that the the filter crosses and the envelope (lower left/upper right corner) of the filter. These values are passed to the RDBMS in the SELECT statement. First stage: The RDBMS receives the SQL statement from ArcSDE, which contains the grid tile and envelope coordinates. The WHERE clause defines the grid tiles to select in the spatial index. Once the grid tiles are located in the spatial index table the envelope search commences on the envelopes associated with the selected grids. Second stage: Using the filter envelope and the shape envelopes in the spatial index table, the RDBMS reduces the initial selected set. The filter envelope and the shape envelopes are compared by the RDBMS for overlap. If there is overlap the shape is selected for the shape comparison stage. Third stage: The shape of the spatial filter is compared to the features’ envelopes from Stage 2 by the ArcSDE server or the client. If the features’ envelopes do not overlap the shape of the filter then these records are not used in the final stage. Fourth stage: The shape on shape comparison is a binary comparison of the filter shape and feature shape. The actual binary shapes are compared to the filter’s shape for overlap by ArcSDE or the client. The resulting records are returned to the client. FILTER (SELECTION SHAPE) LAYER 1. tile tile RDBMS envelope envelope 2. shape envelope ArcSDE server or client 3. shape shape 4. Database Tuning Water/Wastewater DB Design Conference February 2005

Selecting a Spatial Index Size
Size tiles based on average feature envelope Grid tile size too small S-table is too big Many tiles referencing one feature Grid 3x feature envelope S-table smaller Fewer tiles referencing one feature Spatial searches using grid tile index ArcSDE performs a spatial search when the ArcSDE client uses a spatial filter, which acts to limit the number of features the server returns. A spatial search uses the spatial index that the RDBMS builds using the the grid tile size given during loading, to speed feature retrieval. The database uses the grid tile dimension to build the spatial index. Any feature envelope that intersects a grid tile is referenced by that grid. The grid tiles which cross the the client’s spatial filter are used by the RDBMS to search the spatial index for the matching grid tiles. Once the tiles are found the feature envelopes, referenced with the selected grids, are compared against the envelope of the spatial filter. This second stage eliminates features from the initial tile selection. The selected features, from these two steps, are retrieved and returned to the client or gsrvr for the final shape comparison stages. Grid tile size The number of features ArcSDE processes for any spatial search is static. The envelope search will always return the same number of features to ArcSDE for a spatial search, regardless of the tile size. However, the envelope comparison (stage 2) is influenced by the tile size. A very large tile size, compared to the feature envelope size, results in many feature envelopes contained within each tile. The first stage will select the grids and then the second stage will have to process all the feature envelopes within those grids, potentially slowing query time. A small tile size compared to the feature envelope size, results in a large S table, with a large number of grids containing the same feature envelope. This may make the grid tile search slower, as the RDBMS has more records to search and select. Therefore, when determining the spatial index tile size you must balance these two competing factors: the number of features in a tile and the number of records in the index (tiles per feature). The size of 3x the average feature envelope is good for an initial parameter. The grid tile size can be changed at any time. Many rows take longer to search Many tiles referencing one feature inefficient Balance tiles per feature against feature per tile Database Tuning Water/Wastewater DB Design Conference February 2005

ArcSDE Command Line Tools Are Your Friend
sdelayer -o si_stats -l Parcels,shape -i esri_sde -s jayhawk -D demo ArcSDE Build 967 Thu Feb 28 22:31:11 PST 2002 Layer Administration Utility Layer 6 Spatial Index Statistics: Level 1, Grid Size | | | Grid Records: | | Feature Records: | | Grids/Feature Ratio: | | Avg. Features per Grid: | | Max. Features per Grid: | | % of Features Wholly Inside 1 Grid: | | Spatial Index Record Count By Group | | Grids: <=4 >4 >10 >25 >50 >100 >250 >500 | | | | Features: | | % Total: % 1% 0% 0% 0% 0% 0% 0%| Database Tuning Water/Wastewater DB Design Conference February 2005

Spatial Index Tile Size Considerations
Optimal size varies by average feature envelope Recalculate index if feature class changes Most feature envelopes should not exceed grid size S-table becomes too large Features are referenced by many tiles which is inefficient Use multiple resolutions (multiple grids) for feature classes with highly-variable feature sizes Spatial index size considerations Determining the grid tile size is not an exact science, many times trial and effort will result in the best tile size. There are some general guidelines which are used to determine an initial grid size, which can be used to further pinpoint the optimal grid size. The grid size can be recalculated at any time, forcing the RDBMS to rebuild the S table. The grid tile size should not be smaller than the average feature envelope size because then the spatial index will reference feature envelopes in multiple grid tiles. It is impossible to avoid have a feature envelope “split” by a grid tile but if you make the grid tile large enough an average feature should be referenced by four tiles of less. If the tile size is small, the spatial index becomes large and the searching for grid tiles may be lengthy. If the feature envelope size is highly variable, an option would be to create multiple grid sizes. ArcSDE provides the ability to have up to 3 grid tile levels. Each feature envelope is contained within one level, which reduces the occurrence of many tiles per envelope. The RDBMS must search the S table three times, on grid tile level at a time, which may be time consuming. Default grid tile size When loading with ArcCatalog a default spatial index tile size is calculated if none is supplied. ArcCatalog calculates this tile size using a sample of all the features in the layer. The system samples the features and determines the tile size based on the feature shape and envelope dimensions. Database Tuning Water/Wastewater DB Design Conference February 2005

Making Data Editable - Register As Versioned
Performed by data owner in ArcCatalog Two tables are added to feature class Additions table Deletions table Making data editable Once loaded into a DBMS, ArcSDE feature classes become read-only by default. In order to make these feature classes editable their owner must use ArcCatalog to “Register them as Versioned”. Versioning is a topic of its own, and you will learn about it in Lesson 7. For now all you need to know is that making data editable is a byproduct of making it multi-versioned. A and D tables In order to handle multiple editing transactions coming in from different users, a feature class needs two additional tables: an additions (A) and a deletions (D) table. The additions and the deletions tables sometimes are referred to as delta tables after the Greek letter “delta” commonly used by scientists to stand for a change. The delta tables are needed for keeping track of every modification introduced in an editing session. They also come in handy when one needs to undo and redo recent changes. In essence an editing session in ArcMap is analogous to editing a text file using a modern word processor. Word processors remember recent keystrokes and allow the writer to undo actions. However, once saved, the user may not undo any changes made to the file. Database Tuning Water/Wastewater DB Design Conference February 2005

How Tables React to Edits
Business table SHAPE ID Name 1 Roma 2 Palm Deletes table Adds table SHAPE Name ID Citrus 3 Inserting Adds table SHAPE Name ID Citrus 3 Deletes table 2 Deleting This slide demonstrates how delta tables change if a user inserts, updates or deletes a feature. Inserting In the case of an insert operation a new row is appended to the additions table (not to the business table). A new feature with an object ID of 3 appears in the additions table after the insert operation. Deleting Object IDs of deleted object are recorded in the deletions table. In the example feature 3 is marked for deletion. Client applications read business, add and delete tables to display a feature class in its entirety. It is possible to migrate the features from the delta table to the business table. Such an operation is called compressing the database and can only be performed by the ArcSDE administrator. Updating An update is a two-step procedure. First the feature is deleted, and then recreated with new characteristics. In the example, the feature with an object ID of 1 is entered to the deletions table and also into the additions table. Notice how shape of feature 1 is changed. An update generates a new row in each of the delta tables. If one feature is updated multiple times in one editing session (e.g. a marker is moved several times), then the additions and the deletions tables grow accordingly. As soon as the user saves, all intermediate savepoints are removed and the database only keeps the final copy of the feature. The delta tables shrink accordingly. Updating Adds table SHAPE Name ID Citrus 3 1 Deletes table 2 Roma Database Tuning Water/Wastewater DB Design Conference February 2005

Raster Data Storage Raster data is stored in a similar fashion to vector data 4 tables are used to reference a raster feature class Business table Raster bands table (stores bands for each image) Raster auxiliary table (stores metadata for each raster band) Raster blocks table (stores the actual raster in a tiled format) Blocks table is most important as it is by far the largest Database Tuning Water/Wastewater DB Design Conference February 2005

Pyramids Pyramids can be built for a raster layer when loaded into ArcSDE Multiple resolutions of the raster layer are stored in the database At a zoomed out scale the lower resolution is displayed As the user zooms in to the highest level of detail, the raster as it was originally loaded (its highest resolution) is shown This significantly helps to speed display of raster layers in ArcGIS and ArcIMS Database Tuning Water/Wastewater DB Design Conference February 2005

Raster Blocks Table (SDE_BLK_1)
Raster is divided into tiles containing many pixels and each tile is a row in the table Each tile at each pyramid level has a row in the table This table WILL get very large! RASTERBAND_ID RRD_FACTOR ROW_NBR COL_NBR BLOCK_DATA 1 2 Database Tuning Water/Wastewater DB Design Conference February 2005

Database Tuning for Oracle and SQL
Database Tuning Water/Wastewater DB Design Conference February 2005

Query Response Time When the client submits a query, the server must: Analyze query statement Fetch data Transfer results to client over network 2 biggest performance factors are network and disk I/O Network I/O is database independent Scanning large tables increases disk I/O Fetching and transmitting large results increases disk and network I/O Client When an action is performed by the client a request is sent via the network to the server. The request is translated into a SQL statement by either ArcSDE or the client. Once the rows are retrieved by the RDBMS, they are transferred back to the client (potentially through ArcSDE). RDBMS The ArcSDE server sends the SQL statement to the RDBMS to be processed. The RDBMS parses the statement and verifies that the tables exist. Parsing the statement determines the best method of retrieving the data from the database. Once a statement is parsed the RDBMS fetches the data from the database. If the table is frequently accessed, or has been recently accessed, rows may already be in the RDBMS’ memory cache, reducing the amount of time spent retrieving rows from the disk. The RDBMS transfers data to the client. Spatial data is translated into a usable format and then sent in buffers (up to MAXBUFSIZE) across the network to the client. Query Time Searching files on disk may result in a longer total query time for the client request. It is important to minimize the time spent querying the database and reduce the amount of data transferred across the network. ArcSDE clients can reduce the amount of data returned by applying an attribute or spatial filter on the request. This will reduce the total amount of data returned. The spatial index is used to reduce the query time for spatial information. Spatial filters and the spatial index are discussed in more detail in this lesson. Database Tuning Water/Wastewater DB Design Conference February 2005

Network I/O T1 line connecting a remote office to an ArcSDE database in a headquarters office can be tied up with one ArcGIS desktop performing normal operations Solution to this is terminal emulation software such as Citrix or Terminal Server Database Tuning Water/Wastewater DB Design Conference February 2005

SQL Server Basics Terminology is different than that for Oracle A SQL Server instance is made up of many databases SQL Server databases are a logical container for objects such as: Users Roles Tables Etc… Database Tuning Water/Wastewater DB Design Conference February 2005

SQL Server Database A SQL Server database is comprised of data files and log files A database has, at minimum, one data file and one log file Database Log file Data file Database Tuning Water/Wastewater DB Design Conference February 2005

SQL Server File Groups There can be multiple data files and log files and these can be stored in different places A group of data files and/or log files is called a file group Database File group Log file Log file Data file Data file Database Tuning Water/Wastewater DB Design Conference February 2005

Oracle Basics Terminology is different than that for SQL Server An Oracle database is made up of many table spaces A table space can be made up of many data files. Also need to be aware of redo log files and control files Table spaces are logical containers for objects such as: Tables Stored procedures Roles and users are stored at the database level and not the table space level Database Tuning Water/Wastewater DB Design Conference February 2005

DBTune File/Table Actually exists as a table in the SDE database or table space Contains storage parameters for tables and feature classes Whenever a new feature class is created and data is loaded, the ArcSDE server process reads the DBTune table for information on how to store these new objects There can be different parameters for different layers or data types and these are specified via keywords Database Tuning Water/Wastewater DB Design Conference February 2005

Storing Layers in the SDE Database (SQL) or Table Space (Oracle)
Do not store layers in the SDE database or table space Leave the SDE database or table space for ArcSDE application use (managing what layers are where, versioning, etc.) Otherwise this can significantly slow down data retrieval as this database or table space will become large Database Tuning Water/Wastewater DB Design Conference February 2005

Multiple Database Logins
Do not have all your editors in your organization use the same login Each database user has a log table in the SDE database or table space that tracks a user’s actions If all users use one login, all their actions will write to one table and this table will grow very large very fast Doing this also prohibits from restricting user access on a layer by layer basis Database Tuning Water/Wastewater DB Design Conference February 2005

Disk I/O Contention This is the number one way to slow down an ArcSDE database As users view or query data remember that, at minimum, ArcSDE is working through the business tables, feature tables, and spatial index tables for each layer in the map If the layers are versioned, ArcSDE is also working through the adds and deletes tables Database Tuning Water/Wastewater DB Design Conference February 2005

Preventing Disk I/O Contention
Use many disks and spread the tables and logs across those disks In SQL Server, make use of file groups and spread the data and log files across different physical disks In Oracle, make use of different table spaces and store the different tables (business, feature, and index) in the different table spaces on the different disks Database Tuning Water/Wastewater DB Design Conference February 2005

Example SQL Server Configuration
Operating System Pagefile ArcSDE Software Disk 1 SQL Server Software System database SDE database Business table file group Disk 3 Feature table file group Disk 2 SDE Transaction log Raster table file group Spatial index file group Disk 4 Database Tuning Water/Wastewater DB Design Conference February 2005

Example Oracle Configuration
Operating System Pagefile/Swap space ArcSDE Software Disk 1 Oracle Software Business table space Primary control files Disk 3 Feature table space Disk 2 Redo log Raster table space Spatial index table space Secondary control files Disk 4 Database Tuning Water/Wastewater DB Design Conference February 2005

Disk I/O - RAID RAID = Redundant Array of Independent (or Inexpensive) Disks Different levels of RAID RAID 0 = Striping. Data is striped across drives. Offers good performance but no fault tolerance. RAID 1 = Mirroring. Data is written or mirrored to a second disk. Simple fault tolerance and performs better than a single disk. RAID 5 = Striping w/ parity. Data is striped and parity calculations are distributed among disks. Provides good READ performance and excellent fault tolerance. RAID = Mirroring and striping. Excellent read/write performance and good fault tolerance. Database Tuning Water/Wastewater DB Design Conference February 2005

Other Things You Can Do…
Reduce amount of data processed for faster queries Provide a filter to limit number of rows Index tables and layers to reduce searching If you know users will search or query on particular fields, index those fields ahead of time. ArcSDE is fastest for small, indexed queries!!! Client When an action is performed by the client a request is sent via the network to the server. The request is translated into a SQL statement by either ArcSDE or the client. Once the rows are retrieved by the RDBMS, they are transferred back to the client (potentially through ArcSDE). RDBMS The ArcSDE server sends the SQL statement to the RDBMS to be processed. The RDBMS parses the statement and verifies that the tables exist. Parsing the statement determines the best method of retrieving the data from the database. Once a statement is parsed the RDBMS fetches the data from the database. If the table is frequently accessed, or has been recently accessed, rows may already be in the RDBMS’ memory cache, reducing the amount of time spent retrieving rows from the disk. The RDBMS transfers data to the client. Spatial data is translated into a usable format and then sent in buffers (up to MAXBUFSIZE) across the network to the client. Query Time Searching files on disk may result in a longer total query time for the client request. It is important to minimize the time spent querying the database and reduce the amount of data transferred across the network. ArcSDE clients can reduce the amount of data returned by applying an attribute or spatial filter on the request. This will reduce the total amount of data returned. The spatial index is used to reduce the query time for spatial information. Spatial filters and the spatial index are discussed in more detail in this lesson. Database Tuning Water/Wastewater DB Design Conference February 2005

ArcSDE Performance Tuning
Optimize Workflow Data Loading Data Maintenance Reconcile/Post/Compress Database Configuration Keep it simple Increase default cache sizes Editing Operations Use of edit cache is CRITICAL Index Management Regularly rebuild indexes Collect Performance Stats Establish baseline Use for performance validation Versioning Methodology Reconcile performance costs Using versioning to model history or alternatives Reconcile/post/compress procedures Database Design Cost of complex data models While Editing Recursive Relationships Feature linked annotation Types of labels Monitor Table Statistics Adds and Deletes tables SDE.state_lineages, states A good DBA is ESSENTIAL! Database Tuning Water/Wastewater DB Design Conference February 2005

Tools for Managing ArcSDE Databases
Database vendor provided tools SQL Server Enterprise Manager SQL Server Query Analyzer Oracle DBA Studio ESRI provided tools ArcCatalog ArcSDE command line tools Other tools StorageVisitron (available from ArcScripts) Version Manager ArcObjects sample (Developer Kit) Database Tuning Water/Wastewater DB Design Conference February 2005

Excellent Instructor-led Classes
ArcSDE Administration for SQL Server ArcSDE Administration for Oracle System Architecture Design for GIS Database Tuning Water/Wastewater DB Design Conference February 2005

Best Practices for Geodatabase Design
Custom features Geodatabase *ArcSDE Geodatabase (Enterprise) Seamless Data Structure Disconnected Editing* Versioning* Scalability* Multi-User Editing* Integrity: Domains, Subtypes, Relationship classes Topology Database Tuning Water/Wastewater DB Design Conference February 2005

support.esri.com – Project Center

support.esri.com – Project Center (cont.)

Database Design Database Tuning Water/Wastewater DB Design Conference February 2005

Data Design Planning Database Tuning Water/Wastewater DB Design Conference February 2005

Geodatabase Database Tuning Water/Wastewater DB Design Conference February 2005

Data Models Tips and Tricks

Steps to designing geodatabases
Conceptual Design Identity the information products to be produced with your GIS. Identify the key thematic layers based on your information requirements. Specify the scale ranges and spatial representations for each thematic layer. Group representations into datasets. Database Tuning Water/Wastewater DB Design Conference February 2005

Logical Design Define the tabular database structure and behavior for descriptive attributes. Define the spatial properties of your datasets. Propose a geodatabase design. Database Tuning Water/Wastewater DB Design Conference February 2005

Physical Design Implement, prototype, review, and refine your design. Design work flows for building and maintaining each layer. Document your design using appropriate methods. Database Tuning Water/Wastewater DB Design Conference February 2005

Best Practices The schema design phase of your geodatabase implementation is critical to the ultimate performance of your GIS system. Geodatabases can contain feature datasets, feature classes, relationship classes, geometric networks, topologies, feature-linked annotation, and other specialized abstractions. Careful consideration of the cost, limitations, and true purpose of each of these abstractions is required to avoid creating a geodatabase that may not perform well in either a single- or multi-user environment. Database Tuning Water/Wastewater DB Design Conference February 2005

Best Practices – Feature Datasets
A feature dataset is a container for feature classes that share the same spatial reference, along with relationship classes, geometric networks, and topologies. Feature classes edited together in ArcMap can be grouped into feature datasets Feature classes not typically edited together should be segregated into separate feature datasets or geodatabases Feature classes that participate in a geometric network or topology must be grouped in the same feature dataset. Feature classes can participate in no more than one topology OR geometric network. Database Tuning Water/Wastewater DB Design Conference February 2005

Best Practices – Subtypes
Subtypes let you apply a classification system within a feature class and apply behavior through rules. Subtypes are essential to good design because they help reduce the number of feature classes and improve performance. Let you control every configurable behavior: Attribute rules Relationship rules Network Connectivity Rules Topology Rules Reduce the number of feature classes by creating collections of features within a single feature class Database Tuning Water/Wastewater DB Design Conference February 2005

Best Practices – Relationship Classes
Use relationship classes for referential integrity persisted in the geodatabase, on-the-fly relates for editing performance, and joins for labeling and symbology. Messaging occurs between related objects when querying or editing these objects. This incurs overhead in the application. Composite relationship require more messaging, and thus more overhead, than simple relationships. Relationships can only be defined between objects in the same geodatabase. Still requires a join for SQL query, labeling, or symbology. When modeling spatial relationships, you will gain efficiency in your data creation and maintenance procedures by first thinking about how to use geodatabase topologies, geometric networks, shared-edge editing, and geospatial operators in your data model before resorting to relationship classes. Database Tuning Water/Wastewater DB Design Conference February 2005

Best Practices – Geometric Networks
A geometric network is a system of edges (lines) and junctions (points) that convey objects. They are a mechanism to represent connectivity between edges and junctions where they connect. Geometric networks provide fast network tracing capabilities and maintain connectivity during editing. The maintenance of the logical network adds overhead to editing and version management. The greater number of feature classes participating in a network, the greater amount of overhead will be incurred in maintaining the logical network and connectivity during editing. Database Tuning Water/Wastewater DB Design Conference February 2005

Best Practices – Topologies
A geodatabase topology lets you accurately model geometric relationships between features. Topology can be considered a special type of relationship among features. Your data model will be more efficient if you can associate features by topology rather than with a relationship class. But remember that a feature class cannot participate in a topology and a geometric network at the same time. Topologies manage a set of rules and errors associated with the violation of the rules. A geodatabase topology can incur editing overhead and work flow considerations. The greater number of feature classes and rules in a topology, then greater amount of overhead for the management of the rules and errors within a topology. Database Tuning Water/Wastewater DB Design Conference February 2005

Best Practices – Annotation and Labeling
Choosing whether to cartographically edit or automatically place text is an important decision. It depends on the quality of source data, how much effort is to be expended, and the desired map quality. Feature-linked annotation Simple annotation Labels Definition Annotation class joined in composite relationship with feature class. Standalone annotation class. Text properties, such as font and size, and a labeling field, defined in an ArcMap layer. Framework Geodatabase data model ArcMap document or layer property Typical uses Fine cartographic placement of annotation with attribute-based text. Fine cartographic placement of annotation with fixed text. When text values are attribute values that can be placed automatically on the map. Composite objects Yes No Referential integrity Pros The composite relationship lets you update, move, or delete annotation when features are edited. Simple annotation lets you place fixed text with a high degree of control. Labeling features is a quick and easy way to add descriptive text to your map. Cons Adds editing overhead; requires more work than labeling. Requires more work than labeling; text values cannot be updated automatically if attributes change. While labels will not collide with each other, cartographic placement is rough. Recommendations Use feature-linked annotation for text, such as street names, building numbers, and gages. Use simple annotation for static text or text describing large or indeterminate areas. Use labeling for quickly placing descriptive text, such as addresses or measurements. Database Tuning Water/Wastewater DB Design Conference February 2005

Best Practices – Design Tips
Build on your existing GIS designs Most existing designs are suitable for moving forward. You can build on what has worked in the past and find new capabilities than can improve efficiency. Use generic geodatabase types when feasible Combining generic data structures with rich GIS tools provide the best solutions to scale and support multiple users and applications. Leverage the ArcGIS software logic as much as possible and only use customized GIS data structures as a last resort. Integrate independent feature classes using topology Many users will find opportunities to use topologies in the geodatabase. This can create significant increases in efficiency for customization, data maintenance, and user productivity. Database Tuning Water/Wastewater DB Design Conference February 2005

Best Practices – Design Tips
Combine GIS design concepts with traditional relational database design methods Both RDBMS and GIS design methodologies are critical for good GIS design. One is not sufficient without the other. Learn to use and apply both techniques. Prototype and pilot your geodatabase design Prototyping a design using personal geodatabases, ArcMap, and ArcCatalog is a very effective option for creating an efficient design process. During the final stages of design, you will want to test scalability and work flows that represent the work your organization will perform. Use this activity to make final adjustments to your design as necessary. Database Tuning Water/Wastewater DB Design Conference February 2005

ArcSDE Management Roles & Responsibilities
RDBMS ArcSDE Database Tuning Water/Wastewater DB Design Conference February 2005

support.esri.com - Project Center

support.esri.com - Project Center (cont.)

Project Center - Skill Development

ArcSDE Management Roles & Responsibilities
The following discussion outlines the major roles and responsibilities involved in deploying your spatial data and GIS applications throughout your enterprise. Two Broad Categories for ArcSDE Roles & Responsibilities: Administration Development Database Tuning Water/Wastewater DB Design Conference February 2005

Administration Tasks Database Tuning Water/Wastewater DB Design Conference February 2005

ArcSDE Service Management
Responsibility: Manage the ArcSDE service including: Starting and stopping the service Establishing a recovery process, if the service stops Establishing a troubleshooting process Remedy an ArcSDE service that will not restart Fail-over Requirements to do the Job: Must have Administrative or Power User rights on the service host Must have access to the Windows Event Log Must have Administrative access to the DBMS to diagnose database changes or problems Must have knowledge of IT policies and methodologies Owner: IT or GIS Database Tuning Water/Wastewater DB Design Conference February 2005

Server Load Management
Responsibilities: Monitor and remedy server load Monitor server load or automate the monitoring of the server load Create framework for alerts Diagnose which processes and operations are overloading the server Remedy unnecessary load, like a runaway process Build plan(s) to overcome overloaded server Potentially build fail-over infrastructure of mission critical server Requirements to do the Job: Access to system monitoring tools (e.g. PerfMon, Task Manager) Must have Administrative or Power User rights on the host Must have knowledge of IT policies and methodologies Owner: IT or GIS Database Tuning Water/Wastewater DB Design Conference February 2005

ArcSDE Service Anomaly Management
Responsibility: Manage the ArcSDE service if unforeseen troubles occur Requirements to do the Job: Must have Administrative or Power User access to the service host Must have access to the Windows Event Log to diagnose problems Must have access to a technical support group to diagnose and remedy problems Must have Administrative access to the DBMS to diagnose database problems Must have knowledge of IT policies and methodologies Must be able to troubleshoot ArcSDE service anomalies Must be able to understand ArcSDE error logs to help resolve problems Owner: IT or GIS Notes: Timely support required and is part of the criteria for choosing the staff Database Tuning Water/Wastewater DB Design Conference February 2005

Spatial Database Access
Responsibilities: Manage access to the spatial database in ArcSDE, including: Create new users to a spatial database Maintain administration user Create a procedure to support ArcSDE/RDBMS standard security (i.e. ArcSDE does not support integrated security) Prepare and enforce procedure to manage ArcSDE clients (e.g. individual logins/roles) Lock down command line administration tools Requirements to do the Job: Must have administrative access to RDBMS to create logins Must have administrative access to RDBMS spatial databases Must have the ability to control the use of the ArcSDE admin tools (e.g. sdemon) Must know how to enforce RDBMS data access Must have knowledge of IT policies and methodologies Owner: IT or GIS Database Tuning Water/Wastewater DB Design Conference February 2005

Spatial Database Recovery
Responsibility: Establish recovery policy and procedures including: Frequency (incremental, differential) Archive location Versioning Recovery medium (i.e. backup vs. replication) Enforce recovery policy and procedures Automate recovery procedure where practical Requirements to do the Job: Must have administrative access to SQL Server to create and maintain recovery procedure Must know how to create and enforce SQL Server recovery procedures Must have knowledge of ArcSDE layer schema to recover correct database objects Must have knowledge of IT policies and methodologies Owner: IT or GIS Database Tuning Water/Wastewater DB Design Conference February 2005

Replicate Spatial Data
Responsibilities: Configure ArcSDE replication facilities (e.g. f and s tables) Establish appropriate replication for business needs (e.g. transactional, snapshot, merge) Implement replication needs Requirements to do the Job: Must have administrative access to RDBMS to create and maintain replication configurations Must know how to create and enforce RDBMS replication Must have knowledge of ArcSDE layer schema to replicate correct database objects Must have knowledge of IT policies and methodologies Owner: IT or GIS Database Tuning Water/Wastewater DB Design Conference February 2005

Tune ArcSDE Layers Responsibilities: Configure a spatial index for a given layer Measure and monitor the performance of the spatial searches critical to the business needs of the organization (i.e. data maintenance, custom products, general products) Maintain performance and Quality of Service as new spatial applications arrive, data and maintenance grows, and the user base increases Requirements to do the Job: Must know how to adjust a spatial index of an ArcSDE layer Must know how to measure the performance of a spatial search in a given application Must know Quality of Service requirements Owner: IT or GIS Database Tuning Water/Wastewater DB Design Conference February 2005

Tune Server and Network Infrastructure
Responsibilities: Configure the ArcSDE instance parameters for the ArcSDE server Measure and monitor the performance of the ArcSDE server instance, and the anticipated needs (i.e. data maintenance, custom products, general products, usage growth) maintain performance and quality of service as new users and applications arrive, data and maintenance grows, and the user base increases Requirements to do the Job: Must know Quality of Service requirements Owner: IT or GIS Database Tuning Water/Wastewater DB Design Conference February 2005

ArcSDE Upgrade Management
Responsibilities: Ensure ArcSDE service version is up to date, and supports existing and new application functionality Ensure ArcSDE compatibility with applications (i.e. MapObjects, ArcIMS, ArcGIS) Install and configure upgrades to ArcSDE service and its application clients Support and remedy incompatibility issues Implement ArcSDE upgrades (service packs, sdeupgrade) Install and configure upgrades of ArcSDE dependencies (e.g. DBMS, access libraries, operating system) Requirements to do the Job: Must be able to monitor and maintain the versions of ArcSDE and the applications it supports Must be able to monitor and maintain the versions of ArcSDE dependent technologies (e.g. operating system, DBMS, etc.) Must have administrative privileges on ArcSDE server Must have administrative privileges on ArcSDE DBMS Must have knowledge of IT policies and methodologies Owner: IT or GIS Database Tuning Water/Wastewater DB Design Conference February 2005

Research Improvements in Data Management
Responsibilities: Continually investigate how to improve spatial data management Continually test and scrutinize new spatial data management tools, techniques, and technologies Investigate new data management techniques (e.g. DBMS abstract data types) Investigate new spatial analysis to improve overall the organizations effectiveness Requirements to do the Job: Must have knowledge of common GIS tools, techniques, and modern technologies Must have knowledge of common spatial data maintenance issues and requirements Must have knowledge of common spatial data access issues and requirements Must have knowledge of GIS services and practices Must have a willingness to try new things Owner: IT or GIS Database Tuning Water/Wastewater DB Design Conference February 2005

Development Tasks Database Tuning Water/Wastewater DB Design Conference February 2005

Develop Tools and Workflow to Maintain Changes to Spatial Data
Responsibilities: Translate related changes in business data to changes in spatial data (e.g. city annexed 2 new parcels) Use the right GIS tool and approach to maintain the spatial data Automate data maintenance where practical Support automation (tools or workflow) Integrate new GIS tools and techniques to maintain the spatial data Requirements to do the Job: Must have knowledge of the functional capabilities of ESRI client applications Must have knowledge of current software engineering policies and methodologies Must have knowledge of GIS policies and methodologies Must have knowledge of GIS discipline (e.g. coordinate systems, topology) Must have knowledge of spatial editing issues (e.g. topology, spatial processing) Owner: IT or GIS Database Tuning Water/Wastewater DB Design Conference February 2005

Develop Custom GIS Solutions
Responsibilities: Understand and translate specific business need to GIS analysis perspective Identify the right tool and spatial data to deliver that business need Develop spatial solution to satisfy that business need Test spatial solution Deliver or deploy spatial solution Requirements to do the Job: Must have the ability to translate general business needs to spatial processing requirements Must have knowledge of the functional capabilities of ESRI client applications Must have knowledge of current software engineering policies and methodologies Must have knowledge of GIS policies and methodologies Must have knowledge of GIS discipline (e.g. coordinate systems, topology) Owner: IT or GIS Database Tuning Water/Wastewater DB Design Conference February 2005

Develop General GIS Applications
Responsibilities: Develop GIS application for general business needs (e.g. “show all the hydrants inspected this year on a map”) Test GIS application for general business needs Deploy GIS application for general business needs Maintain GIS application for general business needs Requirements to do the Job: Must have knowledge of the functional abilities and limitations in ESRI client applications (e.g. ArcObjects) Must have knowledge of current software engineering policies and methodologies Must have knowledge of GIS policies and methodologies Must have knowledge of GIS discipline (e.g. coordinate systems, topology) Must have knowledge of IT policies and methodologies Must know how to deploy applications Owner: IT or GIS Database Tuning Water/Wastewater DB Design Conference February 2005

Support Deployed GIS Applications
Responsibilities: Support users with technical issues about GIS applications (e.g. “help desk”) Maintain the data sources for GIS applications Troubleshoot functional issues of a GIS application Support web based spatial applications (if web based mapping services apply) Requirements to do the Job: Must have the ability to install and uninstall applications and their dependencies Must have knowledge of IT policies and methodologies Owner: IT or GIS Database Tuning Water/Wastewater DB Design Conference February 2005

Spatial Data Design Responsibility: Determine the appropriate layout of spatial data layers, including: Organize by entity type (e.g. points, lines, polygons) Organize by functional analysis (e.g. SQL queries) Organize by normalization principles Organize by maintenance (i.e. minimize maintenance effort) Organize by spatial data vendors (e.g. Assessor changes, CAD structure) Requirements to do the Job: Must know spatial data maintenance workflow in detail Must know how the spatial data will be used (i.e. map functionality and products) Must know basic principles of normalization and database objects Must know ArcSDE layer schema Owner: IT or GIS Database Tuning Water/Wastewater DB Design Conference February 2005

Thank you! Questions?

Bryan Dickerson, Woolpert Jim Moening, ESRI

Similar presentations

Presentation on theme: "Bryan Dickerson, Woolpert Jim Moening, ESRI"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Bryan Dickerson, Woolpert Jim Moening, ESRI

Similar presentations

Presentation on theme: "Bryan Dickerson, Woolpert Jim Moening, ESRI"— Presentation transcript:

Similar presentations

About project

Feedback