The GDB Can store tables (just information), vector feature classes, and raster layers
Layers and Layer Files All GIS Datasets are considered LAYERs in ArcMap. A LAYER FILE is a file that you save in ArcMap to retain customized settings. –This file refers to the LAYER (shape file, coverage, grid, or feature class) –It displays the data with your saved visualization settings, textual annotation, etc.
GeoDatabasesGeoDatabases Personal File-based ArcSDE Personal ArcSDE Professional (or Enterprise)
Personal Geodatabases Uses the MS Access Jet Database engine –Note: Do not open/edit these with MS Access Limitations –2GB (Access) –Only vector feature classes are actually stored inside the Access database –4 users but only one editor –Does not support versioning
File-based Geodatabase fGDB Stores vector and raster layers in the file/folder structure. Limitations –Multi-user (max = 10) –1 Editor (no versioning) –Max size is 1 TB RDBMS
ArcSDE Personal Uses MS SQL Server Express Limitations –4 GB –Supports versioning/replication but only one editor
ArcSDE Professional Geodatabases Uses DB2, Oracle, Informix, SQL Server, etc. No software size limits and unlimited number of users Can accommodate vector and raster data
Given all these differences, there are really many similarities
Geospatial Data Storage (Vector) Geo-spatial data are stored as Feature classes Non-spatial data are stored as stand- alone tables Vector data is handled by DB2s Spatial Extender. SDE is a broker.
Geo-spatial Data Storage (Raster) Two methods –Stand-alone raster data set –Mosaic ArcSDE is not the best solution to store raster GIS data for the Enterprise –Size considerations –Performance issues Raster data is handled by SDE
Internal Data Storage Within the DB2 RDBMS –All data is stored within table spaces – referred to by Configuration Keyword. –A Configuration Keyword points to a set of two table spaces: Attribute table space Coords table space
Loading Vector Data into a GDB PART 1: Stand-alone feature classes
The Spatial Index Grid Uniform grid of square tiles –Like grid reference on a street map Each feature (lakes) referenced by one or more tiles –Envelope of feature determines tiles occupied Spatial Index Key records occurrences of features in tiles –Empty tiles not stored
Loading Vector Data into ArcSDE PART 2: Feature classes within a Feature Data Set –First, you need a Feature Data Set –What is a Feature Data Set?
A Feature Data Set is: Required to implement Full Topology! What?!
Full Topology The spatial relationship among feature classes participating in a topology layer –Must belong to a feature dataset Feature classes share geographic reference system, and spatial domain. –More realistic representation of data
A Feature Data Set then… Is an organizational tool used to ensure that all feature classes within it use a common: –Geospatial reference system –Spatial domain
Understanding the Spatial Domain Low-precision GDB Based upon LONG INTEGER (32- bit) –What is the domain range of a LONG? High-precision GDB Based upon 64-bit Integer –Covers a geographic reference systems Horizon
Fitting the World into a LONG If we express the X,Y coordinates in the familiar Latitude/Longitude system… By whole degrees, we would use: –Latitude (180 units) –Longitude (360 units) This is only % of the 4B space
Problems with this approach Resolution to 1 degree is terrible Wastes the capacity of LONG INTEGER
What if we use Decimal Degrees? Hold on! Decimals cannot be stored in an INTEGER data type Lets just shift the decimal place to the right by multiplying the coordinate by a scaling factor –e.g., 10 preserves one decimal place, 100 preserves decimal places etc.
Fitting the World into a LONG (revisited) By using a scaling factor of 1M, the world would fit nicely into a 3.6B space (theres even a bit left over!) What is the spatial resolution of 1/1M th of a degree? –Approximately 1/10 th of a millimeter!
More about the High- Precision GDB Can be pGDB, fGDB, or SDE GDB Uses 64-bit integer to encapsulate the spatial horizon What? –64-bit numbers have a range of 18,446,744,073,709,551,616 –Thats 18 quintillion! –http://www.jimloy.com/math/billion.htm
The Spatial Horizon? Essentially, its a spatial domain large enough the contain the entire earth at high- precision
Applying this to ArcGIS –Rule #1, use the high-precision GDB model whenever possible. –Why not always?
Hints and Tips Optimize the spatial domain by using high-precision GDB Feature dataset If not, set up your low-precision Feature dataset to –Allow for spatial growth –Allow for improved instrumentation
ArcSDE Professional Demo –Import a vector data set into ArcSDE
Think about it… Object-relational databases have native geospatial support ArcGIS for Server can make geospatial data available to the Enterprise Do we need an ArcSDE middle-ware? –ArcGIS Spatial Data Server
Geodatabases in an Enterprise Workflow Keith T. Weber, GISP GIS Director, ISU GIS Training and Research Center
UNDERSTANDING AND MANAGING WORKFLOW Presentation and Discussion
Lets Get Started GIS is… –Data-driven –Powerful –Dynamic
GIS Data Life Cycle EditionBackupEditValidate Update Metadat a Create Data Change Happens!
The Bottleneck Distributing the new edition
The Solution Networks and the Internet
A New Problem is Born MY version
GIS Grows Up! RDBMS –Keep the benefit of network connectivity –Eliminate the problem of MY version –Eliminate the bottleneck –And, change the cycle of events
GIS Data Life Cycle Edition Version and/or Replicate Edit Validate: Synchronize or reconcile and post Update Metadata Create Data Change Happens!
Backup vs. Versioning Backups and archiving are still critical steps for the enterprise. BUT, not part of the GIS Life Cycle any longer
In the Beginning… Backups were made in case we really messed up Edits were made to the original Copies of the clean new edition were distributed
Today…Today… The original [parent] is versioned [a child is born] Edits are made to the child, not the parent Clean edits are copied [synchronized or posted] to the parent.
Benefits Of This Approach Brainstorm!!! –Minimize downtime –Processes completed within the RDBMS
The Role of Backups Data retention and deletion Legal requirements
GIS Data Life Cycle…Today Edition Version and/or Replicate Edit Validate: Synchronize or reconcile and post Update Metadata Create Data Change Happens!
REPLICATION AND VERSIONING Presentation and Discussion
What is Replication? Duplication Copying Mirroring
True Replication… Does not need ArcGIS Every RDBMS can be replicated natively However, using ArcGIS to perform the replication –Is easy –Supports GIS workflows better
Why Replicate? Enable disconnected editing for: –Performance/load balancing –Network load reduction –Publishing data to subscribers
Network Load Reduction The network is a primary bottleneck in a High-Capacity Enterprise Note: capacity refers to how many concurrent users a system can support without loss of performance
How Do I Replicate? We will cover this with the hands-on exercise As an overview… –Version the database –Replicate the database –Edit/update –Synchronize changes with the parent
So Replication is Versioning No… but replication uses a versioned database
What is Versioning? One database Parent edition (tables) remains live/usable Child edition(s) simultaneously edited Roll-up is seamless
Versioning: Principal Concepts Edits are stored in Supporting Tables Geographic changes (linework) are stored in Supporting Vector Tables Attribute changes are stored in Supporting Delta Tables.
Delta Tables A = Add (insert) D = Delete U = Update (delete existing then add)
A Tree is Formed As versions are created and changes are made, a tree grows –Q: What kind of tree? –A: A State Tree
Sort of an Upside-down Tree
The State Tree Default: state 0 Arthurs Court sub-division [Another] sub-division Tree Trunk Branche s
Multiple Versions Multiple versions are allowed –Versions can be based upon location (north edits, south edits), projects (sub- divisions), or other logic decided upon my the GIS Manager. Batch reconcile and post are supported
The Day of Reconciliation Arthurs Court sub-division edits have been completed –Time to reconcile –This process looks for conflicts –Once all conflicts have been resolved… –Reconciliation is complete
PostPost To roll-up the edits back to the trunk of the state tree we Post
ConsiderationsConsiderations Performance can degrade with active databases –Workflow itself can generate unnecessary versions –Delta tables will become large over time –DBMS statistics may need to be refreshed or reviewed by the DB Admin
The Cure For many of these ArcGIS-centric performance issues is compressing the database –Moves common rows from delta tables into base tables –Reduces depth of the state tree by removing states no longer needed
Hands-On Exercise Practice both replication and versioning
Your Assignment Complete the exercise handouts –Connecting to and using SDE on DB2 –Practice both replication and versioning Read the PDFs in the SDE exercise folder Visit the URL link for Spatial Data Server and explore this topic
Key Concepts SDE is an engine layer residing between a spatially-enabled RDBMS and the GIS desktop. SDE enables Enterprise GIS SDE reduces data management responsibilities. Understand Enterprise workflow