Presentation on theme: "GIS Mapping of the New York City Sewer System Diagram Diagram Geodatabase Design Geodatabase Design Geodatabase Design Topic Outline Entering spatial/non-spatial."— Presentation transcript:
GIS Mapping of the New York City Sewer System Diagram Diagram Geodatabase Design Geodatabase Design Geodatabase Design Topic Outline Entering spatial/non-spatial data - Brief Overview of Baker's process (scanning, digitizing and scrubbing) Entering spatial/non-spatial data - Brief Overview of Baker's process (scanning, digitizing and scrubbing) Linking spatial and non-spatial data - Indexing Application, Sewer Database design (database structure, Feature Class, table attribute, relationship) Linking spatial and non-spatial data - Indexing Application, Sewer Database design (database structure, Feature Class, table attribute, relationship) Data validation - QA process (Automated and detailed feature inspection). Data validation - QA process (Automated and detailed feature inspection).
Entering spatial data Types of DEP source Documentsource Document As - built Contract Interceptor Plans Index Maps Emergency Contracts Drawings Infiltration and Inflow Maps (I/I maps) Connection Book Field Survey Types of Non - DEP source document Department of Transportation MTA Bridge and Port Authority NYC Orthophotos DCPLION NYC LANDBASE GEOSUPPORT Sanbon - Property Maps
Methods of Data Inputs Scanning The process of translating hardcopy media into digital form that can be recognized by a computer. Create permanent records. The process of translating hardcopy media into digital form that can be recognized by a computer. Create permanent records. Digitizing The method of data capture that involves the conversion of data in hardcopy or raster forms into Digital vector formats. Digitizing produces features (pipes and manholes) used in a GIS. * Features are digitized relative to the NYCMAP with 1 foot horizontal accuracy * Precision : Offset value (distance btw feature and the curb line). Pipe Length Pipe Length Scanning differs from digitizing in that the entire pages of data or map sheet are captured all at once Geo-referencing The process of defining the position of a geographic object relative to a standard reference. Images with spatial data (I/I maps) were geo- referenced. Field Survey
Entering non-spatial data S crubbing The act of cleaning a surface by rubbing it with a brush and soap and water S crubbing The act of cleaning a surface by rubbing it with a brush and soap and water A meticulous process of reviewing and highlighting relevant attribute information for inclusion in the database. * Cleaned for inconsistency * Information retrieved are used to populate and link to their respective feature. respective feature. * Standardization of domain and ranges (spatial and attribute)
Linking spatial and non-spatial data Linking spatial and non-spatial data Indexing Creating a database of information for easy search and retrieval Indexing aid in filtering through a huge amount of information. * Document Indexing A procedure that associates a source document with its respective street centerline A procedure that associates a source document with its respective street centerlinestreet centerlinestreet centerline * Feature Indexing A procedure that associates a source document with its respective feature or A procedure that associates a source document with its respective feature orfeature attribute attribute Where multiply source have been linked the most recent document acts as the default Source. Document are stored using a directories or folders and subfolders by their borough location. Relationships are established within the database. Relationship require ID.
Quality Assurance Procedures Quality Assurance Procedures QA Workflow The QA process consists of several phases that are defined in the procedures below: Check of digital data delivery and hard copy supporting documents Automated database checks using QA software Context check of plots and Engineering Review within the delivery area. Feature inspection based on a randomly selected sample set using ANSI standards
Workflow 7. Conversion Issue Resolution and Communication 5. Accept delivery 6. Reject delivery Accept? YES Delivery Complete? Accept? NO 4. Conduct Sample Feature 3. Conduct Context Check & Engineering Review 2. Conduct Automated Checks 1. Check Data Delivery Package Figure 1: QA Flow Chart YES NO
Quality Assurance Data Validation Techniques The QA process ensures that database design specifications are adhered to during conversion. The automated routine runs checks for data completeness, spatial inaccuracies, context and attribution errors, scheme and data structure and connectivity error. The process is divided into three broad groupings – Automated Checks, Context Checks/ Engineering Review and Feature Inspections Checks. The compliance level for automated checks is 100% while the compliance level for feature inspection is 95%. Automated Checks Automated checks consists of three types of data evaluations—object design, geometry, and connectivity checks. Table 1 summarizes the Automated Validation Checks. The Automated application runs in batch mode with pre-configured tables controlling the parameters. Most feature characteristics and attributes can be checked through automated routines.
Object Design Review all features to ensure that all attribute values match domain tables Review all features to ensure that all mandatory values are populated and only permissible null values are populated in the database All unique values such as Facility ID are in fact unique both in a single delivery, as well as in the entire database Compare the attributes with the various area polygons such as Grid_no, etc., to determine if the attributes are correctly populated Check facilities with assigned linked annotation to assure they have at least one annotation feature linked to them. Geometry test Review all junction features with elevation to ensure that the To and From attribution are correctly populated per pipeline Review all field-surveyed points in the Geodatabase located in the x,y coordinates captured in the field Review all features to identify any features that occupy the same x, y coordinate (i.e. duplication of geometry). Connectivity Test. The connectivity tests will: Validate that all features meet the topology business rules (i.e., identify all floating junction features and/or edge features that are not terminated with a coded junction feature). Conduct an audit and generate a report of all junction features and list features that have 0, 1, 2, and >2 edges connected to them. Per each junction type, generate a list of features that have 0, 1, 2, or > 2 edges connected providing the ability to individually review each feature.
Data Structure and schema Type of Validation DescriptionPurpose/Notes Spatial Attribute Check Grid No, Maintenance Yard, Drainage District, Treatment Plant Features are validated based on there spatial location within the Quality Assurance Layers Document Link and Feature Linked Annotation Check Verifies that the each feature is linked to a source document and its related piece of annotation. Domain Validation Check: *Variation between deliveries *Attribute Domain Test for changes in defined domain set. Detect attribute values that fall outside of the legal values. Ensure consistency btw deliveries Attribute Name Check Verifies attributes/field consistency between deliveries. An error with a coded value attribute error could point to an uncontrolled editing environment. Example, digitized features should have a value of “Active” for the Operational_State field. Null and Uniqueness Value Checks Attribute/fields The nulls test checks for unpopulated values in an attribute. The uniqueness test seeks out duplicate values for an attribute field that is designed to be unique, such as Facility-IDs. Null values for an attribute may signal an uncontrolled data processing procedure. Unique values for an attribute are critical for referential integrity, especially for primary key attributes. These tests are critical for sewer dataset because invalid values can cause symbology and analysis errors..
Context and Attribute Error Descriptions Type of Validation DescriptionPurpose/Notes To and From Pipeline Invert Elevation Check Flow Identify instances where the From[Inv_Elevation of a pipe is lower than the To_Inv_Elevation [ Review all junction features with elevation to ensure that the To and From attribution are correctly populated per pipeline Manhole has two out sewer line Multiple Exit Identifies instance were manhole have more than one exit sewer on it downstream section. Check for only usually configuration Isolated Junction Check (Network Junction, Simple Junction) Connectivity Identifies isolated features e.g. manhole and sewer structure nodes that have no pipes connected to them. Ensure that all features are within the network Elevation Check at Manhole Flow Compares incoming pipe invert elevations to outgoing elevations at each manhole. Ensure that incoming pipes have higher or equal invert elevations than outgoing pipes No Outlet Sewer - Connectivity (Dead End Check) The Check identifies sewer sections that end abruptly therefore obstructing the general effluent flow.
Check Plots Reviews sewer configuration by comparing the geo-referenced I/I maps and digitized sewers. This check is to make certain that no significant segments of the sewer system are missing. Connectivity and flow tracing will support this step. Other step include sewer length verification. Engineering Plots Baker Delivery Plots (Engineering Review) - performed by a team of DEP Engineers/Staff. The check results in identification of configuration, attribute, and annotation errors. Valid errors are entered in the QA application along with other recorded errors Context Checks Infiltration and Inflow Maps (I/I Maps): Infiltration and Inflow maps (I/I) were originally produced from as-built drawings. They cover 14 drainage areas within New York City. The I/I maps show the following sewer information: sewer type, size, shape, invert elevations, flow direction, and year of as-built. Because they have not been maintained, they are considered incomplete, and are generally inaccurate. They should be used for reference only as a secondary source to be used for an understanding of the sewer system within an area.
Sample Feature Inspection This process involves a detailed inspection on a selected set of feature. The selection is chosen randomly and based on statistical principals that establish an overall confidence on the quality of the data delivered. The ANSI test consists of four components: Calculate batch size, Select general inspection level, Calculate sample size and make a random selection from the batch size Summing the errors for acceptance/rejection of the batch. Table 2 shows how the ANSI standard (American National Standard Institute) is used in the Sewer Project.
Table 2: ANSI Sample Size Code Letters14
Table 3. Single Sampling Plans for Normal Inspection
Table 4: ANSI Interpolated Number of Errors Allowed per Sample Size for 95% Acceptance and Rejection Criteria
Table 5: Feature Count and Sample Size
Batch(A) Total Feature Errors Feature Checked Total Attribute Errors Attribute s Checked (B) Weighted Attribute Errors (C) Allowable Errors Pass/Fail G1 Sewer/Force Main Pass M1 Manholes Pass P1 Other Point Features Pass Results of Random Sample Check