Enterprise ETL & the Home Again Web Mapping Application: Columbus, Ohio Sonia Krammes GIS Analyst City of Columbus, Dept. of Technology Erick Lobao GIS Project Manager Stantec Robert Parsons Software Engineer City of Columbus, Dept. of Technology Nick Soltes GIS Analyst Stantec 2009 Ohio GIS Conference September 16-18, 2009 Crowne Plaza North Hotel Columbus, Ohio

Overview Home Again Methodology Technical Issues Web Application Return On Investment

Home Again Home Again is a Public – Private partnership put forth by the Mayor to revitalize areas of Columbus by purchasing, renovating, selling properties, and demolishing vacant homes. The Department of Development maintains data using several applications and creates reports from these data sources to manage the Home Again initiative. Columbus

Project Goals Combine data stored in different file formats with data models / table structures. Recreate and automate reports that were manually produced by a former employee. Display Home Again point information on a web map and provide a basic level of spatial analysis.

Workflow Diagram

Data Sources Description Owner Format Frequency Vacant Housing Application (V.H.A.) Development / Code Enforcement Oracle Table Daily Land Bank Approach Table As Needed Columbus Housing Partnership CHP Excel Worksheet Monthly Affordable Housing Trust AHT Building Permits, Private Demolition Permits Department of Development Acella (Oracle Table) Public Demolitions Microsoft Access RESPEC City Attorney's Foreclosure Information City Attorney's Office Unknown

This is a Job for Model Builder! Model Builder allows you to string together ArcGIS processes to automate multi-step operations. Models can be exported to a script and scheduled to run as a scheduled task. Model Builder scripts can be extended with Java, Python or VBA code if needed.

GIS Workflow Model goes here And here

Why Model Builder? Easy to use. Basic understanding of available geoprocessing tools. Programming language skills not required. Future updates, revisions, changes can be made by GIS staff. Models are re-usable / speeds up repetitive tasks. Execute tasks in a defined order. Open a single file to perform a total of 31 geoprocessing operations.

Under The Hood Models developed with one or two processes at a time. Allows you to debug / tune model properly Export, format, combine data from multiple sources. Join tables based on Parcel ID Assign spatial information to non-spatial data Export to python script for scheduling processes using MS Windows Task Scheduler

Adding Location to Table Issue: Data stored in a table without spatial reference. Address information and “Home Again” attributes stored in a table. Solution 1: Create an Oracle table view to join staging table to City’s Master Address Point Layer. Provides a “Live-View” of final point layer. Significant amount of lag time when viewing this layer. Solution 2: Use model builder to extract Home Again Points utilizing the coordinates of the Address Point Layer. Produces a “static” file each time model is executed. Adding Location to Table

Adding Location to Table Join staging table to address points based on Parcel ID. Update staging table with (X,Y) from address points. Create event feature from staging table. Adding Location to Table

Adding Location to Table Join stage table to address points where Parcel ID = Parcel ID Adding Location to Table

Adding Location to Table Calculate X, Y coordinates within stage table. Adding Location to Table

Adding Location to Table Create event feature class from table. Remove any <NULL> X,Y points Adding Location to Table

Good… But Not Perfect Adds Complexity. Model requires ArcMap to call ArcGIS functions. Limits use to machines with ArcMap installed. ArcGIS file “locking issues” (Not an issue in Python) Some operations in Model Builder were buggy. Model Builder can do a lot, but at what point do you make the jump to something else?

Web Application

Monthly Reports

ROI: Process Who’s available to maintain and update ETL procedures and what’s their skill level? Leverage existing staff, software, and experience. Leverage existing software licenses (ESRI, Crystal Reports). Modular approach spreads responsibility. Time = $ Repeatable and Automated ETL

ROI: Home Again Web Application provides GIS capabilities to non GIS savvy users. Allows GIS staff to work on higher level tasks. Provides spatial analysis capabilities to City leadership. Spatially enabling data. Getting more use / functionality from existing data. Development staff can examine relationships between multiple data sources. Informs City leadership and Home Again program staff.

A More Permanent Solution Streamline ETL procedures by improving data capture and storage. Ex. use common field names & data types Helps with report generation and data aggregation. Web site with forms to enforce data standards. Provide flexible reporting.

Questions & Comments Sonia Krammes Erick Lobao Robert Parsons GIS Analyst City of Columbus, Dept. of Technology SRKrammes@columbus.gov (614) 645-5392 Erick Lobao GIS Project Manager Stantec Erick.Lobao@stantec.com (614) 844-4011 Robert Parsons Software Engineer City of Columbus, Dept. of Technology RGParsons@columbus.gov (614) 645-5711 Nick Soltes GIS Analyst Stantec NSoltes@stantec.com (614) 486-4383