Presentation on theme: "Spatial Disaggregation – A Primer"— Presentation transcript:
1 Spatial Disaggregation – A Primer Tom D’Avello – NRCS-NSSC-GRUcontact:Travis Nauman – NRCS-NSSC-GRU, WVUcontact:
2 Overview Define ‘Disaggregation’ Approaches and Tools Summary West VirginiaIllinoisArizonaSummaryLiterature list for your reference
3 What is spatial disaggregation? The next opportunity for the NCSSAdd value to SSURGO“The process of separating an entity into component parts based on implicit spatial relationships or patterns” – (Moore, 2008)Getting more detailSpatially refining maps to reflect the level of detail for current needsCorresponding increased resolution of attributesTrying to meet new types of demands
4 What is spatial disaggregation? Mapping of components within map unitsUsually complexes or associations for Order 2 & 3 soil surveys (SSURGO)STATSGO2 effortAlaska (Moore, 2008)New needs servedmodeling communitymaintenance and improvement of the product is a primary charge of NCSSThe description of what is mapped and where it occurs has been done well. This is the process of showing where different soils occur. A better defined spatial representation of soil patterns leads to questions and research towards why the pattern occurs as mapped, and how they behave as a system.Modeler's - carbon, water table, hydric soils, tile drainage, drainage management systems, available water etc
5 What is spatial disaggregation? Ultimately, it is a refined segmentation of the landscapeAlong with the spatial, the attributes are equally importantMap units have multiple parts with attributesExample: Ponded parts of a larger map unitRelated to SDJRScope driven!Area of InterestCan be relevant to one, some or all map units.The landscape is used comprehensively here, i.e., the soilscape – all soil forming factors in natural, meaningful classes
6 Purpose of the demonstration Demonstrate case studies across varying physiographic regionsGet feedback from soil scientists on their assessment of current soil mapsInvestigate different digital techniquesEvaluate resultsDevelop materials and guidelines for application by soil scientists
7 West Virginia early efforts Gilpin-LaidigPineville-Gilpin-GuyandotteOtherSSURGO Map UnitsGilpinPinevilleLaidigGuyandotteDekalbComponent SoilsCraigsvilleMeckesvilleCateacheShouns
8 General disaggregation workflow GoalsScopeWhat data is accessible to helpChoose methodImplementValidate Quality(evaluate and iterate earlier steps as needed)Assuming 1 -3 are constant for typical project td
9 Current workflow in West Virginia GoalsSoil series map on field scale gridScopeAll map units in Pocahontas and Webster Counties, WVWhat data is accessible to help~30-meter DEM (NED), Landsat Geocover (Fed. MDA, 2004), lithology, SSURGOChoose methodSSURGO-derived expert rule training sets & classification tree ensemble (100 trees run on random subsets)ImplementRun analysis with Access (SQL), GIS, and Python (or R)Validate QualityIndependent pedons for ground truthAssuming 1 -3 are constant for typical project td
10 1. & 2. Goals and scope Scope is key define what needs to be disaggregatedUniversal vs within map unit(s) (Local)Local model(confined to existing map unit)Keep original linesUniversal modeluses original survey to create but lines not used for finalLocalLocal and Universal/Global may work as terms - TDUniversalFigures courtesy of Dave Hoover, NSSC
11 Components (not explicitly mapped) 3. What data: SSURGOComponents (not explicitly mapped)InclusionLegendMap UnitHorizonGeomorphologyParent materialLandscape attributesHorizon attributesSoil physical propertiesSoil chemical propertiesComponentSSURGO database diagram in graphic formNotes: mention that surveys of different vintage has to be migrated and forced into this format despite being mapped with potentially different procedures, talk about inside-mapunit data for use in disaggregation
12 3. What data: SSURGO Most work done on SSURGO or equivalent scale maps Raster (grids) used for modelingto match environmental dataWest Virginia data
13 3. What data: environmental Raster gridsSometimes other polygon layers converted (e.g. geology)Characterize variation within polygons using data that infer soil forming factorsSSURGO lines over landforms(Schmidt & Hewitt, 2004)SSURGO lines over LandsatSSURGO lines over DEMExamples from West Virginia
14 4. Method: model techniques Training DataMatch environmental data to components of interestUse representative areas or pedon locationsModel TypesExpert landscape rulesHardened or fuzzyStatistical modelsArea to Point Interpolations (Goovaerts, 2011)Dekalb series training areas in WVExample Classification Tree Model
15 5. & 6. Implement & Validate Create raster disaggregation map Validate with ground truth dataDifferent methods available validationSpatial Supportmatch typenearest60-m radiusexact26%39%like soil45%66%any tree57%73%WV example: universal model for Webster and Pocahontas Counties
16 Historical survey of Webster County, WV These folks were pretty goodMilton WhitneyCurtis MarbutHugh BennettNice map, too!Soil survey efforts have a long history. The disaggregation process is a continuation of improving the product if possible to satisfy a need for more detail.
17 Peoria, Illinois investigation GoalsComponents or phases within Sable and Ipava unitsScopeAll Sable and Ipava map units within Peoria CountyWhat data is accessible to help3-meter DEM (NED), SSURGOChoose methodExpert rule training sets & classification treesImplementRun analysis with R, ArcGIS and ArcSIEValidate QualityLocal soil scientist review.
18 Peoria, Illinois investigation 1. Goals Identification of Non-ponded and ponded phases in Sable unitsIdentification of poorly drained components in Ipava unitsIpava and Sable soils are extensive in occurrence
19 Peoria, Illinois investigation 2. Scope - study site The project area is within MLRAs 95B, 108A,108B, 108C and 115C~900,000 acres of Sable~1,186,000 acres of IpavaWhy here?Availability of high resolution DEMsRepresentative setting for Sable and IpavaGood test for developing procedures to complete for entire extent of units when LiDAR coverage is completeIpava and Sable soils are extensive in occurrence
20 General setting 2. Scope - study site Typical cross-section and qualitative description of Sable and Ipava soilssoilslopeprofiletangentialwetnesspositionsinksIpavaLowPlaneHighBroad summit/TalfSomeSableLowestConcave-planeHighestDip on talfYesThe setting is flat – talf – dip -rise
21 Variables developed 3. Data - all derived from 3m DEM with ArcGIS/ArcSIE/SAGA GIS Altitude above channel networkCurvature at numerous neighborhoodsHorizontal distance to flow channelMaximum curvature –numerous neighborhoodsMinimum curvature –numerous neighborhoodsMulti-resolution ridge top flatness indexMulti-resolution valley bottom flatness indexProfile curvature –numerous neighborhoodsRelative position-numerous neighborhoodsSink depth and Depression cost surfaceSlopeTangential curvature –numerous neighborhoodsTopographic position indexVertical distance to flow channelWetness indexThese are commonly derived by SS and GIS staff with CCE software
22 Exploratory Data Analysis 4. Method An extensive sample with soil series as a response was developedClassification Tree in R to determine explanatory variables
23 Purpose of evaluation 4. Method Spatial data needs to be the driver for modeling effortEfficient determination of explanatory variablesEfficient determination of thresholds for variablesPractical tools are needed to assist soil scientists in this effortDo a cursory assessment of all of the major soils to determine how strong the soil-landform relationships are for the given set of potential variables (“predictors”). Provides an indication of similarity among soils in terms of data parameters. E.g. two soils that share geographic settings will not be separable using DSM techniques. However, is “hole mapping” more effective in these cases?
24 Results from classification tree 5. Implement Input variablesImportant variablesAltitude above channel networkHorizontal distance to channelMinimum curvature 120m neighborhoodMulti-resolution ridge top flatness indexProfile curvature 150m neighborhoodRelative position 90m neighborhoodRelative position 60m neighborhoodRelative position 30m neighborhoodSink DepthSlope 30m neighborhoodTopographic position indexWetness indexAltitude above channel networkRelative Elevation (aka Relative position)Sink DepthDeveloped 20+ datasets – 12 showed promise from qualitative review – 3 wereidentified through classification tree as explanatory variables in this example
25 Results from classification tree 5. Implement -Ipava and Sable independentlyClassification trees have the advantage of being a “White Box” technique – everything is displayed and can be interpreted readily as opposed to other “Black Box techniques like AIThe “first” node (lower left), indicates 87% classified as Ipava, 13% misclassifiedThe “second” ( next one to the right) indicates 69% classified as Ipava and 31% misclassifiedThe “third” ( next one to the right) indicates 64% classified as Sable and 36% misclassifiedThe “fourth” ( lower right) indicates 76% classified as Sable and 24% misclassified
26 Results from classification tree 5. Implement – walk through the splitsAltitude above channel network>= 0.25< 0.25
27 Results from classification tree 5. Implement – walk through the splitsRelative position>= 0.595< 0.595
28 Results from classification tree 5. Implement – walk through the splitsSink depth>= 1.472< 1.472
29 Results from classification tree 5. Implement - Results of rules applied for Sable and IpavaDo the rules determined from the classification tree match SSURGO? Another FYI, and to identify potential outliers and improve concepts for the geographic setting
30 Results from classification tree 5. Implement Rule base compared with SSURGO for SableDo the rules determined from the classification tree match SSURGO? Another broad assessment, and to identify potential outliers and improve concepts for the geographic setting. This looks good for Sable, especially given the low relief and limited tools available to soil scientists at the time it was mapped.
31 Ponded vs. non-ponded Sable 6. Validate Local - using depression depth Blue – likely depression/pondedRed -Yellow – no depressionDepression depth is one potential derivative that indicates ponding and possibly ponding duration
32 Ponded vs. non-ponded Sable 6 Ponded vs. non-ponded Sable 6. Validate Local - using depression cost surfaceBlue – likely depression/pondedRed -Yellow – no depressionDepression cost surface is another potential derivative that indicates ponding and possibly ponding duration
33 Ponded vs. non-ponded Sable 6. Validate Local - using 3m USGS NED Zonal statistics indicate 41% of the area mapped as Sable is pondedBased on selected thresholdsVerification and tuning of threshold values is ongoingHigh resolution DEMs have artifacts and some areas identified as depressions may be due to noisy data. Changing the threshold effects estimate, which must be verified with ground truth. Spatial representation provides a perfect medium to supply to the greater field staff for verification across the extent of mapping.
34 Ponded vs. non-ponded Sable 6. Validation/Data Local - using 10m USGS NEDZonal statistics indicate 17% of the area is pondedBigger legendArea “missed” withcoarser 10m DEMTest with 10m just to answer inevitable question, “how does a 10m DEM perform?” Horizontal resolution is too large to match small changes in relief in gentle terrain
35 Ponded vs. non-ponded Ipava 6. Validate Local - using 3m USGS NED Zonal statistics indicate 9% of the area is ponded
36 Future effort for Peoria County Populate component table- based on verified and validated thresholdsRename map unit phases if neededWhat is reasonable to improve product?Accept line work and split components within existing map units?- A working copy in preparation for phase II of data recorrelationmakes this feasible
37 Arizona – arid example Goal Scope Data Method Implement match environmental classification of soil forming factor raster layers to soil types.ScopeEntire soil survey: Organ Pipe Cactus National Monument (ORPI)DataUsed DEM and ASTER imagery to represent topography, vegetation, and geologyMethodUnsupervised classification (clustering)ImplementErdas Imagine and ArcGISValidate (evaluation)Contingency tables (Chi2 Cramer’s V) to MUs; found separation of components in most complexes in field recon. (Nauman, 2009)
39 Arizona More methods trials are planned for northeast AZ Initial spatial data is being compiledModel runs by late 2013
40 Summary Webster and Pocahontas, West Virginia Peoria, Illinois ORPI, ArizonaGoalSoil series map of entire areaComponents/phases within Sable and Ipava unitsMatch environmental raster patterns to MUsScopeFull extent of both surveys (Webster and Pocahontas)Sable and Ipava map units within Peoria Co.Entire ORPI survey areaDataDEM (NED 30m), Landsat, geologyDEM (NED 3m)DEM (NED 30m), ASTERMethodSSURGO component rules and classification treesExpert rules and classification treesClustering (ISODATA)ImplementAccess (or SQL), ArcGIS, and Python (or R)ArcGIS, ArcSIE, RArcGIS, Erdas ImagineValidateIndependent set of pedonsExpert reviewCompare w/ SSURGO, expert reviewHighlightsSeries map, harmonized surveys, maintained accuracyPicked out fine scale depressionsDetected components in complex MUs
41 SummaryDisaggregation is a process that is defined by a need for more detailNeeds a directed scopeTremendous amount of new data and computing abilities to incorporateDisaggregating classic soil surveysimproves the detail of final maps without loss of accuracy and with no new datamore realistic representation of soil distribution (continuous – background probabilities)Can use new field data in future to re-model for easy update (doing this in WV)
42 Next Steps Match disaggregated data to ESDs Further disaggregate to ESD state and transition modelsWould better match imagery because management (e.g. pasture vs forest) is more easily detected with remote sensing.Could map at state and/or community level for direct use in conservation planningNational Range and Pasture Handbook, 2003Currently submitting article for peer review documenting WV case studyNauman, T., J.A. Thompson. (In prep). Semi-Automated Disaggregation of Conventional Soil Maps using Knowledge Driven Data Mining and Classification Trees
43 Resources – Available Training NRCS offers the following courses which provide an introduction to some of these techniques – check AgLearnSpatial Analysis workshop (distance learning)Introduction to Digital Soil Mapping (distance learning)Digital Soil Mapping with ArcSIE (conventional class)Remote Sensing for Soil Survey Applications (conventional class)
44 LiteratureBui, E., B. Henderson, and K. Viergever Using knowledge discovery with data mining from the Australian Soil Resource Information System database to inform soil carbon mapping in Australia. Global Biogeochemical Cycles 23. Bui, E.N. and Moran, C.J., Disaggregation of polygons of surficial geology and soil maps using spatial modelling and legacy data. Geoderma, 103(1-2): Bui, E.N., A. Loughhead, and R. Corner Extracting soil-landscape rules from previous soil surveys. Australian Journal of Soil Research 37: de Bruin, S., Wielemaker, W.G. and Molenaar, M., Formalisation of soil-landscape knowledge through interactive hierarchical disaggregation. Geoderma, 91(1–2): Goovaerts, P., A coherent geostatistical approach for combining choropleth map and field data in the spatial interpolation of soil properties. European Journal of Soil Science, 62(3): Häring, T., Dietz, E., Osenstetter, S., Koschitzki, T. and Schröder, B., Spatial disaggregation of complex soil map units: A decision-tree based approach in Bavarian forest soils. Geoderma, 185–186(0): Kerry, R., Goovaerts, P., Rawlins, B.G. and Marchant, B.P., Disaggregation of legacy soil data using area to point kriging for mapping soil organic carbon at the regional scale. Geoderma, 170: Li, S., MacMillan, R. A., Lobb, D. A., McConkey, B. G., Moulin, A., & Fraser, W. R Lidar DEM error analyses and topographic depression identification in a hummocky landscape in the prairie region of Canada. Geomorphology, 129(3), McBratney, A.B., Some considerations on methods for spatially aggregating and disaggregating soil information. Nutrient Cycling in Agroecosystems, 50(1-3): MDA, Federal Landsat Geocover TM 1990 & ETM Edition Mosaics Tile N TM-EarthSat-MrSID. USGS, Sioux Falls, South Dakota.
45 LiteratureMoore, A Spatial Disaggregation Techniques for Visualizing and Evaluating Map Unit Composition. NRCS 2008 National State Soil Scientist’s Workshop. Florence, Kentucky. ftp://ftp-fc.sc.egov.usda.gov/NSSC/NCSS/Conferences/state/2008/moore.pdf Nauman, T.W., Digital Soil-Landscape Classification for Soil Survey using ASTER Satellite and Digital Elevation Data in Organ Pipe Cactus National Monument, Arizona. MS Thesis. The University of Arizona. Nauman, T., J.A. Thompson, N. Odgers, and Z. Libohova Fuzzy Disaggregation of Conventional Soil Maps using Database Knowledge Extraction to Produce Soil Property Maps, In B. Minasny, et al., (eds.) Digital Soil Assessments and Beyond: 5th Global Workshop on Digital Soil Mapping, Sydney, Australia. Schmidt, J. and Hewitt, A., Fuzzy land element classification from DTMs based on geometry and terrain position. Geoderma, 121(3-4): Thompson, J.A. et al., Regional Approach to Soil Property Mapping using Legacy Data and Spatial Disaggregation Techniques, 19th World Congress of Soil Science, Soil Solutions for a Changing World, Brisbane, Australia. Wei, S. et al., Digital Harmonisation of Adjacent Soil Survey areas - 4 Iowa Counties, 19th World Congress of Soil Science, Soils Solutions for a Changing World, Brisbane, Australia. Wielemaker, W.G., de Bruin, S., Epema, G.F. and Veldkamp, A., Significance and application of the multi-hierarchical landsystem in soil mapping. Catena, 43(1): Yang, L. et al., Updating Conventional Soil Maps through Digital Soil Mapping. Soil Science Society of America Journal, 75(3): Zhu, A.X., A similarity model for representing soil spatial information. Geoderma, 77(2-4): Zhu, A.X., Band, L., Vertessy, R. and Dutton, B., Derivation of soil properties using a soil land inference model (SoLIM). Soil Science Society of America Journal, 61(2): Zhu, A.X., Band, L.E., Dutton, B. and Nimlos, T.J., Automated soil inference under fuzzy logic. Ecological Modelling, 90(2):