1. Yield Cleaning Software and Techniques OFPE Meeting 10-27-2015

2. Yield data cleaning conventions No standardized, accepted approach, but: Commonalities exist among investigated techniques Between 10-50% of data typically removed before analysis Visual interpretation of applied filters near end of processing is common Commercial software automatically applies some filters, but does not allow for manipulation of all important parameters

3. Yield Editor Software Created by Ken Sudduth and Scott Drummond, USDA-ARS Version 1, 2007; Version 2, 2012 Allows for manual and automatic manipulation of important parameters not accessible in commercial software

5. long lat flow gps time log interval dist width moist. header status pass #

7. Flow Delay already incorporated by some extent into the data (typically) use this to slightly adjust further and visually inspect results in some of the data deleted from end of transects Moisture Delay not usually same amount of time as flow delay Start and End Pass Delay trims values off start/end of transect used because as combine enters/exits the crop, values will be unreliable for some distance may interact with flow delay (e.g., a flow and start delay of 5 will each remove the first 5 points of a transect)

8. Max Velocity sometimes useful to remove partial swath transects that were harvest at a faster speed but not marked in the display by the operator Min Velocity one of the most critical to yield cleaning as combine slows, but intake remains high for a few seconds, yield values can become unreasonably high Smooth Velocity filters out areas of rapid velocity changes represents an allowable ratio of velocity change from one point to the next default ratio of 0.2 mean that points not within 20% of the speed of neighbors will be removed

9. Minimum Swath only useful to us if the operator has recorded partial swaths while harvesting have not seen this in the data we have Min/Max Yield Standard Deviation used to remove values that are more/less than a specified number of STDs from the field-wide mean Header Down Requirement removes values where the header was up, but still recording often this is filtered out by default within commercial software have not seen this in the data we have Position removes any ‘fliers’ which are out of the area of interest Adjust for Moisture doesn’t remove data

10. Moisture Adjustments allows for modification of yield values based on moisture status of the grain it is a complicated decision that involves several considerations: “First, if we are going to adjust grain yield back to a market moisture level, will we use the moisture sensor data to make this correction? Sometimes this is not a good idea, as the moisture sensor can have significant problems that may cause very large (and incorrect) adjustments to yield. Even if we do use the sensor, we will need a manual value to use in problem areas (the end of transects, etc.). Another approach is to use a constant manual value to make the correction across the whole field (for example, the moisture value reported at the point of sale). This approach is less prone to major sensor based errors, but may miss real variations that exist within a field. A final option that we need to consider is whether we want to expand the grain yields where the moisture falls below the market moisture level up to the yields they would have been at the market moisture level.”

11. Automatic Yield Cleaning Editor (AYCE)

12. AYCE – auto min/max filters AYCE picks ranges for position, velocity, and yield determined by analysis of histograms Sudduth et. al., 2012; Yield Editor 2.0: Software for Automated Removal of Yield Map Errors

13. AYCE – delay computations Based on Phase Correlation Identification method (Lee et al. 2012) Determines the spatial consistency of flow times at each interval Repeats 10 times – blue line is the ‘normalized’ average

14. AYCE – overlap filter Based on bitmap method developed by Han et al. 1997 Removes observations where the combine travelled over previously harvested areas Extremely effective for fields harvested with one combine Not suitable for fields harvested with multiple combines unless able to sort all records among all combines chronologically Limited by poor GPS accuracy

15. AYCE – Localized STD filter (LSD) Effective at catching fliers missed by yield and velocity filters, as well as reducing ramping at border between end rows and interior field transects Searches an area equal to user-specified neighborhood (# of header widths) and filtering by a user-specified number of STD

16. AYCE – Performance compared to manual filtering (Sudduth et al. 2012) Yield cleaning methods compared on 50 yield maps high variation of field configurations and harvest patterns: multiple combines, operators, yield monitors, variability in harvesting conditions, and a range of required delay predictions Comparisons by ‘expert’ yield cleaners and automatic methods (using default settings) produced comparable results Of yield observations retained by the two methods, 95% were in common

17. AYCE – Performance compared to manual filtering Regressions on values (field mean yield, STD, # rem. obs.) were similar for the two cleaning methods

18. Advanced Editing To be used after AYCE Masks the effects of previously applied filters in small areas where it is not realistic to have applied them For Example: we have a field where a start delay of 7 seconds and an end delay of 12 seconds was applied to all transects, but there is a ditch in the middle of the field where the header was raised briefly and then lowered. From the previous filters the 7 values after and the 12 values before the header was raised to cross the ditch are thrown out but are actually valid values. We can use Advanced Editing to ‘mask’ these small areas from various filters.

19. Recommended Procedures Start with automated cleaning then refine using manual filters and lastly, used advanced editing methods

20. Image before automated cleaning

21. After automated cleaning 14% of values removed

22. Automated cleaning + my filters 15.5% of values removed