1. Assessment of Agreement between SkyBit Predictions and On-site Measurements Henry K. Ngugi, PhD. Penn State Fruit Research & Extension Center, Biglerville, PA.

2. Quantitative Epidemiology and Commercial Fruit Production Scientist IndustryOrchardist Adjudicate between the grower and industry to maintain a commercially viable tree fruit industry in PA

3. Diseases of concern in the mid- Atlantic region

5. SkyBit---Marketed by ZedX Inc. Growers pay ~$60 per month for pest management information Fire blight and apple scab models developed by Dr. James W. Travis (Penn State Univ.) Models were never validated How do SkyBit forecasts perform? Assessment of Agreement Between Forecasts Adjudicating : The SkyBit Example

6. How do SkyBit weather data predictions compare with data collected on-site? Gold standard = Campbell Scientific and Spectrum Technologies weather stations at PSU-FREC, Biglerville, PA

7. How do SkyBit data predictions compare with data collected on-site? Perform statistical agreement tests between data collected on-site and SkyBit predictions Lin’s concordance analysis for continuous variables Limits of agreement statistics for continuous variables Concordance tests for categorical variables Reliability of SkyBit Disease Forecasts

8. Agreement in weather data I. Temperature r = 0.988 ρ = 0.987 r = 0.992 ρ = 0.991 Highly significant agreement between SkyBit data and on-site daily mean temperature measurements

9. Shapiro-Wilk test W = 0.986, P = 0.223 i.e., cannot reject normality hypothesis Differences must be normally distributed to compute limits of agreement

10. Confidence limits agreement Over a 4-month period, only in 4 out of 152 days did the SkyBit measurements significantly differ from on-site temp. measurements 95CL = d  1.96  SD, in this case = -1.82 to 1.65

11. Rainfall data: April to June 2009 Good agreement with on-site data from the Spectrum Tech. weather station but SkyBit underestimates the rainfall amounts No agreement with data from National Weather Service r = 0.882 ρ = 0.875 r = 0.405 ρ = 0.403

12. Spectrum Tech. data Lin’s concordance coefficient, ρ = 0.860; r = 0.876 Spectrum Tech. data Lin’s concordance coefficient, ρ = 0.860; r = 0.876 Campbell Sci. data Lin’s concordance coefficient, ρ = 0.746; r = 0.788 Campbell Sci. data Lin’s concordance coefficient, ρ = 0.746; r = 0.788 Wetness hours: April to June 2009

13. Wetness confidence limits analysis Generally good agreement but SkyBit over-estimates in wet days and underestimates in dry Good agreement between the ‘gold standards’ r = 0.929 ρ = 0.919

14. Test for agreement between fire blight predictions (MaryBlyt and SkyBit) Concordance coefficients and McNemar's test statistics for agreement between SkyBit and MaryBlyt fire blight forecasts for Biglerville, PA April and May, 2009 CoefficientEstimateSE x Remark Kendall’s tau-B0.6070.103Moderate Stuart’s tau-C0.4820.112No concordance Cohen's Kappa0.5700.115Moderate Pearson’s correlation0.6070.115Weak McNemar's test y χ 2 = 6.40P = 0.014; df = 1; n = 55 x Asymptotic standard error y For the null hypothesis that the two data sets disagree

15. SkyBit predictions are more cautious Concordance coefficients and McNemar's test statistics for agreement between SkyBit and MaryBlyt fire blight forecasts for Biglerville, PA April and May, 2009 CoefficientEstimateSE x Remark Kendall’s tau-B0.8840.065High Concordance Stuart's tau-C0.8260.086High concordance Cohen's Kappa0.8830.065High concordance Pearson’s correlation0.8840.064Strong McNemar's test y χ 2 = 0.333P = 0.564; df = 1; n = 55 x Asymptotic standard error y For the null hypothesis that the two data sets are disagree When MaryBlight “H” is counted as “I”

16. Disease forecast assessments No agreement between SkyBit and MaryBlyt forecasts ( χ 2 = 6.4; P < 0.011; McNemar’s test) for infection events Agreement only when MaryBlyt ‘H’ is counted as = ‘I’ ( χ 2 = 0.333; P = 0.564) Good agreement in apple scab forecast ( χ 2 = 2.0; P = 0.15 and χ 2 = 3.6; P = 0.058 for Spectrum and Mill table models, respectively)

17. Summary of Results SkyBit delivers reliable data to growers in the mid-Atlantic region  Temperature measurements highly reliable  Underestimates rainfall amounts  Wetness measurements unreliable in very dry or wet conditions SkyBit forecasts for apple scab are as reliable as Mill’s Table or Spectrum model SkyBit fire blight predictions are conservative relative to those of MaryBlyt

18. Acknowledgements People in Ngugi Lab $$$ FOR Ngugi Lab USDA –CSREES SHAP, PSU-CAS Drs. Jim Travis & N.O. Halbrendt