11 july 2008 European Conference on Quality COMPARISON OF VALIDATION PROCEDURES TO DETECT MEASUREMENT ERRORS IN AN AREA FRAME SAMPLE SURVEY Laura Martino, Marco Fritz, Marjo Kasanko & Javier Gallego European Conference on quality in official statistics, Rome 8-11th July 2008

11 july 2008 European Conference on Quality Contents  Scope of the work  Methods for measurement errors detection  The case study: LUCAS survey  Conclusions

11 july 2008 European Conference on Quality Scope of the work The scope of the work is comparing the efficacy of three selected procedures to detect measurement errors in an area frame sample survey that uses a combination of orthophoto interpretation and ground survey

11 july 2008 European Conference on Quality Methods for detection of measurement errors Three methods have been selected to be compared:  double-blind survey;  check of ground data with ausiliary data sources;  cross-check of orto-photo interpretation and ground- survey outcomes to evaluate the accuracy of stratification

11 july 2008 European Conference on Quality Area frame sample survey carried out by Eurostat since Main objectives are providing: a)coherent and harmonised statistics on land use and land cover; b) a common sampling base (frame, nomenclature, data treatment) to be used for further scopes; c) ground evidence for calibration of satellite images; d) information on aspects relating to the agro- environment Case study: Land Use and Coverage Area frame sample Survey (LUCAS)

11 july 2008 European Conference on Quality Second phase sample:Ground survey  Record land cover and land use according to the full nomenclature East During the ground survey, surveyors:  Reach the point using GPS and cartographic material  Take pictures (Point, Cov,N,E,S,W,irr)  Walk the transect (250 m. East)

11 july 2008 European Conference on Quality 1° procedure: Double blind survey A second survey was organised in 2006 in parallel to the main LUCAS 2006 survey.  It was conducted by an independent company not in charge of the LUCAS survey in the countries involved.  No information were provided on the results of the main survey.  The double blind-survey sample size represented 5% of the total survey sample and counted almost 8200 points.

11 july 2008 European Conference on Quality Outcomes of double-blind survey The possible outcomes of the double-blind survey were:  Land Cover/Land Use being the same between the two surveys (correct points observed from the same location);  Land Cover/Land Use being different between the two surveys but both correct (different rules of observation - look north, east, etc. - or change in the land cover between the two visits (crops harvested, sown, building built, etc...);  Land Cover/Land Use being different between the two surveys and lack of sufficient information to say which survey is correct;  Land Cover/Land Use being different between the two surveys and the double-blind survey being correct;  Land Cover/Land Use being different between the two surveys and the main survey being correct.

11 july 2008 European Conference on Quality Country Rough data comparison BE 69.86% CZ 80.07% DE 77.49% ES 63.22% FR 73.09% HU 64.89% IT 67.12% PL 65.03% SK 81.25% Main survey 169,000 pts D-B survey 8,200 pts Double blind survey D-B survey introduced as many errors as the main survey

11 july 2008 European Conference on Quality 2° procedure: Use of ausiliary data to correct results Country Rough data comparison Post-processed data comparison BE 69.86%96.58% CZ 80.07%95.73% DE 77.49%98.71% ES 63.22%95.03% FR 73.09%96.36% HU 64.89%98.55% IT 67.12%95.08% PL 65.03%90.62% SK 81.25%97.50% Large improvement in data quality through the use of pictures  IACS data  Pictures  Other georeferenced information

11 july 2008 European Conference on Quality 3° procedure: comparison of orthophoto interpretation and ground survey A comparison of the classification of the points according to the ground observation and the photointerpretation has been conducted in a comparable nomenclature of 7 classes.  A proportion of agreement has been computed  Since points belonging to different strata have been subsampled with probability that could be 5 times larger/lower, a weighted proportion of agreement is computed in addition to the unweighted one.

11 july 2008 European Conference on Quality

11 july 2008 European Conference on Quality Measurements of agreement  Frequency: The unweighted proportion of agreement is 70.8%, while the weighted agreement is 74.8%.  Kappa index (Bishop et al., 1975) where are the proportions of each cell of the table The unweighted Kappa is 0.58 (0.66 if we take into account the strata weights).

11 july 2008 European Conference on Quality Rate of agreement Unweightedweighted CtyagreementKAPPAagreementKAPPA BE CZ DE ES FR HU IT LU NL PL SK

11 july 2008 European Conference on Quality Mainly confused items Some points that are agricultural land are photo-interpreted as non-agricultural for several reasons: location inaccuracy, land cover change or simply photo-interpretation errors The main confusions occured between:  permanent grass and bare land are often photo-interpreted as arable.  Woodland or shrubland interpreted as grass. This may be partly due to problems to apply in practice the definition of forest or woodland

11 july 2008 European Conference on Quality Conclusions  The pre-eminent result of the double blind survey was that it introduced as much errors as the main survey for many reasons  The analysis of pictures and the use of auxiliary information appeared to be fundamental to detect measurement errors and improve data quality  Orthophoto interpretation produce some inefficiency (stratification accuracy) -> poststratification???

11 july 2008 European Conference on Quality THANK YOU FOR YOUR ATTENTION!!!

11 july 2008 European Conference on Quality 200 metres Comparison with Corine Land Cover 2000 Mapping based on satellite images and orthophoto interpretation

11 july 2008 European Conference on Quality LUCAS and CLC

11 july 2008 European Conference on Quality Visual metadata