1. Investigation of Macro Editing Techniques for Outlier Detection in Survey Data
Katherine Jenny Thompson
Office of Statistical Methods and Research for Economic Programs

2. Simplified Survey Processing Cycle
Data Collection/
Analyst Review
Micro-editing
And Imputation
Individual Returns
Macro-editing
Tabulated Initial Estimates
Publication
Estimates
Analyst Investigation
And Correction

3. Identifying Outlying Estimates
Set of Estimates
Unknown parametric distribution (robust)
Contains outliers (resistant)
Outlier-identification problems (Multiple Outliers)
Masking: difficult to detect an individual outlier
Swamping: too many false outliers flagged

4. Outlier Detection Approaches
Sets of “bivariate” (Ratio) comparisons
Same estimate from two consecutive collection periods (historic cell ratios)
Different estimates in same collection period (current cell ratios)
Multivariate comparisons
Current period data

5. Method for Bivariate Comparisons
Resistant Fences Methods
Symmetrized Resistant fences
Asymmetric Fences
Robust Regression
Hidiroglou-Berthelot Edit

6. Bivariate Comparisons (Current Cell Ratios)
Linear relationship between payroll and employment
No intercept

7. “Traditional” Ratio Edit (Current Cell Ratio)
Outlier Region
Acceptance Region
Outlier Region
“Cone-shaped” tolerances
Goes through origin
Strong statistical association

8. Resistant Fences Methods
q25-1.5H
q75+1.5H
q25
q75
Different numbers of interquartile ranges (1.5 = Inner, 3 = Outer)
Implicitly assumes symmetry
May want to “symmetrize”, apply rule, use inverse transformation

9. Asymmetric Fences Methods
q25+3 (m – q25)
q75+3 (q75- m)
Different numbers of interquartile ranges (3 = Inner, 6 = Outer)
Incorporates skewness of distribution in outlier rule (“Fences”)

10. Robust Regression Resistant (minimizes median residual)
Least Trimmed Squares Robust Regression
Resistant (minimizes median residual)
Outlier = |residual|  3  robust M.S.E.

11. Issue at Origin (Historic Cell Ratio)
Alternative Approach (Ratio Editing) --Hidiroglou-Berthelot (HB) Edit
Originally designed to detect outlying values in periodically collected micro-data
Requires “complete” set of micro-data in an “industry”
Characterized by dynamic tolerances

12. Hidiroglou-Berthelot (HB) Edit
Accounts for magnitude of unit (variability at origin)

13. Hidiroglou-Berthelot (HB) Edit
Two-step transformation (Ei)
Centering transformation on ratios
Magnitude transformation that accounts for the relative importance of large cases
Asymmetric Fences “Type” Outlier Rule
Key Parameter
U = magnitude transformation parameter (0  U  1)
C = controls width of outlier region

14. Multivariate Methods: Mahalanobis Distance
Multivariate normal (,)
T(X) estimates 
C(X) estimates 
p is the number of distinct variables (items)
Prone to masking (difficult to detect individual outliers)

15. Robust Alternatives M-estimation (not considered) “Production Method”
Minimum Volume Ellipse (MVE)
Resistant (50% breakdown) and robust
Minimum Covariance Determinant (MCD)
Assumption of Normality
Log-transformation

16. Evaluation: Classify Item Estimates
Input Value
Reported
Final Value
Tabulated
Ratio
Input/Final
Not an Outlier
Potential
Outlier
Outlier

17. Evaluation: Classify Ratios (Bivariate)
Conservative
Ratio is “outlier” if numerator or denominator is an outlier
Anti-Conservative
Ratio is “outlier” if numerator or denominator is an outlier or a potential outlier

18. Evaluation: Classify Records (Multivariate)
Conservative
Record is “outlier” at least one estimate is an outlier
Anti-Conservative
Record is “outlier” at least one estimate is an outlier or a potential outlier

19. Evaluation Statistics: Bivariate Comparisons
Individual Test Level
Type I Error Rate: proportion of false rejects
Type II Error Rate: proportion of false accepts
Hit Rate: proportion of flagged estimates that are outliers
All-Test Level
All-item Type II error rate

20. Evaluation Statistics: Multivariate Comparisons
Type I error rate: the proportion of non-outlier records that are flagged as outliers
Type II error rate: the proportion of outlier records that are not flagged as outliers (missed “bad” values)

21. Annual Capital Expenditures Survey (ACES)
Sample Survey (Stratified SRS-WOR)
ACE-1: Employer companies
ACE-2: Non-employer companies (not discussed)
New sample selection each year
Total and year-to-year change estimates
Total Capital Expenditures
Structures (New and Used)
Equipment (New and Used)

22. Capital Expenditures Data
Characterized by
Low year-to-year correlation (same company)
Weak association with available auxiliary data
Editing procedures focus on additivity
Outlier correction at micro-level

23. Bivariate Comparisons
Robust Regression
Resistant Fences
HB Edit
Structures/Total 
New Structures/Structures
New Structures/Used Structures
Equipment/Total
New Equipment/Equipment
Resistant Fences: (Symmetric or Asymmetric)  (Inner or Outer)
HB Edit: (U = 0.3 or 0.5)  (c = 10 or 20 )

24. Results – Individual Tests
Robust Regression prone to swamping
High Type I error rate (false rejects)
Comparable performance with Asymmetric Inner Fences and HB Edit (U = 0.3, c = 10)
Low Type I error rates
High Hit Rates
High Type II error rates
Other variations of Resistant Fences and HB edit not as good

25. Results – All-Tests Very large Type II error rates (approx. 50%)
Robust regression
Symmetric resistant outer fences
HB edit with c = 20
Improved Type II error rates (30% - 40%)
Asymmetric inner fences
HB edit (U = 0.3, C=10)

26. Multivariate Results Original Data: considered methods ineffective
Log-transformed data: improved performance (MCD and MVE)
Reduced Type I error rates
Comparable Type II error rates (to original-data MCD and MVE)

27. Multivariate Versus Bivariate: Different Outcomes (Conservative)
Combined HB edits flag more “outliers”:
Higher Type I error rate
Lower Type II error rates for the complete set of HB edits

28. Comments
Economic data with inconsistent statistical association between items in each collection period
Critical values must be determined by the data set at hand (no “hard-coding”)
Dynamically
Standardize the comparisons (HB edit, log transformation)
Compute outlier limits
Could try hybrid approach:
Multivariate  a few current cell ratio tests with the HB edit
Perform all bivariate tests, but unduplicate cells before analyst review

29. Final Thoughts/Next Steps
Examine one set of economic data and considered only two separate collections from this program.
Extrapolation would be foolish
My results need to be validated on other economic data sets
a more typical periodic business survey and/or
a well-constructed simulation study

30. Any Questions? Katherine Jenny Thompson