Lufthansa Outlier Detection Methods on Booking Data AGIFORS Reservation and Yield Management Study Group Bangkok May 2001 Ulrich Oppitz

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 2 Lufthansa Outlier Detection Methods on Booking Data - Agenda - Definitions and TheoryOutlier Detection MethodsAnalysis MethodSome Words on Quality MeasurementResultsSummaryLiterature

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 3 Lufthansa Booking data in RM systems can be influenced by many disturbances Definition: Outliers are data points which differ in their appearance from the majority of the data. (Rousseeow, 1990) Caused by: system errors schedule changes special events Two approaches to cope with outliers: robust approach: –use robust methods/predictors diagnostic approach: –identify outliers –trimm or ignore them –apply classical methods/predictors Best practice for chain processes

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 4 Lufthansa If ignored, outliers can affect the quality of the forecasting process significantly To measure the robustness of a forecast method, Hodges introduced the term breakdown point. (Hodges 1967) The breakdown point can be loosely defined as the smallest fraction of outliers that seriously offsets the estimator from the true one. (Rousseeuw 1991) The breakdown point of any regression method based on the least squares technique is 1/n, which means a single outlier in a set of n data points can degenerate the LS estimate.

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 5 Lufthansa Outlier Detection Methods on Booking Data - Agenda - Definitions and Theory Outlier Detection Methods Analysis Method Some Words on Quality Measurement Results Summary Literature

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 6 Lufthansa Z-Score Testing calculate empirical average  and variance  based on historical bookings for each DCP check whether number of historical bookings > minimum observations tag as outlying if outside the following interval upper threshold:  +maxSigmaPos*  lower threshold:  -maxSigmaNeg*  trimm outlying data to threshold value before updating  and 

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 7 Lufthansa Z-Score Testing 0 0,05 0,1 0,15  bkgs density function of normal distribution lower bound upper bound

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 8 Lufthansa Determination Coefficient Testing on Residual Regression update exponentially smoothed bookings for each dcp -> reference curve check whether number of historical bookings > minimum observations calculate residuals  bkd(dcp) from actual bookings and reference curve calculate linear regression curve reg(dcp) on residuals  bkd(dcp)

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 9 Lufthansa Determination Coefficient Testing on Residual Regression  bkd(dcp) reg(dcp) dcp

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 10 Lufthansa calculate the determination coefficient Determination Coefficient Testing on Residual Regression  (reg (dcp) - reg)  (  bkd (dcp) -  bkd ) R = if R 2 < minR 2 tag dcp with largest vertical distance to regression curve as outlying and take it out of the set iterate with cleaned data set stop if R 2 > minR 2 or number of outlier > maxOutlier reset outlier taggings if more than maxOutlier

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 11 Lufthansa Outlier Detection Methods on Booking Data - Agenda - Definitions and Theory Outlier Detection Methods Analysis Method Some Words on Quality Measurement Results Summary Literature

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 12 Lufthansa The simulation is performed on real booking data 42 flight numbers (2 multi-leg flights) data type:actual bookings data source:PROS IV data base departure time range:01Jun May97 booking classes:FA CDZ HBLGYKTWE evaluated DCPs:1-15 total flight departes: total DCPs:

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 13 Lufthansa Analysis method: artificial outlier implantation 1) Preprocessing: outlier cleaning with very conservative parameters (high outlier tagging rates) 2)Different manipulations are performed with predefined probabilities XLAenlarge all DCPs x 3.00P XLA = 0.01 XSAshrink all DCPsx 0.33P XSA = 0.01 XL1enlarge single DCPx 3.00P XL1 = 0.01 XS1shrink single DCPx 0.33P XS1 = 0.01 X-Yswap booking classes X and YP X-Y = )Artificially created outliers are tagged. 4)Apply outlier detection method 5)Evaluation: count number of recognized outliers and non-outliers

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 14 Lufthansa Outlier Detection Methods on Booking Data - Agenda - Definitions and Theory Outlier Detection Methods Analysis Method Some Words on Quality Measurement Results Summary Literature

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 15 Lufthansa observables:True PositivesTP True NegativesTN False PositivesFP False NegativesFN TP sensitivity 1 :TP + FN=: sens (masking) TN specificity 1 :TN + FP=: spec (swamping) TP + TN efficiency 1 :TN + FN + TP + FP=: eff TP + FP temperament:TN + FN + TP + FP=: temp The quality measures known in the literature are not sufficient in the RM environment. 1 (Walczak, 1998)

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 16 Lufthansa Quality Measures for Outlier Detection Methods For an outlier detection method on booking data it is most important to detect almost all outliers. Few data points which are erroneously taken out of the valid set, have less impact. weighting of error types TP and TN dynamical adaption of weights to degree of contamination axioms for a quality measure  let A,B  Â denote the complex set of correct classifications, 0 <=  (A) <= 1  (A) = 0  A =   (A) = 1  A= Â A  B   (A) <  (B)  ( A  B) =  (A) +  (B) -  (A  B)

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 17 Lufthansa  TN + (1-  )TP  (TN+FP) + (1-  ) (TP+FN) TP + FN TN + FN + TP + FP (outlier rate)  TN + (1-  )TP  (TN+FP) + (1-  ) (TP+FN) TP + FP TN + FN + TP + FP (temperament) Contamination and Temperament Weighted Efficiency meet the conditions CWE = TWE = with  =

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 18 Lufthansa Outlier Detection Methods on Booking Data - Agenda - Definitions and Theory Outlier Detection Methods Analysis Method Some Words on Quality Measurement Results Summary Literature

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 19 Lufthansa temperament, z-score testing Sensitivity Analysis on Cleaned Booking Data - temperament for z-score testing -

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 20 Lufthansa Sensitivity Analysis on Cleaned Booking Data - sensitivity for z-score testing - sensitivity, z-score testing

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 21 Lufthansa specificity, z-score testing Sensitivity Analysis on Cleaned Booking Data - specificity for z-score testing -

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 22 Lufthansa efficiency, z-score testing Sensitivity Analysis on Cleaned Booking Data - efficiency for z-score testing -

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 23 Lufthansa Max: (0.9, 0.6, ) CWE, z-score testing Sensitivity Analysis on Cleaned Booking Data - contamination weighted efficiency for z-score testing -

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 24 Lufthansa Max: (0.9, 0.6, ) TWE, z-score testing Sensitivity Analysis on Cleaned Booking Data - temperament weighted efficiency for z-score testing -

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 25 Lufthansa temperament, DCT Sensitivity Analysis on Cleaned Booking Data - temperament for DCT - min R 2 max outlier

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 26 Lufthansa sensitivity, DCT Sensitivity Analysis on Cleaned Booking Data - sensitivity for DCT - min R 2 max outlier

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 27 Lufthansa specificity, DCT Sensitivity Analysis on Cleaned Booking Data - specificity for DCT - min R 2 max outlier

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 28 Lufthansa efficiency, DCT Sensitivity Analysis on Cleaned Booking Data - efficiency for DCT - min R 2 max outlier

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 29 Lufthansa Max: (0.45, 14, ) CWE, DCT Sensitivity Analysis on Cleaned Booking Data - contamination weighted efficiency for DCT - min R 2 max outlier

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 30 Lufthansa Max: (0.5, 14, ) TWE, DCT Sensitivity Analysis on Cleaned Booking Data - temperament weighted efficiency for DCT - min R 2 max outlier

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 31 Lufthansa Raw data analysis delivers more realistic results Optimal Parameters on Cleaned and Raw Booking Data z-score testing (ZST) cleaned dataraw data CWE0.9 / 0.6 -> / 0.8 -> TWE0.9 / 0.6 -> / 0.9 -> determination coefficient testing (DCT) cleaned dataraw data CWE0.45 / 14 -> / 14 -> TWE0.50 / 14 -> / 13 -> 0.747

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 32 Lufthansa Proper parameter calibration is more important than method choice. Comparison on raw data

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 33 Lufthansa Z-score testing on booking changes is more efficient than on booking values. Optimal Parameters on Raw Booking Data z-score testing (ZST) on bookingson booking changes CTW1.5 / 0.8 -> / 1.1 -> DTW2.2 / 0.9 -> / 1.5 -> 0.820

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 34 Lufthansa Outlier Detection Methods on Booking Data - Agenda - Definitions and Theory Outlier Detection Methods Analysis Method Some Words on Quality Measurement Results Summary Literature

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 35 Lufthansa We defined new quality measures for outlier detection models which enable a parameter optimization and the comparison of different methods. Symmetric acceptance ranges for z-score testing are of disadvantage –potential for improvement by only adjusting parameters –revenue impact unknown, but positive –low risk Clear superiority of z-score testing on cleaned booking data Slight superiority of z-score testing on raw booking data Parameter optimization incorporates higher potential for improvement than choice of method. Z-score testing can be improved if applied on booking changes Outlier Detection Methods on Booking Data - Summary -

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 36 Lufthansa Outlier Detection Methods on Booking Data - Agenda - Definitions and Theory Outlier Detection Methods Analysis Method Some Words on Quality Measurement Results Summary Literature

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 37 Lufthansa Hodges 1967 J.L. Hodges, Proc. Fifth Berkeleley Symp. Math. Stat. Probab., 1967, 1, Rousseeuw 1987 P.J. Rousseeuw, A.M. Lerroy, Robust Regression and Outlier Detection, Wiley, New York, 1987 Rousseeuw 1990 P.J. Rousseeuw, Unmasking Multivariate Outliers and Leverage Points (with discussion), Journal of the American Statistical Association, 1990, 85, Outlier Detection Methods on Booking Data - Literature -

Outlier Detection Methods on Booking Data Ulrich Oppitz, May 2001, Page 38 Lufthansa Rousseeuw 1991, P.J. Rousseeuw, Journal of Chemometrics, 1991, 5, 1-20 Walczak 1998, B. Walczak, D.L. Massart, Multiple Outlier Detection Revisited, Chemometrics and Intelligent Laboratory Systems, 1998, 41, 1-15 Outlier Detection Methods on Booking Data - Literature, ctd. -

Lufthansa Outlier Detection Methods on Booking Data AGIFORS Reservation and Yield Management Study Group Bangkok May 2001 Ulrich Oppitz