Pre-processing of NIR Åsmund Rinnan

Introduction NIR Fructose Glucose 25 50 75 75 50 25

Introduction NIR Baseline with a slope/ curve Nonlinearity Fructose Glucose 25 50 75 75 50 25 A “fat” baseline

Introduction Techniques Reference dependent O-PLS OSC OS SIS Reference independent MSC/ ISC EMSC/ EISC SNV Detrend Normalization Savitzky-Golay Norris-Williams Finite difference

Introduction Techniques Correction of light scatter MSC EMSC Detrend SNV Derivation Finite difference Savitzky-Golay

Effect of pre-processing Before Specular effect

Effect of pre-processing After

MSC Raw Reference

MSC Raw Reference

MSC Raw spectrum b = Slope a = Intercept Reference P Geladi, D MacDougal, H Martens (1985): Linearization and scatter correction for near-infrared reflectance spectra of meat, Applied Spectroscopy, 39, 491-500

MSC

Extended MSC Wavelength Slope & Intercept Known spectra correction = = Basic MSC = Detrend H. Martens, E. Stark (1991): Extended multiplicative signal correction and spectral interference subtraction: new preprocessing methods for near infrared spectroscopy, Journal of Pharmaceutical and Biomedicinal Analysis, 9, 625-635

Extended MSC The correction Calculated Step 1: Step 2: H. Martens, E. Stark (1991): Extended multiplicative signal correction and spectral interference subtraction: new preprocessing methods for near infrared spectroscopy, Journal of Pharmaceutical and Biomedicinal Analysis, 9, 625-635

X X Extended MSC Summary Parameter setting Paraemter setting MSC Detrend

SNV R.J. Barnes, M.S. Dhanoa, S.J. Lister (1989): Standard Normal Variate Transformation and De-trending of Near-Infrared Diffuse Reflectance Spectra, Applied Spectroscopy, 43, 772-777

SNV vs MSC MSC SNV

SNV vs. MSC Classificafication of barley

SNV vs. MSC SNV MSC Normalization Reference needed No reference Outlier sensitive Only similar spectra No normalization

Derivation

Derivation Finite difference

Savitzky-Golay vs. Finite difference Derivative Method 1 2 3 4 1st SG 0.38 (5) 0.40 (5) 0.47 (6) 0.46 (3) Finite difference 0.34 (5) 0.58 (4) 0.73 (4) 2nd 0.34 (6) 0.51 (4) 0.67 (5) 0.72 (4) 0.88 (5) 0.91 (5) 0.42 (6)

Derivation Savitzky-Golay A Savitsky, M J E Golay (1964): Smoothing and differentiation of data by simplified least squares procedures, Analytical Chemistry, 36 (8), 1627-1639

Derivation Savitzky-Golay Smoothing 11 9 7 5 3 A Savitsky, M J E Golay (1964): Smoothing and differentiation of data by simplified least squares procedures, Analytical Chemistry, 36 (8), 1627-1639

Derivation Savitzky-Golay Smoothing Polynomial 7 4 2 5 1 3 A Savitsky, M J E Golay (1964): Smoothing and differentiation of data by simplified least squares procedures, Analytical Chemistry, 36 (8), 1627-1639

Derivation Savitzky-Golay 7 point smoothing 15 point smoothing 2nd order derivative

Summary Pre-processing of NIR Run the following for near to optimal results MSC with Only 1st order reference correction or SNV 1st order reference correction and 2nd order wavelength correction Only 2nd order wavelength correction Savitzky-Golay derivation with 2nd order polynomial smoothing 7 points smoothing for the 1st derivative 9 points smoothing for the 2nd derivative

Compression factor Barley data Pre-processing MB MH Raw 0.0 % MSC1 88.0 % 90.4 % MSC2 92.3 % 94.4 % SNV SNV + Detrend 90.9 % 93.9 % SG1 91.6 % 92.8 % SG2 93.3 %