Gap filling of eddy fluxes with artificial neural networks Dario Papale, Antje Moffat, Markus Reichstein

Artificial Neural Networks - ANNs What is an ANN It is a network of many simple processors ("units"), each possibly having a small amount of local memory. The units are connected by communication channels ("connections") which usually carry numeric (as opposed to symbolic) data, encoded by any of various means. The units operate only on their local data and on the inputs they receive via the connections. Input dataset Output Input layer Hidden layer Output layer = unit = connection

Σ The unit or node Transfer functions ni = previous layer nodes nn w1 w2 wn x1 x2 wn xn w2 x2 w1 x1 xn Σ g a y 1 b ni = previous layer nodes xi = input to the unit wi = connection weigh g = transfer function y = output from the unit

Multilayer Feed-forward Back propagation ANN (BPN) BPN are a class of ANN with these characteristics: They are able to model also complex relationships The ANN training is supervised The nodes are organized in layers The information flow is unidirectional from the input layer to the output layer There are not connections between nodes of the same layer The error in the output estimate is retro-propagated from the output layer to the input layer

Multilayer Feed-forward Back propagation ANN (BPN) It has been demonstrated that a BPN with one hidden layer and sigmoid transfer functions is able to approximate any kind of continuous function with any degree of accuracy if there are a sufficient number of hidden nodes in the hidden layer (Bishop 1995, Hagan et al. 1996, Rzempoluck 1998) Mat_dem

BPN: how it works The ANN is trained using a dataset of examples with input and correct output. For each example the ANN-output is compared with the correct-output and the error is minimized modifying the connection weighs Once the ANN has been trained, the connections’ weighs are fixed and it is possible to use the Network with only the input data To avoid overfitting, the dataset used in the ANN training is split in 3 subsets: Training set Test set Validation set Epochs Error p

How to back propagate the error? There are different training algorithms available. The simplest but also most used is based on the gradient descent technique. Considering the connection between node k and node j : wkj Ekj a b

Artificial Neural Networks training Input and Output database Used to assess the connection weights Used to evaluate the errors Validation set Training set Test set Input dataset Output Input layer Hidden layer Output layer = node or unit = connection Used to evaluate the trained ANN Trained ANN

There are a lot of tools available, often available for free, simple and funny to use There are also toolboxes or codes that it is possible to integrate in your software (I’m using the Matlab toolbox) However, like for all the methods, you can use it or use it correctly (or in a better way…), and this make the difference in the performances… So let’s look more in depth some characteristics of the ANNs…

ANNs are quite robust in the application if input are wrong or noisy, but they are sensible to wrong data during the training process. The dataset used to train the ANN must be with high quality in both, input and output ANN training is based on an iterative process and you will not obtain two times the same result from the same network trained with the same data ANNs are very good in interpolation but bad in extrapolation… Generally, the common questions when you start to use ANNs are: Which variables should I use as input? How many examples do I need? How many of these examples do I have to put in the three sets? Which ANN architecture should I use?

Which variables should I use as input? Site dependent and different between night and day. Ideally Radiation, Temperature, VPD and Soil Water Content for daytime, no Radiation but Soil temperature or Precipitation in the night.

How many examples do I need? As much as possible…. But the ANNs are not good in extrapolation so it is important to cover the domain UNIFORMLY. Year split in 28 sections (13 days) Daytime and Nighttime separately, both split in three sections 84 periods per year Same number of example from each period in the training set Extracted randomly Test set = 50% of the training set, Valid. set the rest of data

Each network trained 10 times starting from different initial weights Which ANN architecture should I use? The simplest with the best performances. It is important to train different ANNs with different architectures 4, 5, 6, 7, 8, 4+3 hidden nodes Best performance based on observed vs modeled regression (Validation set) and on MAE and RMSE Each network trained 10 times starting from different initial weights If two ANNs have the same performances (+/- 1%) the simplest is selected

But everything starts with the random extraction of the dataset… Bootstrapping would be very interesting but too slow… More extractions are better than nothing… Training, Test and Validation sets are randomly extracted 7 times For each dataset all the process is repeated At the and there will be 7 ANNs selected as “best” The worst is removed and the other 6 applied The final value used to fill each gap is the mean of the 6 results

2 7 6 10 2 x 7 x 6 x 10 = 840 Dataset split in daytime and nightime Network trained! (for one storage!!) Input variables selected Yearly dataset split in 84 periods Same number of example extracted randomly from each period Preprocessing (scale between 0 and 1, chose generalization method between early stopping and regulation) 6 different ANNs are trained Each ANN is trained 10 times 6 best networks applied If gaps in the input variables -> gapfilled with MDS If gaps impossible to fill -> MDS then MDV….. Average of the 6 results for each gap Qflags added + std of the 6 ANN output

Next steps to improve gapfilling the method Method: ANN Training data from three years: 97, 98 and 99. Applied with input data from 2001