1. Gap filling of eddy fluxes with artificial neural networks
Dario Papale, Antje Moffat, Markus Reichstein

2. Artificial Neural Networks - ANNs
What is an ANN
It is a network of many simple processors ("units"), each possibly having a small amount of local memory.
The units are connected by communication channels ("connections") which usually carry numeric (as opposed to symbolic) data, encoded by any of various means. The units operate only on their local data and on the inputs they receive via the connections.
Input dataset
Output
Input layer
Hidden layer
Output layer
= unit
= connection

3. Σ The unit or node Transfer functions ni = previous layer nodesnn w1 w2 wn x1 x2
wn xn
w2 x2
w1 x1 xn Σ g a y 1 b
ni = previous layer nodes
xi = input to the unit
wi = connection weigh
g = transfer function
y = output from the unit

4. Multilayer Feed-forward Back propagation ANN (BPN)
BPN are a class of ANN with these characteristics:
They are able to model also complex relationships
The ANN training is supervised
The nodes are organized in layers
The information flow is unidirectional from the input layer to the output layer
There are not connections between nodes of the same layer
The error in the output estimate is retro-propagated from the output layer to the input layer

5. Multilayer Feed-forward Back propagation ANN (BPN)
It has been demonstrated that a BPN with one hidden layer and sigmoid transfer functions is able to approximate any kind of continuous function with any degree of accuracy if there are a sufficient number of hidden nodes in the hidden layer (Bishop 1995, Hagan et al. 1996, Rzempoluck 1998)
Mat_dem

6. BPN: how it works
The ANN is trained using a dataset of examples with input and correct output.
For each example the ANN-output is compared with the correct-output and the error is minimized modifying the connection weighs
Once the ANN has been trained, the connections’ weighs are fixed and it is possible to use the Network with only the input data
To avoid overfitting, the dataset used in the ANN training is split in 3 subsets:
Training set
Test set
Validation set
Epochs
Error p

7. How to back propagate the error?
There are different training algorithms available. The simplest but also most used is based on the gradient descent technique.
Considering the connection between node k and node j :
wkj
Ekj a b

8. Artificial Neural Networks training
Input and Output database
Used to assess the connection weights
Used to evaluate the errors
Validation set
Training set
Test set
Input dataset
Output
Input layer
Hidden layer
Output layer
= node or unit
= connection
Used to evaluate the trained ANN
Trained ANN

9. There are a lot of tools available, often available for free, simple and funny to use
There are also toolboxes or codes that it is possible to integrate in your software (I’m using the Matlab toolbox)
However, like for all the methods, you can use it or use it correctly (or in a better way…), and this make the difference in the performances…
So let’s look more in depth some characteristics of the ANNs…

10. ANNs are quite robust in the application if input are wrong or noisy, but they are sensible to wrong data during the training process. The dataset used to train the ANN must be with high quality in both, input and output
ANN training is based on an iterative process and you will not obtain two times the same result from the same network trained with the same data
ANNs are very good in interpolation but bad in extrapolation…
Generally, the common questions when you start to use ANNs are:
Which variables should I use as input?
How many examples do I need?
How many of these examples do I have to put in the three sets?
Which ANN architecture should I use?

11. Which variables should I use as input?
Site dependent and different between night and day. Ideally Radiation, Temperature, VPD and Soil Water Content for daytime, no Radiation but Soil temperature or Precipitation in the night.

12. How many examples do I need?
As much as possible…. But the ANNs are not good in extrapolation so it is important to cover the domain UNIFORMLY.
Year split in 28 sections (13 days)
Daytime and Nighttime separately, both split in three sections
84 periods per year
Same number of example from each period in the training set
Extracted randomly
Test set = 50% of the training set, Valid. set the rest of data

13. Each network trained 10 times starting from different initial weights
Which ANN architecture should I use?
The simplest with the best performances. It is important to train different ANNs with different architectures
4, 5, 6, 7, 8, 4+3 hidden nodes
Best performance based on observed vs modeled regression (Validation set) and on MAE and RMSE
Each network trained 10 times starting from different initial weights
If two ANNs have the same performances (+/- 1%) the simplest is selected

14. But everything starts with the random extraction of the dataset…
Bootstrapping would be very interesting but too slow… More extractions are better than nothing…
Training, Test and Validation sets are randomly extracted 7 times
For each dataset all the process is repeated
At the and there will be 7 ANNs selected as “best”
The worst is removed and the other 6 applied
The final value used to fill each gap is the mean of the 6 results

15. 2 7 6 10 2 x 7 x 6 x 10 = 840 Dataset split in daytime and nightime
Network trained!
(for one storage!!)
Input variables selected
Yearly dataset split in 84 periods
Same number of example extracted randomly from each period
Preprocessing (scale between 0 and 1, chose generalization method between early stopping and regulation)
6 different ANNs are trained
Each ANN is trained 10 times
6 best networks applied
If gaps in the input variables -> gapfilled with MDS
If gaps impossible to fill -> MDS then MDV…..
Average of the 6 results for each gap
Qflags added + std of the 6 ANN output

16. Next steps to improve gapfilling the method
Method: ANN
Training data from three years: 97, 98 and 99.
Applied with input data from 2001