Using FLUXNET data to evaluate land surface models Ray Leuning and Gab Abramowitz 4 – 6 June 2008

CSIRO. Using FLUXNET data to evaluate land surface models Land surface model evaluation framework Reto Stockli’s ‘Model farm’

CSIRO. Using FLUXNET data to evaluate land surface models Schematic diagram of model components from a systems perspective Liu, Y. Q. and Gupta, H. V. (2007). Uncertainty in Hydrologic Modeling: Toward an Integrated Data Assimilation Framework. Water Resources Research 43, W07401, doi: /2006/WR system boundary, B 2. inputs, u 3. initial states, x 0 4. parameters, θ 5. model structure, M 6. model states, x 7. outputs, y Errors in each component affects model performance

CSIRO. Using FLUXNET data to evaluate land surface models Parameter estimation Multiple objective functions possible

CSIRO. Using FLUXNET data to evaluate land surface models Parameter estimation Multiple criteria possible, e.g. λE, NEE The dark line between the two criteria’s minima, α and β, represents the Pareto set

CSIRO. Using FLUXNET data to evaluate land surface models Comparing RMSE of models of varying complexity across sites after parameter optimization Hogue, T. S., Bastidas, L. A., Gupta, H. V., and Sorooshian, S. (2006). Evaluating Model Performance and Parameter Behavior for Varying Levels of Land Surface Model Complexity. Water Resources Research 42, W08430, doi: /2005WR Models Sites Ideal result (0,0) λEλE H

CSIRO. Using FLUXNET data to evaluate land surface models SOLO neural network - cluster analysis Abramowitz, G., Gupta, H., Pitman, A., Wang, Y.P., Leuning, R. and Cleugh, H.A. (2006). Neural Error Regression Diagnosis (NERD): A tool for model bias identification and prognostic data assimilation. Journal of Hydrometeorology, 7:

CSIRO. Using FLUXNET data to evaluate land surface models Poor model performance not just due to poor parameter estimation CABLE with 4 different parameter sets SOLO – cluster analysis observed cable solo

CSIRO. Using FLUXNET data to evaluate land surface models No model or single performance measure is best for all fluxes CABLE, ORCHIDEE, CLM, MLR multiple linear regression, ANN artificial neural network

CSIRO. Using FLUXNET data to evaluate land surface models Model comparisons - average seasonal cycle NEE λEλE H Global default parameters for each PFT used

CSIRO. Using FLUXNET data to evaluate land surface models Model comparisons - average daily cycle NEE λEλE H Global default parameters for each PFT used

CSIRO. Using FLUXNET data to evaluate land surface models PDF’s for NEE, λE & H across 6 sites

CSIRO. Using FLUXNET data to evaluate land surface models NEE Perturbed-parameter ensemble simulations Monthly averages Average diurnal cycle

CSIRO. Using FLUXNET data to evaluate land surface models λE Perturbed-parameter ensemble simulations Monthly averages Average diurnal cycle

CSIRO. Using FLUXNET data to evaluate land surface models H Perturbed-parameter ensemble simulations Monthly averages Average diurnal cycle

CSIRO. Using FLUXNET data to evaluate land surface models Partitioning climate space into 9 SOM nodes S↓ T air q air night S↓ T air q air

CSIRO. Using FLUXNET data to evaluate land surface models NEE PDFs at nodes 7 -9 at Tumbarumba night S↓ T air q air 7 8 9

CSIRO. Using FLUXNET data to evaluate land surface models Suggested set of discussion topics Primary objectives Establish a framework that provides standardised data sets and an agreed set of analytical tools for LSM evaluation Analytical tools should provide a wide range of diagnostic information about LSM performance Datasets specifically formatted for LSM execution and evaluation Specific objectives To detect and eliminate systematic biases in several LSMs in current use To obtain optimal parameter values for LSMs after biases have been diminished or eliminated To evaluate the correlation between key model parameters and bioclimatic space

CSIRO. Using FLUXNET data to evaluate land surface models Tasks for meeting 1 Discuss what form the LSM evaluation framework should take PILPS style? What will be asked of data providers? What will be asked of LS modellers? Agree on a minimal set of LSM flux performance measures (model vs observations vs benchmark): Average diurnal cycle? Average annual cycle (monthly means)? Some type of frequency analysis (wavelet, power spectrum etc)? Conditional analysis (SOM node analysis): Overlap of pdfs Multiple criteria cost function set (mean, rmse, rsq, regression gradient and intercept) Discuss other LSM outputs and datasets useful for process evaluation Discuss ways to include parameter uncertainty in LSM evaluation (c.f. Abramowitz et al., 2008)

CSIRO. Using FLUXNET data to evaluate land surface models Tasks for meeting 2 Discuss options for the most effective way to provide these services Will individual groups do benchmarking, evaluation of model states? Preference for an automated web-based interface and data server Automatic processing through a website? Abramowitz suggests automation of basic LSM performance measure plots, including benchmarking (as in Abramowitz, 2005). Uploaded output from LSM runs in ALMA format netcdf could return standard plots to the user and/or post on website. Model detective work and improvement to be done by individual groups

CSIRO. Using FLUXNET data to evaluate land surface models Data analysis will use: Several current LSMs Quality controlled Fluxnet datasets SOFM (Self-organizing feature maps) analysis to classify bioclimatic data into n2 nodes to evaluate model biases for each node to help the ‘detective work’ of identifying areas of model weaknesses to identify upper-boundary surfaces for stocks of C and N and P in global ecosystems as a function of the n2 climate nodes Benchmarking to compare model predictions at each climate node against multiple linear regression (MLR) estimates

CSIRO. Using FLUXNET data to evaluate land surface models Tools currently available from Abramowitz SOLO (SOFM + MLR) software (Fortran) LSMs ‘Model Farm’ of Reto Stöckli plus CABLE CSV to ALMA netcdf conversion routine (Fortran) Plotting routines in R Fluxnet database in CSV and netcdf formats