AgMIP SSA Meeting Accra, Ghana 12 September, 2012 Importing and translating crop model data

AgMIP IT Products Websites Public site (under renovation) Research site research.agmip.orgresearch.agmip.org AgMIP toolshed tools.agmip.org (under construction)tools.agmip.org Tools Climate tools Crop model data translation tools Web and desktop interfaces for modeling tools Open source repositories github.org/agmipgithub.org/agmip Databases Crop Experiment / Sentinel site (operational) Crop model outputs (operational) Crop scenarios (conceptual) Gridded data (conceptual)

AgMIP harmonized databases Model data stored in a consistent format Filters allow import of data from various model-ready files and other data sources Translators allow export of data to various model-ready formats

Web-based data search

Guide for Regional Integrated Assessments Handbook Item 3: Assemble Existing Data from Experiments and Calibrate Crop Models Item 5: Assemble Data and Calibrate Crop Models for Regional Yield Simulations and Assessments Item 10: Archive Data and Analyses Results for Integrated Assessments

Types of data Data rich field experiments for calibration of cropping system parameters sufficiently complete to parameterize crop models Survey data – evaluation of baseline vs future conditions may be incomplete assumptions based on local knowledge of agricultural practices

Steps to data preparation and translation 1. Collect raw data 2. Using spreadsheet templates, transform the data to harmonized units and variables – weather, soils, and field / management / metadata 3. Save data in csv format files 4. QuadUI – desktop application for data translation to DSSAT and APSIM formats 5. View assumptions and errors (repeat steps 2-5 as needed) 6. Run models 7. Collect output data in ACMO.csv format for input to economic models

Step 1 – Collect raw data Machakos data as an example (Survey.xls)

Step 2 – Prepare crop model data 3 ways to input data DSSAT format (use Xbuild and other DSSAT tools to prepare crop model files). These can be translated directly into AgMIP harmonized format. Use spreadsheet templates to fill in data with proper ICASA variable names and units. multiple templates will allow different kinds of data User Interface for data entry – under development

Step 2 – Survey data template

Step 2 – Survey data template (continued) Excel spreadsheet for calculations, conversions, date formats One line per site for metadata, field, and management data separate csv file for soil data separate csv file for weather data separate csv file for regional assumed management data (initial soil conditions, fertilizer rates, planting conditions) See sample template (Machakos_import_2.5.xlsx)

ICASA Master Variable List Harmonized database uses ICASA Master Variable List to interpret data definitions and units. See ICASA_Master_Variables_10Sep2012.xlsx or online (always up-to-date) at NThFSnlGWGlkTjgxWXRwbWc&output=html NThFSnlGWGlkTjgxWXRwbWc&output=html living document – go to web version for the latest adding variables to spreadsheet templates: use code_display column from ICASA list

Step 3 - Save csv format files Separate csv file for each data type: soil weather management / field / metadata (initial conditions, observed data) regional assumptions Beware of date formats when you re-open a csv file with Excel! Zip all csv files together for import with QuadUI

Step 4 - QuaDUI Quick and Dirty User Interface Temporary desktop application for conversion of spreadsheet templates or DSSAT-format files to AgMIP harmonized format, APSIM or DSSAT formats. Near future: web-based application to upload and archive data, search and query, download in multiple formats.

Step 4 – QuaDUI (continued) Open by double-clicking the quadui.bat file and selecting the zip file and location for output files.

Step 4 – QuaDUI (continued) Select format(s) for conversion DSSAT APSIM JSON Currently working on additional translators for: STICS Aquacrop InfoCrop EPIC ORYZA CropSyst Salus WOFOST RZWQM Need to add SARRAH translator! Hit “Convert” button

Step 5 – QuaDUI troubleshooting Look at crop model files for notes on assumptions made by the translators Look at QuadUI log file (where the QuadUI jar file is located) for notes on invalid variables Application requires that headers are spelled exactly as in code_display of ICASA Master Variable List (not case- sensitive) Variables must appear in correct file (e.g., no weather data in soils file)

Step 6 – Run Models DSSAT – bat file provided which will run the experiments created. APSIM – use APSIM UI

Step 7 - Crop model outputs Machakos_ACMO.csv file – format for harmonization of output from crop models.

Spreadsheet template for sentinel site data UFGA8201_MZX_trt1.xlsx Legend: SymbolLocationDescription !Column 1Comment # Column 1, headers Summary data variable names % Column 1, headers Series data variable names (time series or soil layer data) Unique id (1, 2, A, B, …) First field of data Relational data requires unique id to show relationships between summary and series sections of data *First field of dataComplete data (one line per experiment)