GNE: Global NEWS Modeling Environment What it is / what it’s not How it works Components, samples Installation NEWS 2 Implementation Emilio MayorgaMay GNE Training Session 1 Content last edited August 28, 2008

Generic GNE vs. MA Scenarios Specifics Covering all details of Millennium Assessment (MA) Scenarios implementation is not an objective of these training sessions But hands-on sessions and demonstrations will be based on MA 2000 (c00) scenario Will use MA implementation to provide concrete examples

GNE Training Goals Introduce & describe GNE –Design goals, main features, code files Describe a standard GNE model run –Major components, files (code, configuration, and input/output data), organization and conventions Describe input pre-processing and output post- processing Enable “model users” to set up and execute model scenario experiments –Modify input data sources used (especially for MA scenarios) –No Python coding needed Refer to GNE User Manual for more complete details. Sessions provide more interactive training and demonstration, while manual includes extensive details about syntax, usage, installation, etc.

Training Sessions Session 1. Introduction and Broad Overview This Presentation Session 2. Running NEWS models with pre-existing, basin-aggregated input tables Interactive Session, in 2 parts (2 sub-sessions) –Part 1. Model run organization, configuration files, conventions –Part 2. Model execution; changing a model run (“experiments”) Session 3. GIS pre-processing and model input/output post-processing Interactive Session

Session 1 Content, Goals Overall GNE design goals, highlights Why Python? Three GNE User Types –Training focus on “Model Users” (no Python coding needed) GNE vs. end-to-end Global NEWS Run –What is covered by GNE vs. what is not Major Components –Code, configuration files, input/output files, file organization and conventions GNE Model Run folder & files GNE Model Run GIS Pre-Processing (GNE vs. other) Post-Processing (GNE vs. other) Python Installation and Base GNE Setup

Overall GNE design goals, highlights 1.Dissolved models merged into a unified framework, rather than isolated sub-models; particulate models remain separate. 2.Focus on basin-scale calculations. 3.For users not needing to modify model equations, most interactions are handled via structured text configuration files. Together with model equations in code, these files also serve as documentation of input files used and how output was created. 4.Complete model run can be maintained in a folder structure that includes all input data, model code, constants used, configuration files, and output tables. 5.Python/NumPy. Fairly easy to learn. Model equation code implemented in an intuitive, simple array syntax. More complex GNE code handling input/output and overall model coordination hidden away and most users never have to see it. 6.Model run time (including GIS pre-processing) is short or trivial.

Why Python? Free, open source, high-level but general programming language with large user base in all fields of computing. Free = available to everyone. Runs on all major operating systems. Rapidly growing acceptance in science applications, including modeling & GIS. Very clean, simplified syntax that makes it easy to learn, read existing code, and become productive quickly. NumPy “add-on” used to code the model equations; intuitive, simple array syntax similar to Matlab. Large library of default and optional add-ons contributed by users, for all sorts of applications. Preferred language chosen by ESRI for data processing automation in ArcGIS (ArcGIS “GeoProcessing”).

Three GNE User Types 1.Model Users. Setup or change a model run using a set of text configuration files; organize input and output files in a regular, structured fashion. Python skills not required. This training is focused on Model Users! 2.Model Developers. Focus on translating model equations into Python GNE code using simple code tools for variable management provided by the GNE framework. Largely isolated from the “behind-the-scenes”, low-level GNE functionality that handles file read-write and basic error checking. Should be fairly comfortable programming with Python & NumPy. 3.GNE Developers. Modify any aspect of GNE functionality, including model equations. Should be very comfortable programming with Python & NumPy.

GNE vs. end-to-end Global NEWS Run MNP Diffuse sources MNP Point sources UNH Hydrology “Raw”, Heterogeneous GIS Source files Diffuse sources Pre-processed, homogenized GIS Input files Point sources Hydrology Manual steps, Ad-hoc automation Direct Global NEWS inputs Basin-scale Inputs tables Global NEWS outputs Model Outputs GNE Complicated variables other than basin means/sums. Eg, dam retention, FQrem, lithology class Global NEWS model(s) GNE GIS pre-processing Summaries, Analysis, Maps Mass fluxes Other post- processing GNE post- processing

Major Components Overall folder organization –Model run code, configuration, input/output files, etc. Python Code –GNE “core” (file read/write, etc.) –Master controller (globalnews.py). –Science code (Global NEWS model): dissolved.py & particulate.py –Loads post-processor (postprocess.py) –Other, optional supporting files (batch runs, etc.) Configuration files (text/ASCII) –Base folder paths (gensetup.cfg) –Dissolved calibration constants (constants.cfg) –GIS basin inputs pre-processing (gis2tbls.cfg) –Global NEWS model run (vars.cfg) –Loads post-processor (also vars.cfg)

GNE Model Run Folder & Files Files in base folder Base folder and sub-folders Optional files (also in base folder, or in OTHERFILES) Also, documentation files.

GNE Model Run Folder & Files: Python code Files in base folder Base folder and sub-folders GNE Core code Science (model) code Master controller Optional GIS pre-processing add-on Loads post-processor

GNE Model Run Folder & Files: Configuration files Files in base folder GIS pre-processing Model run and loads post-processing Dissolved calibration constants Base folder paths

GNE Model Run Folder & Files: Input and output files Base folder and sub-folders Basin-scale model output and post-processed tables Pre-processed basin-scale input tables Homogenized raster GIS inputs Basins, cell areas, and other basic input data (raster GIS and basin-scale tables)

MNP Diffuse sources MNP Point sources UNH Hydrology “Raw”, Heterogeneous GIS Source files Diffuse sources Pre-processed, homogenized GIS Input files Point sources Hydrology Manual steps, Ad-hoc automation Direct Global NEWS inputs Basin-scale Inputs tables Global NEWS outputs Model Outputs GNE Complicated variables other than basin means/sums. Eg, dam retention, FQrem, lithology class Global NEWS model(s) GNE GIS pre-processing Summaries, Analysis, Maps Mass fluxes Other post- processing GNE post- processing GNE Model Run

Sample GNE Model Run Configuration File vars.cfg

Sample GNE Model Code From dissolved.py calculate FEriv

MNP Diffuse sources MNP Point sources UNH Hydrology “Raw”, Heterogeneous GIS Source files Diffuse sources Pre-processed, homogenized GIS Input files Point sources Hydrology Manual steps, Ad-hoc automation Direct Global NEWS inputs Basin-scale Inputs tables Global NEWS outputs Model Outputs GNE Complicated variables other than basin means/sums. Eg, dam retention, FQrem, lithology class Global NEWS model(s) GNE GIS pre-processing Summaries, Analysis, Maps Mass fluxes Other post- processing GNE post- processing GNE GIS Pre-Processing

MNP Diffuse sources MNP Point sources UNH Hydrology “Raw”, Heterogeneous GIS Source files Diffuse sources Pre-processed, homogenized GIS Input files Point sources Hydrology Manual steps, Ad-hoc automation Direct Global NEWS inputs Basin-scale Inputs tables Global NEWS outputs Model Outputs GNE Complicated variables other than basin means/sums. Eg, dam retention, FQrem, lithology class Global NEWS model(s) GNE GIS pre-processing Summaries, Analysis, Maps Mass fluxes Other post- processing GNE post- processing Other GIS Pre-Processing Mostly automated for MA scenarios, but in “ad-hoc” schemes, different software, etc.

MNP Diffuse sources MNP Point sources UNH Hydrology “Raw”, Heterogeneous GIS Source files Diffuse sources Pre-processed, homogenized GIS Input files Point sources Hydrology Manual steps, Ad-hoc automation Direct Global NEWS inputs Basin-scale Inputs tables Global NEWS outputs Model Outputs GNE Complicated variables other than basin means/sums. Eg, dam retention, FQrem, lithology class Global NEWS model(s) GNE GIS pre-processing Summaries, Analysis, Maps Mass fluxes Other post- processing GNE post- processing GNE “Loads” Post-Processing Calculates basin input and output variables as loads

MNP Diffuse sources MNP Point sources UNH Hydrology “Raw”, Heterogeneous GIS Source files Diffuse sources Pre-processed, homogenized GIS Input files Point sources Hydrology Manual steps, Ad-hoc automation Direct Global NEWS inputs Basin-scale Inputs tables Global NEWS outputs Model Outputs GNE Complicated variables other than basin means/sums. Eg, dam retention, FQrem, lithology class Global NEWS model(s) GNE GIS pre-processing Summaries, Analysis, Maps Mass fluxes Other post- processing GNE post- processing Other Post-Processing

Python Installation and Base GNE Setup Software Installation and Configuration –Is ArcGIS Installed? Potential conflicts with ArcGIS’ Python –Python (if ArcGIS not installed, or very old) –NumPy add-on –ArcGIS needed for GNE GIS pre-processing –(Other add-ons and custom scripts for additional post-processing: maps, regional summaries, etc.) GNE –Copy GNE files (code, inputs, configuration files, folder structure). Unpack training zip file –Avoid blank spaces or characters other than underscores and alphanumeric characters anywhere in the folder path! –Testing See User Manual for complete installation details, links to online Python/NumPy resources

Next, in Session 2 … Session 1. Introduction and Broad Overview This Presentation Session 2. Running NEWS models with pre-existing, basin-aggregated input tables Interactive Session, in 2 parts (2 sub-sessions) –Part 1. Model run organization, configuration files, conventions –Part 2. Model execution; changing a model run (“experiments”) We will change and execute a few model runs Session 3. GIS pre-processing and model input/output post-processing Interactive Session