1. Framework Details  All products may be run from one program  Coordination of input data:  Model Forecast data  Emissivity Data  Instrument Data  Unified Data Structures  Data structures for input – instrument and ancillary data  Data structures for output products  Production rules determine how the system is run  Run any product in order determined by the production rules  Product runs may automatically trigger product precedence  Radiative transfer model calculations performed within the framework for all algorithms  One data format for all products  Common software libraries  Fully integrated C/C++ and Fortran 90/95 program AWG Algorithm Flowchart: Description of the Product System  Flowcharts are delivered by the product teams to the AIT:  Describes the Algorithm Processing System  Identify the inputs and outputs  List Function/Subroutine calls  Describes how the products are generated  Preparation of Framework for algorithm software  AIT has received system flowcharts of each of the product systems being created within the AWG.  AIT has/will breakdown each of the product systems to find the commonality between them.  Commonality between the systems resides within the framework. Input Configuration Files  Determine how the framework is run  Run all products or a single product  Modify the dependencies used to create the products  Multiple settings (flags, constants, coefficient files), different dependencies  Enable backup dependencies  Run using multiple configuration files or a single configuration file  Run using default settings for each product or may customize  Run multiple satellites and instruments  Test different versions/updates of an algorithm against each other  Test different algorithms against each other Infrastructure: Hardware and Software Hardware One IBM Power5+ machine with 16 CPUs 75 TB of disk space on the SAN for all simulated data, proxy data and products. Rack of Dell Processors (33 machines, 144 CPUS) for product development. Software Compilers Intel compiler for Linux Portland group compiler for Linux Compiler on IBM AIX machine (xlf90, xlC) Code Checkers Lint Forcheck Valgrind Home grown Standards Checkers Revision Control Clear Case Clear Quest Development of the GOES-R AWG Product Processing System Framework Walter Wolf 1, Lihang Zhou 2, P. Keehn 2, Q. Guo 3, S. Sampson 3, S. Qiu 2, and Mitchell D. Goldberg 1 1 NOAA/NESDIS/ORA, Camp Springs, MD 20746 2 PSGS, Alexandria, VA 22311, USA 3 IMSG, Kensington, MD 20895, USA Algorithm Working Group (AWG) Integration Team (AIT)  Consists of:  System Lead  Quality Assurance Lead  Four Full Time Programmers  Documentation Specialist  Configuration Manager  Roles  Interface with the AWG Product Teams  Integrate AWG Product Systems  Quality Assurance Check of the AWG Product Systems  Interface with the GOES-R Project Office  Packaging of deliverables to the GOES-R Project Office  Supply the standards for the AWG product teams:  Coding Standards »Code examples  Documentation Standards »Documentation Templates  Delivery Standards  Configuration Management  Coordinate between AWG product development teams  Provide technical and software assistance to AWG product teams  Receive code deliveries from product teams  Verify and Review code deliveries  Distribute updated software package to product teams  Overall Strategy  Develop a processing system where GOES-R algorithms can be developed and tested is a well defined and organized manner  Code will adhere to NESDIS Operational Standards  All pre-operational code is thoroughly tested within the STAR Collaborative Research Environment Interface between Framework and the Product Algorithms  For each algorithm, have to create a common interface that is used both by the AIT framework and the algorithm research system  Common interface enables a plug and play capability between the research system and the AIT framework:  All data required by the algorithm from the framework is passed through this interface  Reduces time that it takes to roll back the algorithm changes to the product team that the AIT makes  Reduces integration time for future deliveries Abstract The GOES-R Algorithm Working Group (AWG) Product Processing System Framework is under development at NOAA/NESDIS/STAR. The framework has been put in place for the development of the AWG geophysical product processing systems. This development is to produce and test the scientific algorithms in a scalable operational demonstration environment -- STAR collaborate environment. The framework includes accommodations for all the Level 2 Advance Baseline Imager (ABI) products. The dependence of one product on the output of another product (product precedence) has been taken into account during the development of the framework. Data common to all the product systems have been read into the framework and passed to the appropriate product systems. These design features and the current status of the framework will be discussed. STAR Collaborative Research Environment  To develop and test scientific algorithms in a scalable operational demonstration environment  Hardware consists of operating systems that are within OSDPD  Multiple compilers are available for maximum code debugging capability  Develop algorithms using large quantities of simulated, proxy, and/or real data  The GOES-R AWG work is conducted within the STAR Collaborative Research Environment  Migrate and test AWG product systems  GOES-R AWG product validation.  Integration of AWG product systems into the AIT framework Organized Ancillary Data  Ancillary data used by more than one algorithm is read into the AIT framework and then passed to the individual product systems  This reduces the I/O time and removes unwanted redundancies  Current ancillary data that has been moved from product systems to the framework are:  NWP  Emissivity Current Status  Framework is in place  Completed parts:  NWP model ingest  Radiative Transfer Model installation  Configuration File Ingest  Current Work  Installation of the following product systems:  Cloud Mask  Sounds  Fire  Land Surface Temperature  NDVI  Aerosols ABIAOT Output retrieval results Retrieval over land 1.) Initialize output structure. 2.) Interpolate aerosol lookup table to current geometry. 3.) Calculate aerosol independent atmospheric parameters. 4.) Estimate surface reflectance. 5.) Retrieve AOT550 and compute residual for each aerosol model 6.) Select the optimal solution. 7.) Calculate the AOTs at all ABI bands. Read Configuration file Read Lookup tables Read input data ABI reflectance data ABI cloud mask Ancillary model data Lookup tables Determine number of pixels Process each pixel Assign pixel input structure Declaration and initialization Land pixel? Suitable for retrieval? Internal check for unfavorable land condition Internal check for unfavorable water condition Pass land internal check? Pass water internal check? Retrieval over water 1.) Initialize output structure. 2.) Interpolate aerosol and sunglint lookup table to current geometry. 3.) Calculate aerosol independent atmospheric parameters. 4.) Estimate surface reflectance parameters. 5.) Retrieve AOT550, fine mode weight and compute residual for each aerosol model combination. 6.) Select the optimal solution. 7.) Calculate the AOTs at all ABI bands. ABIAOT End No Yes No Yes No Detailed Aerosol Optical Thickness Flowchart