The Joint Polar Satellite System (JPSS) is the next generation polar-orbiting operational environmental satellite system. The first satellite in the JPSS series of satellites, J-1, is scheduled to launch in early J1 will carry similar versions of the instruments that are on board of Suomi National Polar- Orbiting Partnership (S-NPP) satellite which was launched on October 28, To run S-NPP and pre-J1 algorithms in a development and test mode, the center for Satellite Applications and Research (STAR) Algorithm Integration Team uses the Algorithm Development Library (ADL). The ADL is an offline test system developed by Raytheon to mimic the operational system while enabling a development environment for plug and play algorithms. The STAR Algorithm Integration Team (AIT) has developed a Perl Chain Run Script to automate the staging and processing of multiple JPSS Sensor Data Record (SDR) and Environmental Data Record (EDR) products. The script has been used to provide test data for verification of algorithm and Look-Up-Table (LUT) changes related to various SDR and EDR products. In this poster, we will discuss the structure of the Chain Run Script and its use. We will use VIIRS Cloud Mask and Vegetation Index as examples to show how to use this script to process JPSS data. Abstract # Link and unpack RDRs # Link and unpack RDRs, convert ancillary data, prepare run set for each granule. # 1 st granule: 0002 d _t _e _b13235 Constructing Algorithm Chain... Construction Complete. Validating Algorithm Chain Prior to Run... Validation Complete. Algorithm Chain and Status: ProSdrViirsController:A1:NPP : Not Yet Built ProSdrViirsController:A1:NPP : Not Yet Built ProSdrViirsController:A1:NPP : Not Yet Built ProEdrViirsMasksController:A1:NPP : Not Yet Built Delta Algorithm Chain Run Started... Running Algorithms With 2 Threads. Algorithm Chain Queue: ProSdrViirsController:A1:NPP :Not Yet Built ProSdrViirsController:A1:NPP :Not Yet Built ProSdrViirsController:A1:NPP :Not Yet Built ProEdrViirsMasksController:A1:NPP :Not Yet Built Creating ProSdrViirsController.exe TK file for granule ID NPP /data/data020/weizhong/ADL4.1/CSPP/ADL4.2_mx8.4_base_a1/log/ProSdrViirsController_NP P xml Running ProSdrViirsController.exe for granule ID NPP :A1 /data/data020/weizhong/ADL4.1/CSPP/ADL4.2_mx8.4_base_a1/bin/ProSdrViirsController.exe Creating ProSdrViirsController.exe TK file for granule ID NPP /data/data020/weizhong/ADL4.1/CSPP/ADL4.2_mx8.4_base_a1/log/ProSdrViirsController_NP P xml Running ProSdrViirsController.exe for granule ID NPP :A1 /data/data020/weizhong/ADL4.1/CSPP/ADL4.2_mx8.4_base_a1/bin/ProSdrViirsController.exe Success for ProSdrViirsController.exe with granule ID NPP Creating ProSdrViirsController.exe TK file for granule ID NPP /data/data020/weizhong/ADL4.1/CSPP/ADL4.2_mx8.4_base_a1/log/ProSdrViirsController_NP P xml Running ProSdrViirsController.exe for granule ID NPP :A1 /data/data020/weizhong/ADL4.1/CSPP/ADL4.2_mx8.4_base_a1/bin/ProSdrViirsController.exe Success for ProSdrViirsController.exe with granule ID NPP Success for ProSdrViirsController.exe with granule ID NPP Creating ProEdrViirsMasksController.exe TK file for granule ID NPP /data/data020/weizhong/ADL4.1/CSPP/ADL4.2_mx8.4_base_a1/log/ProEdrViirsMasksControll er_NPP xml Running ProEdrViirsMasksController.exe for granule ID NPP :A1 /data/data020/weizhong/ADL4.1/CSPP/ADL4.2_mx8.4_base_a1/bin/ProEdrViirsMasksControll er.exe Success for ProEdrViirsMasksController.exe with granule ID NPP Chain Run Completed Successfully! Example 1: Cloudmask Overview The ADL Chain Run script is developed by the STAR AIT. The ADL chain run script is mainly Perl script to automate the staging and processing of multiple JPSS SDR and EDR products. The script calls ADL’s “runADLChainRunner.pl” to process data. The script uses a specified date with time range and/or a specified granule ID as an option to identify and stage input data. Get an ADL snapview from ClearCase in STAR’s network: 1.export $CSPP_HOME ($ADL_HOME= $CSPP_HOME/ADL) 2.source $ADL_HOME/build/envSetup.sh 3.Build ADL: “$ADL_HOME/buildAdl.ksh” or “perl ${ADL_HOME}/script/build_adl.pl -clean -makefiles -src -library -program -log $logfile” Build ADL Automated JPSS Products Processing of the Algorithm Development Library (ADL) by using Chain Run Scripts Weizhong Chen 1, Bigyani Das 1, Kristina Sprietzer 1, Valerie Mikles 1, Marina Tsidulko 1, Yunhui Zhao 1, Qiang Zhao 1, Vipuli Dharmawardane 1, and Walter Wolf 2 1 IMSG, Rockville, MD NOAA/NESDIS/STAR, College Park, MD Go to scripts location: “$CSPP_HOME/AIT_Scripts/ADL_Chain_Run/int_chainrun” 1.Modify “mruns_cm.sh”. 2.Modify “algorithm.list” file. (only keep the algorithms you want to run) 3.Modify “chainrun_config.txt”. Run “mruns_cm.sh algorithm.list &”. Run Chain Run Script 1. Modify “mruns_cm.sh” # ADL location in ClearCase: CM_ADL_Name="ADL42_Mx84_snap" export CSPP_HOME="/data/data020/weizhong/CSPP_CM/ $CM_ADL_Name/home/pub/ClearCase/STAR/JPSS /ADL" export ADL_HOME=$CSPP_HOME/"ADL” # Processing Date and rough time range: ProcessDate=" " stime="t " etime="e " #granID=“NPP “ # Log file Location of Chain Run Script for each algorithms log_dir=“./log" Structure and Functions ScriptDescription mruns_cm.sh Shell wrapper: Define the ADL location, date, start/end time, and/or granule ID chainrun_config.txt Configuration file of chain run script: define the input/output directory, case name, LUT location and Tiles location algorithm.list List of algorithms to be run. Upstream algorithms automatically processed in run. run_withMetadat a.pl Main chain run Perl script. ChainRuns.pm Primary Perl module: Prepare run set for each granule, call ADL’s “runADLChainRunner.pl” script to process data, then organize outputs. LinkConvertH5.p m Functions Perl module: include all functions to find and link RDR data from SCDR, then unpack them to ADL internal format as well as convert all ancillary data which include TLE, PW, NCEP and NAAPS. 2. Modify “algorithm.list” Cloud Mask: # alg.vcm VIIRS-EDR-Masks Surface Reflectance: # alg.surf VIIRS-EDR-SurfReflect Vegetation Index: # alg.vi VIIRS-EDR-VI 2015 NOAA Satellite Conference, April 27-May 1, Greenbelt Marriott Hotel in Greenbelt, MD Corresponding Author: We have: Developed and utilized an ADL chain run script. Demonstrated the script capability and usefulness by assisting several algorithm development teams. Included both the script and documentation in ClearCase for configuration management Demonstrated ability to keep pace with new MX build releases. By using ClearCase with the ADL chain run script, running ADL becomes an easy job. Within STAR’s network, a user can stage ADL chain runs for any algorithm and any date and time and begin getting results in minutes. Summary # -- user script location chainrun_dir=".“ # -- location to keep all inputs and output data. run_ADL_dir = "/data/data095/weizhong/run_aerosol_dr7723“ # -- unpacked ancillary data location run_ADL_dir_datasource "/data/data020/weizhong/run_ADL“ # -- Get the multiple inputs from the previous run’s “chainrun_config.txt” file in the following format. The default is empty. # input_dirs =“ 'algorithm name' 'CaseName' 'run_ADL_dir' ” # use ":" to separate different algorithms # use space to separate each items in every algorithm # input_dirs = " “ input_dirs = "VIIRS-EDR-Masks aerosol_vcm /data/data095/weizhong/run_aerosol_dr7723:\ VIIRS-EDR-SurfReflect aerosol_base /data/data095/weizhong/run_aerosol_dr7723“ # -- specific the folder's name of all results CaseName = "aerosol_vi_base" # -- specific the folder to keep all links of h5 files in the given time range. UserData = "user_data" # All the ancillary source data location: including all PolarWonder, TLE, NCEP and NAAPS data. pub_datasource="/data/data020/pub/NPP_DATA" # Default LUTs location: $CSPP_HOME/LUTs # Run “$CSPP_HOME/AIT_Scripts/STAR_AIT_ADL_Setup.pl” to link all associated LUTs to your “$CSPP_HOME/LUTs”.The user can replace them with their own LUTs under “$CSPP_HOME/LUTs”. AllLuts="LUTs:" # location of all Tiles all_tiles_withmeta="/data/data020/pub/NPP_DATA/TILES/NovGroundCCR692Tiles/with-meta-all" All those three directories have been prepared and maintained by STAR AIT. 3. Modify chainrun_config.txt 0003 d _t _e _b13235 NPP Constructing Algorithm Chain... Construction Complete. Validating Algorithm Chain Prior to Run... Validation Complete. Algorithm Chain and Status: ProSdrViirsController:A1:NPP :Built ProSdrViirsController:A1:NPP :Built ProSdrViirsController:A1:NPP :Built ProSdrViirsController:A1:NPP :Built ProSdrViirsController:A1:NPP :Built ProEdrViirsMasksController:A1:NPP :Built ProEdrViirsMasksController:A1:NPP :Built ProEdrViirsMasksController:A1:NPP :Built ProEdrViirsAerosolController:A1:NPP :Built ProEdrViirsSurfReflectController:A1:NPP :Built ProEdrViirsVI:A1:NPP : Not Yet Built Delta Algorithm Chain Run Started... Running Algorithms With 2 Threads. Algorithm Chain Queue: ProEdrViirsVI:A1:NPP :Not Yet Built Creating ProEdrViirsVI.exe TK file for granule ID NPP /data/data020/weizhong/ADL4.1/CSPP/ADL4.2_mx8.4_base/log/ProEdrViirsVI_NPP xml Running ProEdrViirsVI.exe for granule ID NPP :A1 /data/data020/weizhong/ADL4.1/CSPP/ADL4.2_mx8.4_base/bin/ProEdrViirsVI.exe Success for ProEdrViirsVI.exe with granule ID NPP Chain Run Completed Successfully! linking... VIVIO to userOutputDir: /data/data095/weizhong/run_aerosol_dr7723/data/output/use r_data/withMetadata/ProEdrViirsVIOutputs/ vi_base job_output_dir: /data/data095/weizhong/run_aerosol_dr7723 /data/output/withMetadata/ProEdrViirsVIOutputs/ vi_base/d _t _e _b13235 NPP : VIIRS-EDR-VI 0.65 min. Example 2: Vegetation Index