1. Research To Operations (R2O, R2X and O2R)
Sid-Ahmed Boukabara
NOAA/NESDIS/STAR & JCSDA
The JCSDA Summer Colloquium, August 7th, 2015, Fort Collins, CO

2. Contents What Is R2O or R2X in General? 1
What is an O2R and why is it important? 2
What is needed for a successful R2O/O2R? 3
How do we measure Success in R2O? 4
Specific Example of the JIBB and S4 5
Conclusions and Lessons Learned 6

3. What Is R2O or R2X in General?
R2O is the process by which a scientific project led by a researcher, proven to provide added value, is transitioned to the Operational environment
R2X refers to the transition of a research project outcome to operations, applications, and services (for example to decision makers). R2O is an example of R2X

4. Why is R2O Important for NOAA? (and other operational partners)
R2O is an efficient way to leverage research done in NOAA and in the outside research community, to improve NOAA operational systems
R2O allows NOAA to maximize the return on investment (ROI) of its outsourced research
R2O allows NOAA to remain competitive, and at the edge of science in its products.

5. Researcher in Academia
R2O ?
R2O means often different things to different people
NESDIS
STAR
R2O
R2O
OSPO
Researcher in Academia
R2O
NWS
EMC
R2O
NCO

6. JCSDA R2O Concept
Goal: Accelerates the use of satellite data in NWP centers
Ensures resources are in place to achieve successful transition: supercomputer, a software integration team, etc.
Ensures consistency and demonstrates benefits with operational systems like: Hybrid GDAS system (global), HWRF (Hurricane forecast).
Scientific efforts in satellite DA in research community
Diverse R&D activities
Scientific efforts in satellite DA in academia, CIs
Scientific efforts in satellite DA in NOAA/OAR
Scientific efforts in satellite DA in NOAA/NESDIS
(funded by GOES-R PGs, JPSS PGs, etc)
JCSDA’s own DA Activities
Products, techniques, improvements, with direct and immediate relevance to NWP Operational Models
(both global and regional)
Made consistent with Operational Systems
This was possible thanks to the JIBB supercomputer (JCSDA In a Big Box) and the S4 supercomputer (Supercomputer for Satellite Simulations and data assimilation Studies).
The main goal of this environment is to (1) mature a diverse set of R&D activities (funded for instance by JPSS PG&RR, by GOES-R PG&RR, NESDIS, etc) to become operations-relevant and tested and then (2) transition the most mature projects into operations, thanks to a strong collaboration between the PIs, the JCSDA scientists and the scientists from the NWP operational centers.
Select projects for R2O Transition
NWP Operational Centers
Objective: Improvements in Forecast skills
Ongoing Baseline Improvement in NWP centers

7. Contents What Is R2O or R2X in General? 1
What is an O2R and why is it important? 2
What is needed for a successful R2O/O2R? 3
How do we measure Success in R2O? 4
Specific Example of the JIBB and S4 5
Conclusions and Lessons Learned 6

8. What is an O2R and why is it important?
O2R infrastructure brings the Operational environment to researchers
It allows scientists to perform their research in a system that is compatible with operational needs
When a successful research project is performed in an O2R environment, it increases its chances to transition to Operations (R2O)
Without an O2R environment, R2O is at best difficult, (often does not happen).

9. O2R Is the Bridge Between R and O
Research
Operations
O2R
Valley of Death

10. How does O2R/R2O work?

11. Contents What Is R2O or R2X in General? 1
What is an O2R and why is it important? 2
What is needed for a successful R2O/O2R? 3
How do we measure Success in R2O? 4
Specific Example of the JIBB and S4 5
Conclusions and Lessons Learned 6

12. Components of an O2R/R2O Infrastructure(1)
The infrastructure is not just the supercomputer:
What Money can Buy….
Supercomputer (Hardware, basic software, and IT) (almost the easiest part!)
Scientific Software Integration (this is the hardest part: keep in synch with operational models)
User Support & Documentation
Management of resources to allow R2O and O2R to be sustained
Documented Utilities to help scientists (CRTM, assessment, formatting tools)
Constant engagement between Research and Operations
Rigorous software configuration system to track changes
Rigorous (and independent) testing mechanism to demonstrate added value

13. Components of an O2R/R2O Infrastructure (2)
What Money can NOT Buy….
And perhaps more importantly: the right culture.
Transparency of R2O protocols and willingness to accept research performed by researchers & work collaboratively (on the operational partner)
Willingness to work/start with operational system and follow protocols and accept constructive feedback (on the research partner)
Sense of Mutual respect between Researchers and Operational partners: respect for innovation in research and pragmatism in the operational implementation
Accept that R2O is not for everyone: careful choice of research and operations partners is important

14. Contents What Is R2O or R2X in General? 1
What is an O2R and why is it important? 2
What is needed for a successful R2O/O2R? 3
How do we measure Success in R2O? 4
Specific Example of the JIBB and S4 5
Conclusions and Lessons Learned 6

15. Metrics for Assessing O2R Success
When R2O transition has occurred: assess specific scientific metric(s) that was(ere) improved upon transition to operational environment.
Example: Anomaly correlation, hurricane track improvement, etc.
When R2O transition has not occurred yet, assess progress in the maturity of the project, as a result of the O2R infrastructure

16. Example of a project that led to an R2O transition (Improved QC)
Significant
Southern hemisphere 500-hPa height ACC for 10 September – 24 October 2012 for the control (black) and LNVD (red).
Bottom: Variation between LNVD (red) and the control (operational QC of MODIS winds: black) based on forecast hour.
All ACC differences outside of the bars are significant at the 95% confidence level.
The circle represents a statistically significant improvement for the Day-4 and Day-5 forecasts.
Plot courtesy of D. Santek, UW (S4 project)

17. How do we assess R2O-maturity? (Case of JCSDA/NESDIS)
JCSDA Modes of Operation
NOAA Technical Readiness Level (TRL)
TRL 1
Basic principles observed and reported
TRL 2
Technology concept and/or application formulated
TRL 3
Analytical and experimental critical function and/or characteristic proof-of-concept
TRL 4
Component/subsystem validation in laboratory environment
TRL 5
System/subsystem/component validation in relevant environment
TRL 6
System/subsystem model or prototyping demonstration in a relevant end-to-end environment
TRL 7
System prototyping demonstration in an operational environment
TRL 8
Actual system completed and “mission qualified” through test and demonstration in an operational environment
TRL 9
Actual system “mission proven” through successful mission operations
TRLs 1-2 are nominally considered Research,
TRLs 3-5 are Development,
TRLs 6-8 are Demonstration, and
TRL 9 is Deployment, Implementation, or Operational Transition.
FFO
Directed Research
R2O Path
Visiting Scientist Program

18. Contents What Is R2O or R2X in General? 1
What is an O2R and why is it important? 2
What is needed for a successful R2O/O2R? 3
How do we measure Success in R2O? 4
Specific Example of the JIBB and S4 5
Conclusions and Lessons Learned 6

19. The S4 system, hosted by University of Wisconsin.
Supercomputer for Satellite Simulations and data assimilation Studies (S4)
Overview:
NESDIS, in collaboration with UW is making S4 available to scientists involved in satellite data assimilation and other activities related to it.
One of the purposes is to consolidate data assimilation activities performed in the research community
and to encourage the usage of NOAA operational models/tools/codes installed on it.
Open policy for users (since hosted in academic institution).
Only open source/non-restricted data allowed on S4.
Status: Upgraded in 2014.
The S4 system, hosted by University of Wisconsin.

20. S4 technical Description
Brief Technical Description
The S4 system is a Linux cluster (Dell/Intel hardware),
4700 CPU cores (60 Tflops) with 18TB of total RAM
1700TB in storage nodes
Hosted in the UW/SSEC Data Center (with UPS)
Able to run 4 GSI/GFS experiments simultaneously at T1534
System Administration and Resource Allocation
SSEC provides system administration services and security protocols for the system,
NESDIS/STAR provides overall guidance on system resource allocation and selection of users as well as scientific support (code management, software integration, etc)

21. S4 Intented Activities
This system is intended to allow primarily (but not exclusively) the following major activities:
(1) Undertake satellite data assimilation experiments at global and/or regional scales and the assessment of their impacts on forecast models skills, using currently flying satellite sensors and allowing scientists to test new science/methodology and
(2) In support of the activity above, undertake all necessary satellite data simulations, calibration, algorithms development/improvement, radiative transfer modeling and validation, quality control (QC) procedures, etc
(3) Perform Observing System Simulation Experiments (OSSEs) for new sensors (such as GOES-R and JPSS).
This component is expected to address key questions that NESDIS constantly faces: (a) what is the impact and the added value of flying new sensors, (b) what is the optimal orbital configuration, (c) what are the best mitigation strategies possible, from an NWP perspective, given the budget and other constraints, (d) how to best coordinate with other national and international partners to achieve the most optimal Earth satellite observing system

22. S4 Governance & Management
S4 is governed by a NESDIS/STAR policy memo identifying roles and responsibilities
IT support is handled by the host institution UW.
Software Integration is handled by NESDIS (the same team handles the software integration on the JCSDA sister machine JIBB)
To request an account, a simple to STAR POC will suffice. Accounts usually set up the same day.
The S4 committee includes NESDIS/STAR director, deputy, and three senior program scientists (JCSDA, JPSS and GOES-R), makes recommendations concerning 90% of the machine's resources. The remaining 10% is managed by UW.

23. Policies and Code management on S4
Code management is the hardest part
The software integration team is dedicated to making sure the NOAA codes installed on S4 are in synch with the operational models (including scripts, codes, datasets, libraries, etc)
SVN is extensively used when appropriate.
The following is a list of S4 Policies:
User account request
Managing resources
Account expiration
Tools and softwares guidance
User guide
Compiler & Software Availability
Intel Fortran / C++
Gfortran /gcc/g++
Netcdf 3 & 4
Matlab
IDL
NCO 4
GRADS
HDF 4 & 5
NCL

24. How much did Projects benefit from S4?
Project Title
Project Leads
Operational Partner
Point of Contact at Oper. Partner
Expected Date (best guess) for R2O
R2O Current Stage
(TRL)
TRL before using S4
1. A New Quality Control Method for Satellite-derived Polar Winds in the NCEP GDAS/GFS
David Santek, and Brett Hoover
NCEP
Andrew Collard, John Derber
Dec. 2015 8 6
2. Proactive Quality Control (PQC) based on Ensemble Forecast Sensitivity to Observations (EFSO)
Daisuke Hotta, Tse-Chun Chen and Eugenia Kalnay
Daryl Kleist, Mark Iredell
January 2016 7 3
3. Development of Community Satellite Data Thinning and Representation Optimization Tool (CSTROT)
Tong Zhu and Sid Boukabara
Andrew Collard
June 2016 2
4. A near real time regional satellite data assimilation system for high impact weather research and forecasts
Jun Li, Jinlong Li, and Pei Wang
NHC and EMC
NHC: Mark DeMaria, John L. Beven
EMC: Vijay Tallapragada
October 2014 to NHC
June 2016 to EMC 1
5. Tropical cyclone data assimilation of MSG SEVIRI all-sky infrared satellite radiances
Milija Zupanski and Man Zhang
NCEP/EMC
was Steve Lord
begin in 2016
6. Studying tropical cyclone genesis and rapid intensification with the NCEP GSI data assimilation system
Zhaoxia Pu
Vijay Tallapragada
7. Assessment of Temperature and Humidity Prediction with Microwave and Hyperinfrared Satellite Data Assimilation over United States
Jianjun Xu, and Alfred Powell,Jr
STAR
Mike Kalb &
Sid Boukabara

25. Maturity Assessment of S4 projects
Added value of S4 O2R Environment measured by the maturity-index (before and after using the S4)

26. Examples of S-NPP Projects Benefiting from O2R (Not directly funded by JCSDA, but supported through the O2R: HPC, user support, Porting of codes, etc) #
PI/Scientist
Project Title
Main Sensor used 1
Jinlong Li
Near real-time assimilation system development for improving tropical cyclone forecast with NPP/JPSS sounder data
CrIS, ATMS 2
Banglin Zhang
Improve Hurricane Structure Monitoring and Intensity Forecast Using NPP ATMS and GCOM-W AMSR2
ATMS 3
Jianjun Xu
Application of hyper-spectral satellite data in severe weather forecasts with GSI/WRF regional data assimilation system
CrIS 4
Lin Lin
Evaluation of BG-Remapped ATMS on Hurricane Sandy Forecast & Test of Mie and DDA Scattering in the Simulation of ATMS 5
Marouane Temimi
Assessment of Assimilating NPP/JPSS ATMS Ground Surface Observations 6
Li Fang, et al.
Dual Assimilation of Microwave and thermal-infrared observations of soil
moisture into NLDAS for improved drought monitoring
ATMS, VIIRS 7 8
Hu Yang
Scientific Support for JPSS Instrument Calibration
S-NPP Sensors (ATMS) 9
Daisuke Hotta
Proactive-QC based on Ensemble Forecast Sensitivity to Observations (EFSO)
All S-NPP assimilated sensors (ATMS) 10
Xinjia Zhou
BTs validation of CRTM simulations against AVHRR, MODIS and VIIRS BTs in SST bands
VIIRS 11
Johnathan W. Smith
WRF-Chem data assimilation of NOAA-Unique CO retrievals: Impact study on tropospheric ozone enhancement observed during the 2010 AEROSE Campaign
IASI (potential application to CrIS) 12
Agnes Lim Huei Ni
Impact analysis of LEO Hyperspectral Sensor IFOV size on the next generation NWP model forecast performance
1 of 6

27. Resources installed for Global and Regional DA and OSSE Activities
The following models & tools are installed on S4:
Global Data Assimilation System (GDAS) (including the new Hybrid Ensemble system)
Grid Statistical Interpolation (GSI)
Weather Research & Forecast (WRF) (WRF-ARW and WRF-NMM)
Hurricane Weather Research & Forecast (HWRF)
Microwave Integrated Retrieval System (MIRS)
Community Radiative Transfer Model (CRTM)
Verification Statistics Data Base (VSDB)
Ocean Model HYCOM
Etc…
The following models & tools will be installed in the future:
NOAA Environmental Modeling System (NEMS)
Unified Post Processor (UPP)
Land Information System (LIS)
Etc..

28. Major projects on jibb
General data impact and OSSE experiments for Several Sensors
Next generation NOAA hybrid data assimilation system port
Hyperspectral IR methodology
Hyperspectral IR cloudy radiances
Satellite wind impact experiments
Lightning assimilation methodology
FFO projects of the JCSDA
NASA Roses Projects of the JCSDA

29. JIBB and S4 Benchamarking
Efforts have been devoted to compare satellite data experiments results on S4 and JIBB, to those performed on the NOAA R&D
Benchmarking performed for currently operational GDAS (GSI+GFS)
The results show that the operational version of GDAS is running well on S4 and JIBB (both T382 and T574)
Efforts included the benchmark of the hybrid data assimilation system.
Anomaly correlation performed using GDAS system on NOAA R&D machine (Vapor), JCSDA JIBB machine and the S4 system. Benchmarking performed by J. Jung and the STAR/JCSDA SI team. Courtesy of J. Jung.

30. Contents What Is R2O or R2X in General? 1
What is an O2R and why is it important? 2
What is needed for a successful R2O/O2R? 3
How do we measure Success in R2O? 4
Specific Example of the JIBB and S4 5
Conclusions and Lessons Learned 6

31. Lessons learned for an effective R2O
Importance of the O2R environment
Importance of availability of an accessible HPC
Importance of a transparent R2O protocol
Importance of code portability & synchonization
Importance of enforcing the exclusive usage (by researchers) of the operational systems
Importance of a rigorous traceability of changes
Importance of the S4 and JIBB approaches
Importance of nurturing a culture of mutual respect between research and operations partners

32. Coordination & Synchronization: JCSDA Accelerated R2O coordination for SNPP/ATMS
SNPP Was launched Nov 2011
Dec 2011, the JCSDA coordinated the signature of the R2O Action Plan for ATMS R2O between NESDIS and NWS
The main points were: Who, does what and when and where (between NESDIS and NWS)
Strong backing from STAR and NCEP Directors (present at kick off meeting)
7 months after launch, ATMS was assimilated operationally in NOAA
Important of ‘coordinate & synch’

33. Conclusions
JCSDA and NESDIS are involved in R2O by making significant resources available to scientists and collaborators (HPC, support, ..)
The two major activities targeted are: (1) Advanced Satellite data assimilation and (2) OSSE activities.
The O2R infrastructure is more than the supercomputer: components include software management, integration and user support, etc
The JIBB and S4 offer an O2R that is both operationally relevant as well as open and inclusive
S4 and JIBB access offers an O2R environment conducive of innovation, but with a clear R2O objective.

34. Questions?