1. CDAAC Status and Strategy Bill Schreiner and CDAAC Team COSMIC Retreat Nov 4, 2010

2. CDAAC Objectives CDAAC Overview CDAAC Status Summary of CDAAC mini-review Processing COSMIC Follow-on missions and missions of opportunity Outline

3. Provide high quality GNSS RO data to the community in NRT, post-processing, re-processing (re-analysis) Improve data processing algorithms to maximize impact of RO data on research and operations Support data users, TACC Conduct scientific investigations when appropriate and efficient CDAAC Objectives

4. CDAAC Personnel Program Director: Bill Kuo Chief Scientist: Chris Rocken CDAAC Manager: Bill Schreiner S/W Engineers –Doug Hunt, Lead –Maggie Sleziak-Sallee, Web –Teresa Vanhove, GPS –Jason Lin, SE –Karl Hudnut, System Admin/Engineer Scientists –Bill Schreiner, GPS Lead –Sergey Sokolovskiy, RO Inversions –Xinan Yue, Ionosphere –Janet Zeng, RO Inversions –Tae-Kwon Wee, 1DVAR

5. Main CDAAC Functions RO Payload Operation –Configuration control (firmware and tracking configuration) –Scientific/technical guidance for commanding, operation –Near real-time monitoring, trouble-shooting, and Q/C data analysis Data Processing and Analysis –Level0 unformatting and QC –GPS ground processing (ZTD, site estimation, clock estimation) –LEO POD, and atmospheric excess phase –Absolute TEC generation –Inversions (neutral atmosphere and ionosphere) –Retrievals (1DVAR) –NWP and correlative data handling –Product QC and analysis

6. CDAAC Functions (cont) System Operation and Monitoring -H/W, O/S and NFS filesystem -System fail-overs -CDAAC operations Input data stream monitoring –RTS downlink –GPS Bit-grabber operation and monitoring –GPS ground data (IGS, NRCan, EUMETSAT, COSMIC sites) –IGS and IGU orbits, clocks and EOP –Bernese configuration –NCEP GFS, ECMWF, radiosonde, ionosonde Data Management, Dissemination, and Archival Support Data Users

7. Active I/O Computer CDAAC Hardware Real-Time Setup –Dual String system –Redundant Input/Output IO- computers –Redundant Real-Time Operational cluster computers (with RAIDS) –Real-Time staging cluster to stage (test) software updates –Backup power and cooling Post-Processing Setup –R&D cluster –Re-Analysis cluster for climate re-analysis –FTP Server with RAID Backup I/O Computer

8. CDAAC Processing Flow LEO dataLevel 0--level 1 POD (Orbits and clocks) Excess Phase Full Spectrum Abel Inversion 1-D Var Moisture Correction Fiducial data Atmospheric processing Excess Phase Processing Abel Inversion Combination with other data Ionospheric processing Profiles Real time Task Scheduling Software atmPhsatmPrf (S4) ionPrf (EDPs) ionPhs (L1,L2 amp/phase) bfrPrf podTec (TEC) scnLv1 (S4) tipLv1 (Radiance) wetPrf CDAAC File types in RED

9. Best Effort Monitoring Monitor the system regularly throughout the day M-F 8am-8pm CDAAC Ops team monitors the system 3 times per day on weekends and holidays Available by email and cell phone CDAAC reliability estimated > 99.5%

10. Near Real-Time -COSMIC -SAC-C and C/NOFS Post-Processing (2007.3200-2009.2650) -GPSMET, GPSMET-AS, CHAMP, SAC-C, GRACE, COSMIC -Will attempt to understand 3 “noisy” CHAMP months (2007.3200) Re-Processing (2009.2650) -Published new CHAMP re-processing -COSMIC re-processing finished, published soon -Then start re-processing for GPSMET, GRACE, TerraSAR-X, SAC- C, Metop/GRAS, C/NOFS CDAAC Data Processing Status

11. Output file saved from each process Hard failures logged in cosmic_badprf table Most products have BAD flags in files and database Some product files include error characteristics LEO Orbit Determination (Post-fit residual test, Orbit overlap test) Neutral atmospheric excess phase files (SNR and phase noise tests) Neutral atmospheric profiles (Bending angle noise and bias tests, checks for L2 quality, presence of scintillation, gross checks on N/T/P) Absolute TEC data arcs (cycle slip detection/correction, outliers) Electron Density Profiles (under development) Scintillation Indices (under development) TIP Night-side Radiances (BAD flag for incomplete integration time, motor not at proper destination, ill-defined sensitivity) COSMIC Data Product QC

12. To be released by late November 2010 RPM-based install system Improved data access, website, DB interface, monitoring, configuration management, archiving, documentation All other packages updated to latest versions: perl, postgres, apache, Bernese,.. Real-time clock subsystem Improved excess phase for SD processing Use of higher resolution real-time and post-processing first guess models (GFS high resolution and ERA-40 interim) CDAAC 3.0 Status

13. Held Oct 27, 2010 What CDAAC development efforts are required for COSMIC and other missions? How much effort? How do we know our CDAAC GPS RO data processing is optimal? –Review of algorithms and software internally and via publication –Comparison of results (POD, BA, N, T, WV) with other data processing software and investigate and understand differences (i.e. more detailed GRAS study) –Comparison with correlative data, models? CDAAC Review

14. CDAAC System changes Better system for managing code for production/research Simplify addition of new missions names in CDAAC code Restructure ROAM Refactor genXphase Add more test suites Clean up unused code Continue to add missions Improved QC, bad flags, error estimates Multi-GNSS capability, new observables! Low latency processing Better systems management Need to develop CDAAC Level0-Level1 processing modules Improved monitoring scripts

15. GNSS Processing Add GNSS capability, new observables Improve NRT ZTD -Compare algorithm with Suominet Improve NRT GNSS clock estimation -5-sec GNSS clocks Improve NRT and PP LEO POD (to 0.1 mm/sec) –Process data from both POD antennas –Compute/Apply PCV maps –Implement SLR processing for validation Improve excess phase processing -Review algorithms, fix bugs, -5-sec GNSS clocks -refactor Perform additional validation studies

16. Neutral Atmospheric Inversions Restructure ROAM Add GNSS capability, new observables Improve wave optics processing –looking for better filtering approaches –looking for alternative methods (like recently introduced WDF) Improve bending angle optimization -testing methods with reduced weight of climatology -validation by independent data sources in the stratosphere Improve 1DVAR retrieval code and documentation Perform additional validation studies (e.g. integrate ROPP package, RO-Trends+)  Requires investigation and understanding the sources of the differences

17. 1) Phase Connection 2) Bending Angle Generation (WO,GO) 3) Bending Angle Correction, Connection 4) Bending Angle Optimization, Inversion atmPhs conPhs benPrf bcnPrf atmPrf 1) Mission-dependent; 2-4) Mission-independent. 1) Input positions, velocities, raw phase and amplitude, clim. model. Processing removal of NDM, connection of the phase, down-sampling to one rate. Output positions, velocities, connected phase, amplitude, HSL, lat, lon, height of TP. 2) Input output from 1 Processing retrieval of WO (Phase Matching and FSI or CT2) and GO bending angles for L1, L2, (L5) Output GO and WO bending angles for L1,L2,(L5) 3) Input output from 2 Processing ionospheric correction (incl. additional smoothing of L4), connection of GO and WO bending angles. Output ionosphere free connected bending angles 4) Input output from 3, clim. model, atm. model. Processing optimization of bending angles, inversion of N,T,P. Output N,T,P.

18. Comparison of COSMIC retrieved temperatures to ECMWF for Different optimizations of BA 2009.015-045 Current CDAAC approach: - clim. fitted to obs. BA - mixed with weighting function - WF different for different occs. - on average fitted clim. >50% above h~45 km Tested approach: - obs. BA smoothed with window increasing from 1.5 km at h=30 km to 15 km at h=60 km - clim. >50% above h=60 km for all occs.

19. Absolute TEC uncertainty under investigation -DCB estimation, Code/phase leveling uncertainty Electron Density Profiles -Improving EDPs by using information on horizontal gradients with DA Improve scintillation products Add GNSS capability, new observables Perform additional validation studies -With Paul Straus of Aerospace, JPL Ionospheric Processing

20. 1-2 TECu uncertainty can be ignored for ground based GPS or low elevation LEO GPS observations. But it might have problem if we want to study the plasmasphere using COSMIC sTEC [eg,2006.230-245]

21. TEC Precision from Collocated Tracks COSMIC (2006.230-245)

22. Comparison of the latitude and altitude Ne and its retrieval error from simulation LT=13 Manmade plasma cave disappears in DA retrieval. Data assimilation retrieval

23. Processing NRT data from MOOs at CDAAC development, testing, monitoring of RTS downlink decoding and formatting binary level 0 science payload data. Effort depends on payload architecture system modification, configuration and management data processing modifications testing and validation NRT operations and monitoring, post-processing, archival, maintenance program management Recent experience with C/NOFS and SAC-C valuable

24. C-2 Operational Data Processing Requirements Reliable and low latency input data streams Primary and Backup DPC (Data Processing Center) Development system (GNSS capable) Staging system to test processing changes Configuration management of systems Communication access between systems (DPCs, SOCC, RTSs). Increased IT Security? 24/7 monitoring of near real-time operations Maintenance and on call technical support Operator Training Secondary Science payload processing Post-Processing

25. Configuration Management for C-2 Cfengine controls Log rotation Shared file systems Processing Daemons Software integrity Fail-over between I/Os and processing clusters Disk space/cleanup Syncing SVN configuration repository Cfengine Configuration Control Machine Primary DPC Backup DPC Staging Single string Development Single string Post- Processing

26. C-2 DPC Options with CDAAC Option AOption B PrimaryNOAA (single string hw ~$100K) Hot BackupUCAR (single string hw ~$100K) UCAR staffing (TBD) TDPC (single string hw ~$100K) TDPC staffing (TBD) StagingNOAA (single string hw ~$100K) Monitoring24/7 at NOAA (~5 FTE) Best effort at UCAR (TBD) 24/7 at NOAA (~5 FTE) 24/7 at TDPC (~5 FTE) R&DUCAR (single string hw ~$100K) UCAR staffing (TBD) UCAR (single string hw ~$100K) UCAR staffing (TBD) On Call Tech Support UCAR staffing (TBD) Post- Processing UCAR (single string hw ~$150K)

27. CDAAC R&D Schedule for C-2

28. June 30 th 2010 meeting with NOAA/OSDPD (Kathy Kelly) -Review of GPS RO processing at CDAAC -HW/SW data processing architecture at NSOF -Held additional telecon -Working to test CDAAC within NSOF architecture -Providing info for NOAA CLASS archive Oct 14 th 2010 meeting with AF/Aerospace -Review of CDAAC and SW architecture -Strong interest in using CDAAC to process space weather payloads -Additional meetings in planning F-7/C-2 JPMO Meeting Dec 7-8, 2010 -Discuss data processing issues Recent Activities

29. Investigated structural uncertainty of RO in LT. Published in JGR Comparing different methods of estimation of the ABL depth from RO. Investigating global variability. Publishing results soon. Detection of the sporadic E clouds from RO. In press at GRL Investigation of the response of the UTLS in terms of gravity waves and tidal variations from COSMIC data to the sudden stratospheric warming event in 2009 COSMIC EDPs used to estimate ionosphere High Transition Heights (HTH). In press at JASTP Investigated global ionospheric response observed by COSMIC Satellites during the January 2009 Stratospheric Sudden Warming Event. In press at JGR-SP Submitted data assimilation paper to improve EDPs. In revision at JGR Recent RO Research and Science Apps

30. Global Evaluation of Radiosonde Water Vapor with RO data, Published in J of Remote Sensing Evaluation of AMSU and AIRS with RO Impacts of GPS radio occultation data assimilation on regional weather predictions. Published in GPS Solutions Working on Round-2 of RO-Trends inter-agency comparison UCAR funded by EUMETSAT to evaluate Metop/GRAS RO data. Final Report due in early Nov Recent RO Research and Science Apps