The Collaborative Radar Acquisition Field Test (CRAFT): A Unique Public- Private Partnership in Mission-Critical Data Distribution Kelvin K. Droegemeier University of Oklahoma Internet2 Spring Meeting 20 April 2004 NCDC

NEXRAD Doppler Radar Network

The Moore, Oklahoma Tornado of 3 May 1999

Storm A Storm B Storm D Storm E Storm C Storm H Storm G 100 km range ring Damage Paths Ground Truth - 53 Tornadoes Courtesy D. Zittel

Trimmed Detections and Ground Truth Damage Paths Hits (142) Misses (25) FAs (21) Courtesy D. Zittel

USA Today Doppler Weather Radar: Measuring the Wind Speed

A Unique Tool for Sampling the 3D Atmosphere in Detail n The radar scans a 3-D conical volume of the atmosphere every 5-6 minutes through a series of 360 degree sweeps…

n … at several elevation angles A Unique Tool for Sampling the 3D Atmosphere in Detail

NEXRAD Facts and Figures n 159 radars (142 in the Continental US) –121 National Weather Service radars –26 Department of Defense radars –12 Federal Aviation Administration radars

NEXRAD Data Types n Archive Level I –analog, time domain output n Level II data –digital base data at full precision and all elevation angles n Archive Level III –digital products + lowest four elevation angles –reduced precision: data are binned n Archive Level IV

n Archive Level I –analog, time domain output n Level II data –digital base data at full precision and all elevation angles n Archive Level III –digital products + lowest four elevation angles –reduced precision: data are binned n Archive Level IV NEXRAD Data Types

The Value – Hazard Detection

The Value – Real Time 3D Analyses

Carbone et al. (2002) The Value – Fundamental Predictability

The Value – Storm Climatologies

The Value – Computer Prediction Models

6 pm 7 pm8 pm Radar Hourly Radar Observations (Fort Worth Shown by the Pink Star)

6 pm 7 pm8 pm Radar Computer Forecast 2 hr 3 hr 4 hr

Fcst w/o Radar 2 hr 3 hr 4 hr 6 pm 7 pm8 pm Radar

Space Shuttle Columbia Debris Trail

Bird Migration Studies

NEXRAD data are *THE* most valuable data for mesoscale meteorology… but they were not available in real time to the academic and corporate communities

The Collaborative Radar Acquisition Field Test (CRAFT) n Begun in fall, 1998 as a collaborative R&D project to demonstrate the real time compression and Internet-based transmission of WSR-88D base (Level II) data n Philosophy: Leverage existing infrastructures (Internet), public-domain software (Unidata LDM), and secure, low-cost services (56K phone lines) n Short-term goals: Storm-scale research and improvement of NCDC archive capability n Long-term goal: National real-time capability for operations, research, and private industry

n Initiated and led by the OU Center for Analysis and Prediction of Storms (CAPS) n National Severe Storms Laboratory (NSSL) n NOAA WSR-88D Radar Operations Center (ROC) n UCAR Unidata Program n National Climatic Data Center (NCDC) n University of Washington TREMENDOUS support from the US Weather Research Program Coordinating Organizations

Foundation #1: NSSL’s RIDDS Courtesy NSSL

Foundation #2: Unidata Local Data Manager (LDM) Courtesy D. Fulker, Unidata

Foundation #3: Data Compression – BZIP2 The Internet Adaptation to LDM by Harry Edmon, U. of Washington

Foundation #4: The Internet The Internet Invention courtesy A. Gore Commodity NGI Internet2 Abilene

Foundation #5 n Incredible cooperation, collegiality, and agency support NCDC

Original Technical Strategy RIDDS WSR-88D At the radar site

Original Technical Strategy RIDDS Linux PC + LDM + Compression ($1000) WSR-88D At the radar site

Original Technical Strategy RIDDS Linux PC + LDM + Compression ($1000) Router ($2000) WSR-88D At the radar site

Original Technical Strategy RIDDS Linux PC + LDM + Compression ($1000) Dedicated 56K line, T1, ($ $6000/year) or NWS LAN Router ($2000) WSR-88D At the radar site LDM Server (OU)

Original Technical Strategy RIDDS Linux PC + LDM + Compression ($1000) LDM Server (OU) Dedicated 56K line, T1, ($ $6000/year) or NWS LAN Router ($2000) WSR-88D At the radar site Users (LDM) Internet or Abilene

Slow But Sure Growth February, 2000

n BZIP2 off-the-shelf compression algorithm –maximum 16:1 in clear air; 8:1 in the worst case; average of about 12:1 n Maximum aggregate data rate is 15 Mbytes/radar/5 min uncompressed = 48 Mbits/sec compressed for entire national network –bandwidth isn’t the main issue, but rather topology, quality of service, and data storage n 56K lines found to be inadequate in some cases (squall lines) – maybe 10% of time –Exceed available bandwidth by 10-20% –Can create latencies of 10s of minutes (more later) –Up to 26% improvement in compression possible in BZIP2, but requires changing code at user end –A pre-processing algorithm (NSSL) provides 10% improvement – no changes in BZIP required Technical Highlights

Humble Beginnings… January, 1999

Eight More in 1 Year January, Radars

Latency Data

The Idea Catches Hold January, 2002

The NWS Makes a Bold Decision Fall 2004

Regional Collection Concept Proposed 5 Yrs Ago NCEP NCDC

Reiterated Fall 2002

n NWS is implementing a data delivery system based upon the CRAFT concept (regional hubs, Abilene as the backbone) n This infrastructure could become THE NWS data and model grid delivery system of the future Current Status

We’ve Come a Long Way! Today – 123 Radars!!

–Initiated after 2001 Stakeholders Workshop –NWS agreed to let OU provide real time Level II data to private companies and others (academia), as R&D partners, in exchange for latency and other information to help evaluate system performance –Any group could participate –Provided an opportunity for private sector partners to gain experience using Level II data and develop new capabilities with them –Had 6 private sector partners –Companies agreed to the collaborative nature of the project, i.e., everyone wins when everyone contributes Private Sector Joins the Effort

The Benefits – Industry Development Baron Services, Inc.

A True Public-Private Partnership Recent NOAA + Pvt Sector Money  $3.0M

n 123 NEXRAD radars are delivering Level II data to University of Oklahoma in real time –Subsequent distribution to more than 30 users including NCDC, NWS HQ, UCAR Unidata, numerous universities and NOAA laboratories, and several (paying) private companies n Extremely high reliability (99+%) with latency less than 10 sec n Top-tier nodes announced during week of 12 April 2004 –University of Oklahoma –Purdue University –ERC Current Status

IRaDS will continue to collect and analyze network information for R&D

Data Ingest is Sized to Handle Future NEXARD Improvements Enhancement Factor of Data Flow Increase (Relative to August 2002) Possible Implementation Date Cumulative Data Flow Increase (Relative to August 2002) 4.1 min VCP 1.24 Jan TCP/IP Wideband Interface 1.00 July km Reflectivity Data 1.56 October deg Azimuth Sampling 2.00 January Doppler Data to End of 2 nd Trip 1.49 October Clutter Filters and SNR Thrshld 1.97 January Dual Polarization 2.32 January

n OU and NSSL have worked with FAA Mike Monroney Aeronautical Center in OKC to transmit TDWR data in real time n Can be extended to all TDWR, ASR and ARSR systems n The idea of using Abilene to transmit data was suggested by Lincoln Laboratory 2 years after CRAFT was underway Weber (2000) Addition of FAA Air Traffic Radars

n Baron Services centrally collects data now (Huntsville, AL) n TV stations only scan lowest elevation angle, which would be very useful in biological target ID and tracking Possible Addition of TV Station Radars Baron Services TV Radar Sites

Linked Environments for Atmospheric Discovery (LEAD) A Cyberinfrastucture for Mesoscale Meteorology Research and Education

Analysis/Assimilation Quality Control Retrieval of Unobserved Quantities Creation of Gridded Fields Prediction PCs to Teraflop Systems Product Generation, Display, Dissemination End Users NWS Private Companies Students Traditional Prediction OBSERVATIONS Radar Data Mobile Mesonets Surface Observations Upper-Air Balloons Commercial Aircraft Geostationary and Polar Orbiting Satellite Wind Profilers GPS Satellites

Analysis/Assimilation Quality Control Retrieval of Unobserved Quantities Creation of Gridded Fields Prediction PCs to Teraflop Systems Product Generation, Display, Dissemination End Users NWS Private Companies Students Traditional Prediction OBSERVATIONS Radar Data Mobile Mesonets Surface Observations Upper-Air Balloons Commercial Aircraft Geostationary and Polar Orbiting Satellite Wind Profilers GPS Satellites The Process is Entirely Serial and Static (Pre-Scheduled): No Response to the Weather! The Process is Entirely Serial and Static (Pre-Scheduled): No Response to the Weather!

Analysis/Assimilation Quality Control Retrieval of Unobserved Quantities Creation of Gridded Fields Prediction PCs to Teraflop Systems Product Generation, Display, Dissemination End Users NWS Private Companies Students The LEAD Vision: No Longer Serial or Static OBSERVATIONS Radar Data Mobile Mesonets Surface Observations Upper-Air Balloons Commercial Aircraft Geostationary and Polar Orbiting Satellite Wind Profilers GPS Satellites

Analysis/Assimilation Quality Control Retrieval of Unobserved Quantities Creation of Gridded Fields Prediction PCs to Teraflop Systems Product Generation, Display, Dissemination End Users NWS Private Companies Students OBSERVATIONS Radar Data Mobile Mesonets Surface Observations Upper-Air Balloons Commercial Aircraft Geostationary and Polar Orbiting Satellite Wind Profilers GPS Satellites The LEAD Vision: No Longer Serial or Static

Mesoscale Weather LEAD: Users INTERACTING with Weather

NWS National Static Observations & Grids Mesoscale Weather LEAD: Users INTERACTING with Weather

NWS National Static Observations & Grids Mesoscale Weather Local Observations LEAD: Users INTERACTING with Weather

Users ADaM ADAS Tools NWS National Static Observations & Grids Mesoscale Weather Local Observations LEAD: Users INTERACTING with Weather

Users ADaM ADAS Tools NWS National Static Observations & Grids Mesoscale Weather Local Observations Local Physical Resources Remote Physical (Grid) Resources Virtual/Digital Resources and Services MyLEAD Portal LEAD: Users INTERACTING with Weather Interaction Level I

Users ADaM ADAS Tools NWS National Static Observations & Grids Experimental Dynamic Observations Mesoscale Weather Local Observations MyLEAD Portal Local Physical Resources Remote Physical (Grid) Resources Virtual/Digital Resources and Services LEAD: Users INTERACTING with Weather Interaction Level II

New NSF Engineering Research Center for Adaptive Sensing of the Atmosphere (CASA) n UMass/Amherst and OU are lead partners n Total funding over 10 years: $100M n Concept: inexpensive, phased array Doppler radars on cell towers and buildings n Adaptive dynamic sensing of multiple targets including aircraft

Thank You Abilene and Internet2!!!