1. Lessons for CDAC from Environmental Data Dr. William B. Gail Co-Founder & Chief Technology Officer Past-President © William B. Gail 1

2. Value of Environmental Data to CDAC WHAT – Includes weather, water (land, oceans, ice), climate, air quality, ecosystems – with large, time critical data sets WHY – Experience with environmental data provides valuable lessons to CDAC HOW – Focus on NOAA as primary element of Commerce involved © William B. Gail 2

3. Much More than Weather “We’re a science-based services agency, here to provide local communities, states, businesses and individuals with the information each needs to make smart decisions when it comes to the oceans and atmosphere. I call this information environmental intelligence.” © William B. Gail 3 Kathryn Sullivan, NOAA Administrator Keynote at American Meteorological Society Annual Meeting 2014

4. Discussion Framing Where does data stop and intelligence start? How do we value environmental information? What kind of data does NOAA use/produce, and how does it do so? What data-related issues has NOAA faced © William B. Gail 4

5. Financial Value of Environmental Data © William B. Gail 5 On average, weather variability alone alters economic output up to 3% on average at the state level in the US from one year to the next – some states more than 10%. “The U.S. public obtains several hundred billion forecasts each year, generating $31.5 billion in benefits compared to costs of $5.1 billion.” From Lazo et al (2009, 2011), Bulletin of the American Meteorology Society From Kraznow et al (1986), Policy Aspects of Climate Forecasting

6. Time Value of Environmental Data Data largely associated with prediction based on timely physical modeling, not on data mining or statistical inference © William B. Gail 6 Deterministic (weather) Statistical (climate) 2 weeks Overall Forecast Lead Time Forecast Value ? Time Value of Meteorological Forecast Data

7. Types of NOAA Data OUTPUT from NOAA – Weather stations and satellite data – Numerical Weather Prediction (NWP) models – HPC – Warnings and analysis – automated and human – Guidance: resilience planning, response planning INPUT to NOAA – Existing: weather station networks, lightning observation networks – Emerging: commercial weather satellites, NWP, alerting © William B. Gail 7

8. The Weather Enterprise Context PUBLIC-PRIVATE PARTNERSHIP ISSUES 1.Commercial competes with free 2.Commercial leverages NOAA investment 3.Who does what 4.Enables startups 5.Value of standards 6.Joint R&D 7.Goal: serve public Scientific & Professional Societies Government Academia Private Sector Science & Innovation Efficiency & Customization Public Welfare & Major Infrastructure National Academy of Sciences 2003 © William B. Gail 8

9. The International Context Data Use Driven by Global Problems – Types and uses of environmental data are generally universal, independent of nations and cultures – Problems and models are often global-scale, requiring global data as input and output Organizations Cross Boundaries – Government activities are driven by strong international treaties and partnerships (World Meteorological Organization (WMO) for weather and climate, UN Convention and other organizations for oceans) Open and Shared Data Sometimes Conflict – Nations treat data both as a national asset and a shared resource Regional is Becoming Global – Regional capabilities (national met offices) are being superseded by global capabilities, including growing role of commercial providers Commercial Role is Widely Underdeveloped © William B. Gail 9

10. The Public Welfare Context Data is Considered Public Welfare – Environmental information – particularly weather – is considered public welfare data, particularly when safety involved Shared Progress is the Norm – Enabling developing nations is considered an obligation of developed nations Authoritative Sources are Valued – Many nations believe they must be the sole authoritative source of such data, or treat data as a national asset, to avoid public confusion © William B. Gail 10

11. The Open Data Context WMO Resolution 40 – “Members shall provide on a free and unrestricted basis essential data and products which are necessary for the provision of services in support of the protection of life and property and the well-being of all nations...” – “Recognizing further... The risk arising from commercialization to the established system of free and unrestricted exchange of data and products” Issues with Commercial Data – Risk: data withheld from non-paying customers is safety issue – Benefit: commercial innovation, efficiency – Challenges: open data has only one paying customer, so costly © William B. Gail 11

12. NOAA Data Initiatives Environmental Information Services Working Group (EISWG) of the NOAA Science Advisory Board – White Paper 2011 – “An Open Weather and Climate Service is proposed in which both NOAA and the community share equal and full access to NOAA information and development.” – NOAA response: supportive NWS CRADA - 2014 – Plan halted after concerns raised about working through one partner NOAA Big Data RFI – Feb 2014 – “... intelligently positioning NOAA’s vast data holdings in the cloud, to be co- located with easy and affordable access to computing...” – “...intended to inform NOAA on the feasibility of partnering with one or more industry partners using no-cost arrangements”. High Impact Weather Prediction Project (HIWPP) Open Data Initiative - Early Access to R&D Data – Feb 2015 – Ongoing initiative, welcomed by community © William B. Gail 12

13. Challenges in Practice Expectations mismatch – Issue: problem is harder than most outsiders think Multiple and often unexpected barriers – Technical, resources, policy, security Legacy systems sometimes preclude data access – Example: airport weather observations Limited bandwidth to outside – Issue: how to use commercial gatekeepers Which data are actually useful – Example: intermediate model computations What gets done first (prioritization) – Examples: weather versus ocean data, which model runs Who should do what – Issue: government versus commercial roles © William B. Gail 13

14. Summary NOAA, and NWS in particular, provides a good example for many Commerce data issues CDAC will discuss Some issues are unique to environmental data, but many being faced by NOAA have broad applicability to CDAC © William B. Gail 14