User studies: contributing to developing regional decision support for climate services Andrea J. Ray, Roger S. Pulwarty and Robert S. Webb NOAA-CIRES Climate Diagnostics Center Climate Prediction Assessments Workshop Tallahassee, FL 9-11 March 2004

“The timely production and delivery of useful climate data, information, and knowledge to decision makers.” National Research Council, 2001

NRC recommends that a regional focus is needed to address societal needs, developed through regional decision support systems. Pillars of climate services include: – Regional decision support networks, – Research, – Observing systems, and – Mechanisms to enable user-centric design and improvement.

To be effective, regional decision support systems must not only provide data and forecasts, but also determine: – what problems and questions are important and critical, and why, to be determined by consensus between users and providers of products; – what information is needed to solve problems, on short and long-term time scales; – what transaction costs are involved in developing and maintaining long term partnerships and interactions.

Many “regional decision support,” networks exist, – NOAA weather forecast offices,data centers, river forecast centers, – Regional climate centers, – State climatologist offices, – Agricultural extension services – Some private companies These networks provide data and forecasts, but in order to meet the NRC user-centric goals of climate services, these organizations would also: – adapt products to meet specific user needs, – support major user groups, – encourage dialogue with users, and – develop a regional focus to address societal needs. These networks generally are not set up to do effective user- studies

Effective user-centric design and implementation To achieve user-centric design and implementation these interactions must: – identify entry points for information within existing decisionmaking and the space for communicating this information; – identify research needed, including strategic, applied, and adaptive; – create nodes for users to engage with researchers and providers, to help determine what products are to be developed and tested; – Characterize needed products: e.g., scale, units, formats, how accessible, when issued. – Inform the development of alternative decisionmaking strategies as needed Mechanisms for studying users need to be built in to regional networks.

Effective user-centric design and implementation Sustained and systematic communication is required on three fronts: – between users and providers (NOAA and regional); – between researchers and providers, and – between researchers and users. Why user studies? – More than consultancy – More than a simple conversation or communication (not anecdotal) – Understand the complexity of user context and differences in perspectives between users and scientists

Designing effective user studies User-study approach Based on case studies Based on – need to get past differences in perspective; – the innovation-decision process; – definitions of “use” Dimensions to characterize about a user-case

Differences in perspective FactorScientist’s perspectiveWater Manager’s Perspective Identifying a critical issue Based on a broad understanding of the nature of water management Based on experience of a particular system Time frameVariableImmediate (operations) Long-term (infrastructure) Spatial resolutionDefined by data availability or fundingDefined by institutional boundaries or authorities GoalsPrediction Explanation Understanding of natural system Optimization of multiple conditions and minimization of risk Basis for DecisionsGeneralizing multiple facts and observations Use of scientific procedures and methods Availability of research funding Disciplinary perspective Tradition; Procedure Professional judgment; Training Economics; Politics Job risks ExpectationUnderstanding Prediction Ongoing improvement (project is never actually complete) Statistical significance of results Innovations in methods/theory Accuracy of information Appropriate methodology Save money and time; Protect the public; Protect their jobs, agendas or institutions Product Characteristics Complex Scientifically defensible As simple as possible without losing accuracy Importance of context FramePhysical (atmospheric, hydrologic, etc.) conditions as drivers Dependent on scientific discipline Safety and well being Profit Consistency with institutional culture, policy, etc. Nature of UseConceptualApplied

Innovation-Decision process: conceptual steps in deciding whether to adopt an innovation Knowledge phase: learns about the existence of an innovation, perhaps thru an outreach function Persuasion phase: forms a favorable/unfavorable opinion Decision phase: engages in activities leading to adopting or rejecting Implementation: puts the innovation into use Confirmation phase: decision to adopt is reinforced, or could be reversed if experience is adverse (Rogers,1995)  Path towards adoption and use: studies and mechanisms for developing user-centric products should take advantage of this

Definitions of “use” Consult: the product is consulted, e.g., looked up on a web page or received as a briefing or from other source (type1); Consider: after consulting the product, it is considered in management deliberations as a factor potentially influencing decisions, but not necessarily in operational models (type 2); - forecasts may be used in this way when they are not in appropriate forms for use in operational models Incorporate: some form of the forecast is incorporated into an operational model that is utilized in operational decisions (type 3). Dialogue about risks: communication of risk, i.e. the forecast is used to communicate with other managers and stakeholders about the risk of certain conditions and about the need to take actions, or to justify actions (type 4).

User Study Approach One mechanism for assessing user needs Dimensions to characterize about the user-context: –Identify critical problems which are sensitive to climate –Identify decisionmakers and their key stakeholders –Assess how climate variability interacts with their critical problems Decision calendar may help organize recurring decisions and show when products are needed –Identify user groups who may be willing partners in testing and prototyping this new technology Potential users in a rapidly evolving phase of their critical problem may be more open to interacting as partners Social changes, legal or policy change, or a climate event such as drought or flood

User group is the reservoir managers in the Colorado headwaters and Gunnison basins who operate reservoirs originally built to provide reliable irrigation supply, hydropower, and recreation Redlands Diversion dam and fish ladder, Gunnison river near Grand Junction, CO Case study: Multi-purpose Reservoir Management in the Upper Colorado River Basin

Dimensions for Aspinall Unit Case Critical problem sensitive to climate –Evolving reservoir management in which demands now are closely balanced with supply but new uses are being legally proscribed –More intensely managed water system is more sensitive to shortage because there is less buffer in the system Decisionmakers and their key stakeholders –USBR has authority to manage, but USFWS, NPS, have interacting legal authorities, other stakeho How climate variability interacts w/ the critical problem –Decision calendar for annual operating plan helped organize recurring decisions and ID potential climate User groups who may be willing partners/adopters –Deadlines to write ESA recovery plans and formal filing for a federal reserve water right for a national park are requiring USBR to accommodate these in their operations –2002 drought as a climate event (not forecasted) also resulted in finding flexibility and new ways of operating

Potential Uses in the Aspinall case Identified a number of potential uses of seasonal climate forecasts in their operational models, as well as shorter term forecasts However, seasonal forecasts are currently not available in a form that can be incorporated into their operational models. Current use is to consult these products and consider them in management deliberations Aspinall decisionmakers use climate information (e.g. during drought) in a dialogue with their stakeholders about the risk of certain conditions and about the need to take actions, or to justify actions. Projects with Martyn Clark (WWA) and Dave Brandon (CBRFC) are working to translate both MRF forecasts and CPC seasonal forecasts

Conclusions I Regional decision support networks exist, but are not designed to carry on the systematic study of and interaction with users to ensure the development of user-centric products and services Mechanisms need to be built into the regional decision support networks These mechanisms could ensure user-centric design and implementation by: – identifying entry points for information within existing decisionmaking and the space for communicating this information, e.g., through basin water management groups, water availability task forces, emergency management meetings – identifying research needed, including strategic, applied, and adaptive; – enabling RDS networks to function as nodes for users, researchers and providers to engage in dialogue and to help determine what products are to be developed and tested; – effectively characterizing needed products and synthesize user needs from among many interactions (beyond consultancy)

Conclusions II User studies are a means to create and sustain the communication and learning required for user-centric design and development – and also a cost. – create a dialogue about climate risks and not just provide information; – create a dialogue about the types of climate products desired by a broad spectrum of users by integrating knowledge about what climate products are needed. NOAA’s mechanisms for developing user-centric products and services need to: – Recognize that differences in perspective exist and limit scientists ability to create user-centric products – unless we deliberately work to see other perspectives – “Use” of products is broader than just in operational models: recognize that there is a range of uses and create products for all types. Evaluation of “use” should consider all types. – Adoption is a process: develop mechanisms to draw potential users along this path and evaluate whether activities support evolution of use Training needed across the RDSS networks in how to conduct user- assessments

EXTRAS

Summer season forecast for Peak Augmentation planning Provide for late Summer/early Fall irrigation while maintaining target flows Next water year runoff unknown, reserve water until February snowpack observations Winter season precipitation forecast for Fall release decisions Winter releases based on January/February snowpack observations Operational issues Potential use of forecasts Legend: Planning Process Winter/Spring forecast for Winter release decisions Peak Flow Augmentation - fill curve Week 2 forecasts for Peak Augmentation Peak Flow Augmentation releases Plan releases for Summer irrigation & hydropower Week 2 forecasts for Summer irrigation & hydropower release decisions Reservoir Management Decision Calendar Provide for Summer irrigation & hydropower needs while maintaining target flows Water Year Planning Next Water Year Planning Ray et al. (2000) Figure 4-28

Differences in perspective Identifying a critical issue Time frame Goals Basis for decisions Expectiation Product characteristics Frame

Characteristics of effective user studies Dimensions to characterize about a user-case

Desired information not available Understanding Barriers New Services How to they understand climate? What do they use now? Can we produce forecasts in a manner related to how they think about climate? Are the problems critical to the user influenced by climate? If so, are there products available to address these? Are there circumstances that minimize constraints? User studies seek answers to the right hand questions Other factors more important Constraints prevent flexible response Fail to understand