1. Cooperative Institute for Research in Environmental Sciences UNIVERSITY OF COLORADO BOULDER and NOAA Evaluating the Climate.gov Web Portal: A Key Resource for Climate Information Susan Lynds, Susan Sullivan, Anne Gold, Amanda Morton University of Colorado Boulder Karin Kirk Independent Educational Consultant

2. Cooperative Institute for Research in Environmental Sciences UNIVERSITY OF COLORADO BOULDER and NOAA Climate.gov: Climate Information, Data, and Resources

3. Cooperative Institute for Research in Environmental Sciences UNIVERSITY OF COLORADO BOULDER and NOAA Interactive Dashboard

4. Cooperative Institute for Research in Environmental Sciences UNIVERSITY OF COLORADO BOULDER and NOAA Four Primary Audiences  General Public (including media)  Scientists  Educators  Decision Makers

5. Cooperative Institute for Research in Environmental Sciences UNIVERSITY OF COLORADO BOULDER and NOAA Climate.gov Evaluation NOAA conducts regular usability studies of Climate.gov using the Quality of Relationship (QoR) metric. QoR is adapted for governmental and non-profit organizations from the American Customer Satisfaction Index. 2010-2012 University of Wisconsin Madison 2013-2014 CIRES Education Outreach, University of Colorado Boulder

6. Cooperative Institute for Research in Environmental Sciences UNIVERSITY OF COLORADO BOULDER and NOAA Quality of Relationship (QoR) Approach QoR evaluates r elationships between an agency and its audiences along five dimensions:  Awareness—know exists, what it does, and why  Trust—accurate, credible, and authoritative  Satisfaction—relevant, reliable, and complete  Usability—easy to use  Control Mutuality—interaction with the agency is possible

7. Cooperative Institute for Research in Environmental Sciences UNIVERSITY OF COLORADO BOULDER and NOAA Research Questions  To what extent do the four audiences perceive that the Climate.gov site demonstrates Quality of Relationship elements?  To what extent do audiences exhibit cross-section behavior on Climate.gov?  How does social media affect Climate.gov visitorship?  How can we characterize the visitorship?  In what ways should the site be further developed to better serve visitors needs and expectations?

8. Cooperative Institute for Research in Environmental Sciences UNIVERSITY OF COLORADO BOULDER and NOAA Evaluation Methods  Literature review informed the evaluation plans  Google analytics  Online (pop-up) survey (3,227 respondents)  Usability studies (24 participants), selected by survey-screening for representatives of each audience; used virtual webinar session with scenario-based tasks  Telephone interviews with power users (6 users) from each audience

9. Cooperative Institute for Research in Environmental Sciences UNIVERSITY OF COLORADO BOULDER and NOAA Evaluation Methods

10. Cooperative Institute for Research in Environmental Sciences UNIVERSITY OF COLORADO BOULDER and NOAA QoR Values Derived From Pop-Up Survey Data  Survey ran May 13-June 30, 2014  Survey appeared after 30 seconds of visit time  Response rate ~6% overall  7-10 questions, depending on skip logic  Opt-out option  3,227 respondents

11. Cooperative Institute for Research in Environmental Sciences UNIVERSITY OF COLORADO BOULDER and NOAA QoR Values Derived From Pop-Up Survey Data QoR values 1.A score is assigned for each response 2.Percent of possible score is the QoR value

12. Cooperative Institute for Research in Environmental Sciences UNIVERSITY OF COLORADO BOULDER and NOAA Results: Quality of Relationship (QoR)  Awareness=61%  Slightly lower than 2010-2012, likely due to survey respondents being drawn entirely organically from visitors to the site, without any auxiliary recruiting.  Trust=83.7%  Higher than 2010-2012, testimony to authoritativeness & relationship-building with NOAA  Satisfaction=89%  Higher than 2010-2012; 91.6% of respondents are likely or very likely to suggest Climate.gov to others.  Usability=86%  Higher than 2010-2012; usability studies provided further insights into this visitor experience.  Mutuality of Control=61%  Slightly lower than 2010-2012; again, survey respondents were not recruited for this survey.

13. Cooperative Institute for Research in Environmental Sciences UNIVERSITY OF COLORADO BOULDER and NOAA Results: Quality of Relationship (QoR)

14. Cooperative Institute for Research in Environmental Sciences UNIVERSITY OF COLORADO BOULDER and NOAA Results: Quality of Relationship (QoR)

15. Cooperative Institute for Research in Environmental Sciences UNIVERSITY OF COLORADO BOULDER and NOAA El Niño Outlook 6/13/14 Winter Outlook 10/16/14 White Christmas, Arctic Report Card 12/13/13  Social media and other government websites provide many visitors.  Facebook and Twitter Climate.gov posts on popular topics drive spikes in visitors. Results: Web Analytics

16. Cooperative Institute for Research in Environmental Sciences UNIVERSITY OF COLORADO BOULDER and NOAA  Visit characteristics during survey period were similar to visits overall, with slightly more referred from social media than usual (due to the El Niño post on 6/13/2014).  Survey respondent geographic distribution was similar to visitor distribution overall—all 50 states and 76 countries.  Due to the 30-second delay for the pop-up, survey respondents were more engaged (more pages viewed, more time on site, lower bounce rate) and more likely to be return visitors Results: Web Analytics During Survey

17. Cooperative Institute for Research in Environmental Sciences UNIVERSITY OF COLORADO BOULDER and NOAA  News & Features provides 75% of the visits/visitors but a higher-than-average bounce rate.  Maps & Data (9%) has highest pageviews per session/user.  Teaching Climate (8%) has the highest average session duration.  Supporting Decisions (1%) has highest percent of return visitors.  Most visitors arrive via referrals (especially from.gov site and social media). Results: Web Analytics Overall

18. Cooperative Institute for Research in Environmental Sciences UNIVERSITY OF COLORADO BOULDER and NOAA Overall, the site’s clean design and attention to engaging graphics is effective. Images and summaries especially appreciated. A few of the key points were  An improved search algorithm would be helpful.  Presenting search results in newest-to-oldest order would be best; other enhancements include icons for resource type, color coding to make scanning easy.  Maps & Data functionality could be made more intuitive.  Educators used resources from the entire site. Results: Usability Studies

19. Cooperative Institute for Research in Environmental Sciences UNIVERSITY OF COLORADO BOULDER and NOAA  Develop better connections between website sections and pathways through the site, especially from News & Features to other sections. This will improve visitor awareness of the site, retain visitors, and enhance the sense of relationship between users and Climate.gov.  Enhance mobile device access; currently 22% of visitors (lower than market average).  Mobiles have much higher bounce rate, fewer pages, and lower visit time.  More mobile visitors are new visitors than the average for the site. General Recommendations

20. Cooperative Institute for Research in Environmental Sciences UNIVERSITY OF COLORADO BOULDER and NOAA For more information, contact Susan Lynds 303-492-1714 Susan.Lynds@colorado.edu Susan Sullivan 303-492-5657 Susan.Sullivan@colorado.edu http://cires.colorado.edu/education-outreach/ Sponsor: NOAA; Award No: NA13OAR4310210