1. Using Customer Feedback to Improve Data Delivery
Colleen Blessing
U.S. Dept. of Energy
The Hague Sept. 2005

2. Two Customer Feedback Efforts
The American Customer Satisfaction Index Survey (ACSI)
Survey of customer satisfaction
Run by the University of Michigan
Used by many government agencies
Uniform and independent measure
Website Usability Testing
Evaluating a new web design

3. ACSI Methodology Includes 15 standard questions
Content, navigation, search, site performance
EIA added 11 custom questions
demographics, suggestions for improvement
Invitation to survey tagged on L2 pages
Survey began August 2004
Received 4650 responses by July 2005

4. Concerns about the Methodology
Does ACSI capture a representative sample of EIA’s customers?
Sample could be skewed because:
Pop up blocking technologies prevent customers from seeing the survey.
Repeat customers only respond to the survey once.
Some EIA customers under deadlines are reluctant to respond to a web survey.
The ACSI invitation is not compatible with all EIA web pages.
Customers’ browsing techniques – search and bookmarking.

5. ACSI Results—Overall Scores
The overall score is an index rather than a % satisfaction score.
Search and Navigation are consistently rated lowest.
Content is consistently rate high.
Respondents indicate high likelihood to return and to recommend the site to others.
Similar results to those from Statistics Canada’s survey – gaps between satisfaction and importance are largest for search and ease of finding information.

6. Countries with the most ACSI Respondents (#)
More than one-third of ACSI respondents are from outside the United States (n=4650 through July 2005)
ACSI Respondents by Country
Countries with the most ACSI Respondents (#)
Canada (234)
France (60)
UK (186)
Brazil (57)
Mexico (76)
Italy (53)
Australia (75)
Germany (49)
India (66)
Venezuela (44)

7. Main Customer Groups = Business/Industry and Research/Consulting (n=4650 through July 2005)
Distribution among sectors is similar for U.S. and International respondents

8. Most ACSI respondents are familiar with the energy industry (n=4650 through July 2005)
More than 80% of respondents were at least somewhat familiar with the energy industry
Cross tabs indicated that international respondents were less familiar with the energy industry
Familiarity with the energy
industry and energy terms

9. Most respondents visit EIA’s website regularly but 30% are first-time users (n=4650 through July 2005)
Nearly two-thirds of respondents visit EIA’s website at least monthly
30% of respondents were first-time users
Cross tabs indicated that more international respondents were first time users
How frequently do you visit this site?

10. A New ACSI Question: How do you navigate the site
A New ACSI Question: How do you navigate the site? (n=1538 from March to July 2005)

11. Who Has Trouble on our Site?
Customers less successful finding information are:
International users
Those unfamiliar with the energy industry
First-time visitors
People who used the search box
Respondents’ perceptions of data quality are highly correlated with their overall experience at the website.

12. Respondents who were unsuccessful in finding information were less satisfied with EIA’s website (n=3325 through March 28, 2005)
“What is your overall satisfaction with this site?”

13. Respondents who did not find information rated the quality of EIA’s information much lower (n=3325 through March 28, 2005)
“Please rate the quality of information on this site.”

14. Redesigning the Website: Testing
ACSI results showed we needed to improve navigation and information architecture.
Needed to help first-time and non-energy-expert users
Usability testing on new designs
Paper tests first
Tests in the lab
20 common user questions
Turned off search function
- Tested 36 people

15. Results of the Usability Testing
Much higher finding rate
2001=40% 2002=53% 2005=79% corp=66%
What worked well
Hints, hot box, publications list, less clutter, major link names, more consistency on L2
What didn’t work very well
Global navigation, stovepiping, inconsistencies on L2, glossary at top of page, frustration not having search function

16. New EIA Home Page