1. Our Most Important Challenge Raising the Assessment Bar: A Challenge to our Community Rick Luce, Emory University Library Assessment Conference Seattle – August 4, 2008

2. Environmental Scan 1. University mission evolving: Education reform and the Spellings commission focus on outcomes and accountability Globalization and competition 2. New research methods in a networked world: Rise of eScience / eResearch = new ways to work, new needs and expectations Data science & data scientists require new organizational environments 3. Social drivers: Technology enabled social tools to connect & collaborate

3. My View after 2 years at Emory Aggressive Strategic Plan completed in first 4 months: Strategic direction: (1) digital innovations, (2) special collections, (3) delivery of 21C access, resources, and services $100M requested in new funds over 5 years Concept approval for ~$36M for 3 new capital projects Implement annual Business Plan – reviewed quarterly None of which utilize current ARL statistics

4. We Need a Systems Approach The human body is a system, our subsystems work together to keep us healthy Research libraries are systems, requiring a management system to keep the subsystems working together to be healthy Anyone who learns to see the organization as a system can never again feel satisfied with improvement initiatives which simply change staffing and the org chart but do not tackle the system itself

5. Where Assessment Fits Assessment – a method of planning for improvement Catalyst for organizational change (not a quick fix) Gain staff understanding for need for improvement & commitment to shared improvement goals Ideally underpinned by a performance measurement matrix balancing: Quality = customer defined goodness – internal & external Time = speed, how fast is the response, agility Cost = resources spent on people, processes, or organizational shifting or rework

6. Performance Measures: our Vital Signs Statements without performance measures are wishful thinking -- without data, we dont know We all have volume or transaction data - provides no process insight Move focus on product metrics to process metrics Process performance (statistics, run charts, variation) Getting to the right metrics What is the value equation? How do we compare & differentiate ourselves? Focus on : Customer, process, sponsorship metrics

7. Hedgehog View: Constancy of Purpose What are we best at? What are we passionate about? What drives our economic (or value) engine? Adapted from: Jim Collins. Good to Great. 2001

8. ID Key Success Factors Characteristics of successful organizations: * 1. Do something others cannot do 2. Do something well that others do poorly, or 3. Do something others have great difficulty doing well ~10-15% (max.) of research libraries content / services are unique What % of the budget resources support that? *Prahalad and Hamel. The Core Competencies of the Corporation. HBR, May/June 1990.

9. Management Principles of Successful Organizations Strong customer focus Effective leadership Continuous improvement and learning Management by fact Fast response Long-range view of the future Results orientation Cooperation, teamwork, partnering

10. A Value Proposition Example (LANL) Scientific Productivity & Competitiveness Enhancing research productivity and competitiveness ROI = $4.50 : 1.00 Quality and Business Focused 10 year average: 96% satisfaction rate; 25% delighted Reduced cost per transaction by a factor of 16 Quality New Mexico Roadrunner awards: 1997 & 2000 Federal Library of the Year 1999 Worlds Best Science Library 2005 Blue Ribbon Review

11. Follow the Leaders Malcolm Baldrige National Quality Award: recognizes organizations practicing the most effective management methods High performance is sustainable due to good management practices Analysis of 600 winners over 10 years: growth = > 2.5 times as fast as peers, more than 2X more profitable Examines approach, deployment and results

12. Baldrige Criteria for Performance Excellence 1. Leadership 2. Strategic Planning 3. Student, Stakeholder, and Market Focus 4. Measurement, Analysis, and Knowledge Management 5. Workforce Focus 6. Process Management 7. Results Requires a system both in approach and deployment

13. Baldrige Applications: Lessons Learned Accelerated learning using Baldrige framework System is tough to integrate all at once Be patient, have discipline Deming Importance of supplier partnerships Difficulty of language translation Benchmarking data - time series data for competitors – couldnt be obtained from libraries

14. Benchmarking Process for gaining and applying knowledge to improve business process performance from a study of current practices. A means of using data to identify magnitudes and reasons for variances in performance. Intent: comparative process data, best practices Analyze the operation, know the competition & industry leaders Incorporate the best of the best - become the new benchmark

15. Missing from Research Library Portfolios Customer satisfaction index: (delight & loyalty) and perceived value Product / service quality (defined by the customer) Process and operational performance - cycle time, productivity Employee satisfaction – learning, morale, training, alignment of strategy direction and rewards Measuring supplier performance - quality, process variables, price competitiveness, overall ease of doing business Financial: cost/value matrix, return on investment, cost avoidance

16. Customer Satisfaction Metrics Maturing our satisfaction assessment Level 1 – Satisfaction surveys: happiness meters Level 2 - Whats important analysis of customer importance & satisfaction levels Level 3 - How do we rate against best in industry See it from the customers eyes

17. Measures that matter Align library vital signs with the organizations drivers Quality of product and processes Innovation Research leadership Brand identity Growing market share Reducing new product development time Ability to attract and retain employees Credibility

18. Avoiding Pitfalls Measures that dont focus on strategy No accountability Too many initiatives Forgetting larger organizational drivers Lack of discipline Insulating researchers and managers from scholarly communication issues No action without a plan, no plan without data

19. 1. Customer focus: Satisfaction, loyalty, value-added 2. Product quality: E.g., accessibility, usability, accuracy, completeness 3. Operational process performance: Productivity, competitiveness, cycle time 4. High performance workforce: Organization performance assessment, formal process changes, employee satisfaction 5. Prestigious reputation -- output results Strategic performance results, benchmark results, external assessment scores Business Scorecard - Desired Business Results

20. A Tale of 2 Libraries: Budget Allocations $M

21. Payoff for Successful Quality Implementation

22. Any road will do if the destination is unknown The journey to truly superior performance is neither for the faint of heart nor for the impatient. The development of genuine expertise requires struggle, sacrifice, and honest, often painful self- assessment. HBR: Anders Ericsson (FSU), Michael Prietula (Emory), Edward Cokely (Max Planck)

23. If you wish to do something for the community, build a road. If you wish to do something better for the community, build a bridge – Chinese Proverb Together lets build the bridge to a new level of assessment practice, supporting continuous improvement and focusing on outcomes and impact.