1. Tell a Vision: 3 Vignettes
Denise A. Troll
Ex-tinguished Fellow, Digital Library Federation
Associate University Librarian, Carnegie Mellon
DLF Forum – November 2001

2. Assessment Scene Investigation
How & why libraries assess
Assessment methods & tools
Issues & challenges
Organization of assessment
Trends indicative of practice
Agenda for future work

3. Tell a Vision: 3 Vignettes
ASI Methods
Qualitative DLF telephone survey
Collection & review of instruments
Desk-based research
Peer review
Follow-up quantitative survey
What data are being gathered?
Why, how, & by whom?
Who uses the results?
Why, how, & to what effect?
What worked & what didn’t?
Denise A. Troll,

4. Tell a Vision: 3 Vignettes
ASI
Rate
92%
Institutions
86%
Individuals
Contacted Participated
Denise A. Troll,

5. Tell a Vision: 3 Vignettes
Case ID ASI-DT
EVIDENCE:
Similar purposes & methods
Different organizations & approaches
Different resource commitments & expertise
Diverse outcomes
Similar purposes & methods
Few standard instruments
Little comparable, generalizable data
Different organizations & approaches
Systematic or ad hoc
Centralized or decentralized
Different resources & expertise
Different outcomes – study results can
Reinforce librarian assumptions & preferences
Challenge librarian assumptions & preferences
Provide conflicting, ambiguous, misleading, or incomplete information
Be used
Never be used
Denise A. Troll,

6. Tell a Vision: 3 Vignettes
Case ID ASI-DT
EVIDENCE – Diverse outcomes
Reinforce assumptions & preferences
Challenge assumptions & preferences
Provide problematic information
Often a breakdown in the research process
Incomplete, conflicting, misleading, ambiguous
Denise A. Troll,

7. Tell a Vision: 3 Vignettes
Articulate research purpose
Implement the plans
Assess available resources
Choose research method
Present, interpret, & plan
Assign responsibilities
Prepare schedule
Analyze & interpret
Is the schedule realistic and comprehensive?
How many people are involved in interpreting the results and planning what to do with them?
How many people are involved in implementing the plans? What are their priorities?
Prepare & test instruments
Conduct
Denise A. Troll,

8. Tell a Vision: 3 Vignettes
Case ID ASI-DT
CRIME – Assessment slaughter 2
Purposeless data collection
Discouraged staff & users
Inadvertent negligence
Involuntary harm
Assessing rapidly moving targets
Incomplete, conflicting, misleading, ambiguous
Denise A. Troll,

9. Issues & Challenges

10. Tell a Vision: 3 Vignettes
Surveys
Survey fatigue
Cost & benefits
Composing questions
Lack of analysis & application
Lack of resources, comprehensive plans, & commitment
Recruiting representative research subjects
ONE OF YOU IS ABOUT TO LEAVE WITH NOTHING
Relative ranking of priorities & frequencies
Open-ended questions
Follow-up research
Denise A. Troll,

11. Focus Groups & Protocols
Tell a Vision: 3 Vignettes
Focus Groups & Protocols
Lack of skilled moderators & observers
Lack of prior agreement about what to observe & how to record the observations
Analyzing, interpreting, & using qualitative data
Recruiting representative research subjects
Lack of resources, comprehensive plans, & commitment
WHO IS TOO SLOW TO STAY?
Commitment – to architectural changes as well as surface level changes
Denise A. Troll,

12. Transaction Log Analysis
Tell a Vision: 3 Vignettes
Transaction Log Analysis
Volume of data
Getting the right data
Analyzing & interpreting the data
Managing, presenting, & using the data
Lack of resources, comprehensive plans, & commitment
User privacy
WHO IS IT THAT NEEDS TO BE DISPATCHED WITHOUT DELAY?
Lack of standards & adequate software
Denise A. Troll,

13. Perceptions of Weakest Links

14. Perceptions of Weakest Links

15. Conclusions & Next Steps
A S S E S S M E N T

16. Tell a Vision: 3 Vignettes
Organize Assessment
Develop case studies & best practices
Build consensus for e-metrics lite
Gather baseline data & benchmarks
Develop specification for an MIS
E-metrics lite – a limited subset of digital library statistics and performance measures to facilitate gathering baseline data and enable comparisons 
A specification for the design and functionality of an MIS to capture traditional and digital library data and generate composite measures, trend data, and effective graphical presentations
Case studies of successful organizations & data-based strategic planning
Denise A. Troll,

17. Develop How-To Manuals
Tell a Vision: 3 Vignettes
Develop How-To Manuals
Research process & methods
Planning & scheduling
Resource commitment & skill development
Field-tested schedules, budgets, workflows, & instruments
Organizing assessment as a core activity
Collecting only purposeful data
Securing commitment
Training
·        E-metrics lite – a limited subset of digital library statistics and performance measures to facilitate gathering baseline data and enable comparisons 
·        Case studies of
-        The costs and benefits of different approaches to compiling, presenting, interpreting, and using vendor TLA data in strategic planning
-        How institutions successfully organized assessment as a core library activity
A specification for the design and functionality of an MIS to capture traditional and digital library data and generate composite measures, trend data, and effective graphical presentations
Denise A. Troll,

18. Acquire Interpretive Framework
Tell a Vision: 3 Vignettes
Acquire Interpretive Framework
DLF & Outsell, Inc. survey of students & faculty
Understand why library use is changing
Evaluate current & future role
Inform strategic planning & resource allocation
Focus library efforts appropriately
Facilitate development of tools & services
What resources do students & faculty use?
How do they locate, evaluate, & use them?
Do the resources & activities differ across user groups, disciplines, or type of institution?
What needs are not being met?
Random sample >3000 students & faculty -- Public universities -- Private universities -- Liberal arts colleges
1.      The scholarly information landscape is a complex and evolving combination of information resources, only some of which are managed by academic libraries.
2.      Academic libraries have a distinct and vital role to play in the evolving networked information landscape.
3.      Academic libraries must promote themselves in the networked environment because they are not the only or most accessible provider of scholarly information in this environment.
4.      The nature and use of the scholarly information landscape varies by institution type and academic discipline.
5.      Information use is conditioned by many factors, including speed of access, ease of use, and the quality of the resource.
Denise A. Troll,

19. Thank You
And thanks to Dan Greenstein, CLIR, DLF library directors, survey respondents, ARL, authors, & readers