1. MAC Fall Symposium: Learning What You Want to Know and Implementing Change Elizabeth Yakel, Ph.D. October 22, 2010

2. Outline Asking good research questions Identifying the right method to get the answers Operationalizing your concepts Piloting your instruments Implementing the study – Identifying subjects – Sample size Human subjects issues

3. Asking Good Research Questions What do you want to know? Selecting the right method for what you want to answer

4. Implications of Questions Reference – Satisfaction levels; user needs Collections usage – Measuring use; Citation analysis Website or finding aids – Ease of use; usage patterns Instruction – Evaluation of teaching; impact of primary sources on learning

5. Reference User-based evaluation – How satisfied are researchers with public services in my repository? – Getting reseachers’ opinions about operations to improve service User needs – How do researchers’ questions evolve during an archives visit and how can I better support their resource discovery? – Understanding general information seeking patterns and research questions

6. Collections Usage Measuring use – What are the patterns in collection requests and does my retrieval schedule and reading room staffing support this level of use? – Assess workflow around collection retrieval (rate and times for retrieval, time from request to – Assess levels of staffing in the reading room Quality / Nature of use – How do researchers actually use collections in my repository? – Citation patterns / rates

7. Website Ease of use – What problems do users encounter on my website? – Make changes to the website to improve the user experience Patterns of use – How do visitors find my website and what do the committed visitors do when they are there?

8. Instruction Evaluation of teaching – How can I improve my lecture on using the archives? Assessment of learning – What is the impact of using the archives on students’ critical thinking skills?

9. METHODS Non-invasiveInvasive Reference Observation Statistics (# of readers, registrants, questions) Content analysis (online chat or email reference) Surveys Focus Groups Interviews Web site Web analytics Content analysis of search terms Surveys Usability tests Focus Groups Interviews Collections Usage Statistics (circulation) Call slips Citation analysis Surveys Focus Groups Interviews Finding aids Web analytics Content analysis of search terms Observation Web 2.0 (analysis of comments/tags) Surveys Focus Groups Interviews Usability tests Instruction Statistics (# of classes taught, # of participants) # of archives visits after session Survey Grades / Student assessment Surveys Focus Groups Interviews Field experiment

10. Operationalizing Concepts Reference – Satisfaction with services Which services? What is the nature of the satisfaction? – Friendliness of the staff – Time to retrieve materials – Reference process or procedures – Find what you were looking for?

11. Operationalizing Concepts (2) Ease of use – Ability to use online finding aids – Ability to locate key information in < 1 minute (hours, parking) – Specific tasks that you want researches to do on the website

12. Operationalizing Concepts (3) Instruction - How does archives use contribute to critical thinking skills? – Confidence – Use of skills learned in the archives – Explain archives to a peer – Assessment

13. Measuring Your Concepts Lib Qual – Gap measurement – Minimum service level, desired service level, and perceived service performance – Service superiority = Perceived minus desired service – Service adequacy = Perceived minus minimum

14. Measuring Your Concepts Explore your options Qualitative measures – Find what you were looking for? Quantitative measures – Approachability of the reference staff Yes / no Scale

15. Concepts and Instruments ConceptType of instrument Reference Satisfaction with services Work processes Survey Focus groups Web site Ease of useSurveys Collections Usage Nature of useInterviews / Focus groups Finding aids Ease of use (search, navigation, locating important information) Usability tests Instruction Evaluation of orientation Learning Survey

16. Administration Issues ProsCons Paper Immediate Size readily recognizable Allows for a wider set of data Have to enter data Online Data is already entered Allows for skip logic Gives people constraints Hard to assess length Need more contextual cues

17. Pre-Implementation Pilot testing – Staff – Targeted participants 1-2 make changes 1-2 make more changes

18. Population / Sample Reference: – All Researchers using the archives / special collections / Researchers in the past month Collections usage: – All call slips / the past year of call slips Website: – All search terms used / Last 1000 searches Instruction – Students in the classes to which you gave talks / students with a requirement to use the archives

19. Sampling Getting enough “n” Recruiting participants Representativeness Response rate

20. Administration Instructions Getting staff on board Creating a script / email message System for recruiting – Data range – Type of user – Random sampling

21. Common Issues in User-based Research Memory – Recent of contact 3-6 months – Frequency of contact Several times a year Knowledge of Archives – What prompts do you need to provide?

22. Recruiting Website survey – Link on website – Send link to recent Email reference requestors Recent onsite researchers – In-house researchers – Stakeholders – Partner groups (genealogical societies) Focus groups – Composition

23. Recruiting Usability testing – Single individuals representing specific types – “Friendly dyads” Interviews – Single – Researchers working together

24. Representativeness Does the sample mirror or approximate the population? Does the sample truly represent the group you are targeting?

25. Response Rate Surveys – Large surveys 15%-20% – Archives surveys (n=50) – Percentage of respondents mirrors percentage of like users? – Responses per question

26. Human Subjects Check with your IRB if you have one – Research for internal improvement – Publication Incentives Consent Responsible data management