Two birds with one evaluation How to assess your services and publish too? Suzana Sukovic, Jane Van Balen, Ashley England & Janet Chelliah Research for LIS Practitioners, February 2011

Overview Part 1. Evaluation (with a touch of mystery) Part 2. Information literacy evaluation: Bachelor of Nursing Self Discovery Learning Part 3. Discussion

Part 1: Evaluation with a touch of mystery Presented by Suzana Sukovic

Evaluation Assessment Librarian The University of Tennessee Libraries 6 January 2011 The University of Tennessee, Knoxville (UTK) Libraries invites applications for the newly created position of Assessment Librarian. The Assessment Librarian fosters a culture of assessment within the University Libraries and leads the design and implementation of a successful, sustainable assessment program. Engaging staff throughout the Libraries, the Assessment Librarian analyzes systems and services for continuous improvement.

Evaluation Critical part of evidence based practice and action research Cycles of action research (State of NSW, Department of Education and Training, Professional Learning and Leadership Development Directorate, 2010, Action research in education: guidelines, 2 nd ed, p. 4)

Evaluation: approach Where to start? Strategic direction Particular purpose (e.g., solving a problem, grant application, part of a larger project) Opportunity (e.g., additional resource, a particularly suitable/innovative project)

Evaluation: approach Purpose Why evaluate? Evaluation focus Which aspect to evaluate? Evidence What is the evidence? Data- gathering methods How to obtain evidence? Analysis & interpretation How to interpret evidence? Service improvement Demonstrating value etc. Development of students’ IL skills Cost benefit Improvement of clients’ satisfaction etc. Clients’ ability to describe A do B apply A&B in new situations etc. Tests Experiments Observation Interviews with teachers & students etc. Consider Criteria Benchmarks Interpretation framework & methods Software Example: Evaluation of an information literacy program

Evaluation: approach Brainstorm Focus on – the best fit for the purpose – most effective & efficient approach – ‘maximizing potential’ ‘Cutting salami’ approach

Maximising evaluation projects Make a snow ball effect! snowball.jpg

Maximising evaluation projects Choose a general theme – relates to strategic direction, a large or long-lasting project, a significant problem or opportunity Keep research on mind LINK – several small projects – with projects in other departments/organisations – with your skills, knowledge and experiences – with literature There is nothing so practical as a good theory (Kurt Lewin, 1951) Publish & present – to your department, at conferences, in professional and academic journals

Wire Guys Chillin at the Baltic, New Castle UK / Marc Hartley b8053c53d/

Part 2: Information literacy evaluation Bachelor of Nursing Self Discovery Learning Presented by Jane Van Balen

Background Directive/Need for library to better evaluate Information Literacy programs generally Newly revised Bachelor of Nursing course - a worthy pilot because: – New course – fresh opportunity – Follow-up possible over next years of course – InfoLit training embedded by faculty – Self Discovery approach suggested by faculty

Background New course has a whole day allocated for Library information literacy as part of the subject “Workshops in Practice Readiness”. The Faculty suggested a self discovery approach for one of the components of the training which the library was already developing ideas for. Several factors came together fortuitously which allowed us to maximise this opportunity. We were allowed freedom to incorporate elements to better enable us to use for research. Eg Pre-Test addition.

Project objectives/directives To determine whether Information Literacy programs offered by the library are effective in enabling students to become independently information literate. To determine whether they continue developing over time after having received information literacy training.

Library Program – First year BNurs Online Pre-test : students could do this before they arrived or in a classroom at the start of the day. Not assessed by their tutor, but formed some of our data. Self discovery questionnaire : students grouped in pairs could answer a range of questions about library services. Token prizes were offered to the first pair who finished, and informal feedback was given to many as they returned their completed answers. Databases session : presented by their tutors using library’s lesson plan and handouts. Tutors attended a “Train the trainer” session with library staff. Online Post-test : This was compiled by the subject co- ordinator and was assessed. Library had access to overall results.

Data collected to date Results from the pre-test. Available online. Answers from the Self Discovery questionnaire. Written form only. Results from the post-test. Available online

What does the data tell us? If the overall scores from the post test were higher than the pre test, can we conclude that learning took place? Until we follow up with the next phase of training, it will be difficult to determine our original objective.

Issues – how to deal with Inconsistencies: – Various quality of db session presenters – Timetabling – Not everyone did the pre-test first – Self discovery questionnaire feedback varied considerably. Compiling data and interpretation

Next steps We will meet the same groups of students when they do an Evidence Based Practise subject in 2011 Associated with more in-depth Databases skills session. (PICO) Another opportunity to assess information searching skills and determine lasting benefits of training.

Next steps We have a number of elements we can use to proceed: – Another online pre-test (stimulus for learning) – online tutorials on Evidence Based Practice approach to finding specific types of articles. – Library workshop with specialist subject Librarians – Another online post-test AND/OR obtain access to the results of their related assessment task.

Gift from Two birds... Diigo group Research for practitioners Explore Add Comment Do we have your ?