Assessing Library Instruction for Military Students: A Study for Assessment in Action Initiative Elizabeth Torres, Librarian (Team Leader); Timothy Martin, Head of Reference & Instruction Services; Farris Sanders, Librarian; Nori Leong, Director of Libraries & Learning Commons; Sheryl Sunia, Instructor of Justice Administration & Curriculum Area Liaison for Military Campus Programs; Han Nee Chong Wester, Instructional Design & Technology Specialist

Poster presentation - June 2016 ALA Orlando This project is part of the program “Assessment in Action: Academic Libraries and Student Success” which is undertaken by the Association of College and Research Libraries (ACRL) in partnership with the Association for Institutional Research and the Association of Public and Land-grant Universities. The program, a cornerstone of ACRL's Value of Academic Libraries initiative, is made possible by the Institute of Museum and Library Services. HPU Libraries was part of the third year cohort along with 54 other Academic Libraries Participating institutions were divided into smaller cohorts of 9-10 institutions The project included 14 months of webinars, collaborations, homework, and conducting the study 4 days of in-person workshops - June 2015 ALA San Francisco, January 2016 ALA Boston Poster presentation - June 2016 ALA Orlando

Assess What You Value Not Value What You Assess

Our Focus Information Literacy, one of HPU’s ILOs, is the primary contribution of Reference & Instruction towards student success. Military Campus Programs students are a unique and vital component of our student body. Hawaii Pacific University’s Military Campus Programs (MCP) is offered through the College of Extended and Interdisciplinary Education. HPU was once again named a “Best for Vets” college and “Military Friendly School” for 2016.

Measurable changes that benefited students Our Outcome MCP students participating in library sessions regardless of method of delivery (face-to-face or hybrid) will demonstrate the ability to formulate complex search strategies and select appropriate sources to defend their position in their final essay.  WRI 1200 ILO Bloom’s Taxonomy Impact of experience Aligned with institutional values- we used specific language consistent with language used in WRI ILOs Used active verbs that demonstrated the desired level of student achievement Measurable changes that benefited students

Pre-study We applied for an expedited level of review within our Institutional Review Board - IRB Created a consent form Protected the identity of students by creating codes Created a debriefing form Completed FERPA training Created tools to use for collecting data Pre- & post test In-class search assignment with feedback Rubric for assessing students final papers Collaborated with AiA campus partners Determined which MCP courses would participate Determined how to use Blackboard & Google Forms

Our Study Research Question: Does the method of delivery (face-to-face vs. hybrid) impact the effectiveness of library instruction for off campus military students?   Study Population: Four WRI 1200 courses (two face-to-face, two hybrid), two MCP faculty, 28 students.  

Our Assessment Our study compared two library interventions: Traditional MCP face-to-face 50 minute library instruction session at military bases. Hybrid intervention that included 3-5 minute library tutorials in addition to a 50 minute library session at military bases. Our assessment tools: Pre- & post tests: 10 multiple Qs Search assignment: 16 open-ended Qs Final argumentative paper: Evaluated by 2 librarians with in-house rubric We assessed 76 artifacts from 24 students, 11 from face-to-face and 13 from hybrid. Unexpected Challenge: Students were not signing consent forms prior to instruction session. We had to ask for consent verbally, we signed up students during class and provided codes which impacted the amount of time we had to deliver instruction.  

Our Tools Pre- & Post Test Short questionnaire (ten questions) – due to time constraints with students. Questions ranged from selecting proper resources for searching, to confidence in doing library research.

Our Tools Library Assignment 16 open ended questions Questions prompted students to search specific databases, employ advanced search strategies, cite a relevant source and provide feedback on our instruction.

Our Tools Rubric Two librarians reviewed the final papers We focused on selection of sources used in paper (variety and quality). How were the sources used to support arguments? Were students providing proper credit for their sources?

Results Overall post-test scores improved for both intervention modes. Hybrid students had a higher pre-test score overall. We attribute this to the fact that 50% of students in the hybrid model took the pre-test after viewing the library tutorials. Unexpected Challenge: Rubric scores were higher for face-to-face. We attribute this to the unanticipated difference between the two instructors’ argumentative essay requirements.

Results Most students found appropriate sources in the search activity, yet still had problems implementing some advanced search techniques such as truncation. Hybrid students were more likely to express being confident in their research skills than face-to-face students.

Conclusions Overall, there was an improvement in the students' search skills and their ability to find and use appropriate sources regardless of the intervention type. There was not enough statistical difference to prove one model was better than the other. Due to class sizes, our sample was small. We hope more MCP writing courses will be part of our future assessment efforts, including courses that are online only. We acknowledge the uniqueness of MCP students and learned through the project that some elements of our instruction should be modified: Spending additional time highlighting the benefits of the eBook collection for distance learners Providing citation guidelines for resources Emphasizing complex search strategies such as truncation Adding more tutorials and expanding embedded services as teaching assistants

Our AiA Experience Used a variety of assessment tools Flexibility - expect the unexpected Sharing our findings within our community to cement future collaborations Leadership - acting as facilitators within our projects/community/institution Strong outcomes go a long way Pressure on presenting at ALA - use not only as a deadline but as motivation Assess what we value…

Value: What is of value to your institution/community, what is your library's added value to it? Outcomes: How do you want patrons to be changed as a result of the library’s activities/actions? Criteria: How will you know you have succeeded in proving value? What will be happening? Actions: What will you do to make this happen? Evidence: How will you collect results? What will you observe, judge or measure?

Thank You Any Questions?