Rubric Assessment of Student Responses to an Information Literacy Tutorial Megan Oakleaf Librarian for Instruction & Undergraduate Research Steve McCann NCSU Libraries Fellow

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 Presentation Objectives Share our experiences. –In changing the way we assess our program –In adapting ACRL outcomes to our project –In selecting a learning artifact to assess –In piloting our assessment plan Facilitate your experience with this type of assessment. Provide “take away” ideas.

NCSU Libraries Instruction Program

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 Information Literacy Information literacy is a set of abilities requiring individuals to "recognize when information is needed and have the ability to locate, evaluate, and use effectively the needed information." Association of College and Research Libraries /informationliteracycompetency.htm

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 How have we measured our success?

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 The Decision to Change Goal: To assess student learning of information literacy skills using outcomes-based assessment. Need 2 things… An Artifact to Assess Outcomes to Measure

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 What artifact will we assess? Interactions with students that could yield assessment artifacts… 50-minute one-shot workshops Library Online Basic Orientation (LOBO)

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 Disadvantages of Using Workshops for Assessment Perceived lack of time for open-ended responses, only m.choice and T/F are options. Taught by numerous librarians who lack assessment knowledge. Inconsistent audiences & content. Incomplete spectrum of outcomes addressed.

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 Advantages of Using LOBO for Assessment Forms basis for IL instruction at NCSU. Reaches virtually all incoming freshmen. Recently redesigned, includes open- ended questions. Captures student responses in a searchable database. Potential for rich data.

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 What outcomes will we assess? Information Literacy Competency Standards for Higher Education 5 Standards 22 Performance Indicators 87 Outcomes

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 What outcomes will we assess? Objectives for Information Literacy Instruction: A Model Statement for Academic Librarians Standards = 5 Perform. Indicators = 14 Outcomes = 35 “Bullets” = 133

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 What outcomes will we assess? LOBO Objectives & Outcomes Objectives = 5 Measurable Outcomes= 45

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 First Steps Set up DB to access student answers. Match outcomes to questions. How will we know the outcome’s been met? Beginning, Developing, Exemplary Pilot test a section of LOBO.

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 The Pilot Test Problem: Students use web sources for academic purposes without evaluating their quality. Are they duped by low-quality sites? Can we teach them to be more critical consumers of information?

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 Evaluating Web Sites What criteria are you looking for? What clues can you find? What specific example can you give from the web site at hand? Is the web site a good one for you to use?

Evaluating Web Sites

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 Pilot Test Results What we found out from 50 randomly selected student accounts …

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 Lobo Data—Jan-April, ‘04 1,830 total accounts new accounts 46% of total questions answered Evaluated 50 accounts

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 Objective 5: Evaluating Resources #3 “Evaluate Web Sites” questions: 1.Locate a website 2.Evaluate website’s authority 3.Evaluate recency/currency 4.Identify Bias/Point of View

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 Q1: Locate a Website 4 points possible: Average score 2.7 Question Text: “Type the title and URL (web address) of the web site you will evaluate here:”

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 Q2: Evaluate a Website’s Authority 8 points possible: Average score 5.0 Question Text: “Answer the questions above for the web site you're evaluating. Overall, does what you know about the URL of the web site indicate that it's a good resource? ” “Answer the questions above for the web site you're evaluating. Overall, does what you know about the authorship of the web site indicate that it's a good resource? ”

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 Q3: Evaluate a Website’s Currency 8 points possible: Average score 5.6 Question Text: “Answer the questions above for the web site you're evaluating. Overall, does what you know about the currency of the web site indicate that it's a good resource? ”

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 Q4: Identify a Website’s Bias 8 points possible: Average score 4.1 Question Text: “Answer the questions above for the web site you're evaluating. Overall, does what you know about the bias of the web site indicate that it's a good resource? Overall, is this web site a good resource to use for your assignment?”

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 Pilot Test Findings Students proven successful with mechanical tasks like checking currency and identifying URLs. Students are shown as “developing” with judgment tasks such as authority and bias. Bias is a potential target area.

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 How are we using our assessment results? Changes to LOBO Add and Reorganize Content Improve Question Format Enlarge Response Space Make Rubrics Available? Train Course Instructors

Changes to LOBO Answer these questions about the web site you’re evaluating in the space below: Who created the site? What point of view do they represent? What organizations support the site? What biases might they have? Are links included that point to other viewpoints? Are there signs of bias included in the site? Are you biased toward the site? Overall, does what you know about the bias of the web site indicate that it’s a good resource? Replace Questions with Content Sample Student Answers Get Help with Your Answer Fix Questions, Add Links, Enlarge Answer Space Add Viewlet to Model Application of Content Changes Coming Soon!

Changes for Instructors Share Rubrics, Continue Ongoing Training, Add Lesson Plans

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 How are we using our assessment results? Changes to the Instruction Program Including assessment in departmental 3-yr goals Sharing data with “subject-specialist” librarians Initiating rubric assessment of advanced instruction Reporting results to library administration and colleagues on other campuses

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 Key Learning Rubrics require lots of revision! Rubrics are effective in measuring higher-level thinking skills. Rubrics provide information administrators can use for reporting and instructors can use to improve teaching and learning. Our colleagues are interested in our progress.

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 Take Aways 1.Can higher-level thinking skills like information literacy or critical thinking be adequately described in a rubric? 2.Are rubrics good tools for assessing student responses to tutorials? 3.What problems did you find in your practice with these rubrics? 4.What part of this process/project could be applied at your institution?

M. Oakleaf and S. McCann, Undergraduate Assessment Symposium 2004 Muddiest Points?

Contact Information Megan Oakleaf Librarian for Instruction & Undergraduate Research Steve McCann NCSU Libraries Fellow

Rubric Assessment of Student Responses to an Information Literacy Tutorial Megan Oakleaf Librarian for Instruction & Undergraduate Research Steve McCann NCSU Libraries Fellow