1. School of Information Studies
Using Rubrics to Collect Evidence for Decision-Making: What do Librarians Need to Learn?
Megan Oakleaf, MLS, PhD
School of Information Studies
Syracuse University
4th International Evidence Based Library & Information Practice Conference
May 2007

2. Overview Introduction Definition & Benefits of Rubrics Methodology
Emergence of Expert Rubric User Group
Characteristics of Expert Rubric Users
Barriers to Expert Use of Rubrics
The Need for Training
Directions for Future Research

3. Rubrics Defined
describe the 1) parts, indicators, or criteria and 2) levels of performance of a particular task, product, or service
formatted on a grid or table
employed to judge quality
used to translate difficult, unwieldy data into a form that can be used for decision-making

4. Rubrics are often used to make instructional decisions and evaluations.

5. Potential Rubric Uses in Libraries
To analyze and evaluate:
Information-seeking behavior
Employee customer service skills
Marketing/outreach efforts
Collection strengths
Information commons spaces
Student information literacy skills

6. Rubric for a Library Open House Event for First Year Students
Indicators
Beginning
Developing
Exemplary
Data Source
Attendance
Attendance rates are similar to the 2006 Open House
Attendance rates increase by 20% from 2006 Open House
Attendance rates will increase by 50% from 2006 Open House
Staff [Committee and Volunteers] records
Staff Participation
Staff participation is similar to 2006 Open House, no volunteers
Increase in participation by library staff [librarians and paraprofessionals] and student volunteers
Increase in participation with library staff [librarians and paraprofessionals], student volunteers, student workers, and academic faculty
Budget
Budget same as 2006 Open House, $200
Budget increases by $100 from 2006 Open House
Budget increases by $300 from 2006 Open House
Budget, Financial Statements
Reference Statistics
Reference statistics similar to 2006
Reference statistics increase by 20% from 2006
Reference statistics increase by 50% from 2006
Library Reference Department Statistics
Student Attitudes
Students are pleased with Open House
Students enjoy the Open House, are satisfied with information
Students are excited about the Open House, volunteer to participate with the next year’s event
Survey
Rubric for a Library Open House Event for First Year Students
Rubric created by:
Katherine Thurston & Jennifer Bibbens

7. Post-Training Surveys
Rubric for a Virtual Reference Service
Indicators
Beginning
Developing
Exemplary
Data Source
Transactions
0 – 4 reference transactions per week.
5 – 7 reference transactions per week.
8 + reference transactions per week.
Transaction Logs
User Satisfaction
Students, faculty and staff report they are “dissatisfied” or “very dissatisfied” with reference transactions.
Students, faculty and staff report they are “neutral” about reference transactions.
Students, faculty and staff report they are “satisfied” or “very satisfied” with reference transactions.
User Surveys
Training
Librarians report they are “uncomfortable” or “very uncomfortable” with providing virtual reference service.
Librarians report they are “neutral” about providing virtual reference service.
Librarians report they are “comfortable” or “very comfortable” with providing virtual reference service.
Post-Training Surveys
Technology
Between 75 % and 100 % of transactions a week report dropped calls or technical difficulties.
Between 25 % and 74% of transactions a week report dropped calls or technical difficulties.
Between 0 % and 24% of transactions a week report dropped calls or technical difficulties.
System Transcripts
Electronic Resources
0 – 50 hits on electronic resources a week.
50 – 100 hits on electronic resources a week.
100 + hits on electronic resources a week.
Systems Analysis Logs
Rubric created by:
Ana Guimaraes & Katie Hayduke

8. Study Rubric

9. Benefits
rubrics provide librarians the opportunity to discuss, determine, and communicate agreed upon values
rubrics include descriptive, yet easily digestible data
prevent inaccuracy of scoring
prevent bias
When used in student learning contexts…
reveal the expectations of instructors and librarians to students
offer more meaningful feedback than letter or numerical scores alone
support not only student learning, but also self-evaluation and metacognition

10. The Research Question
To what extent can librarians use rubrics to make valid and reliable decisions?
Library service: an information literacy tutorial
Artifacts: student responses to questions within the tutorial
Goal: to make decisions about the tutorial and the library instruction program

11. Methodology
75 randomly selected student responses to open-ended questions embedded in an information literacy tutorial at NCSU
25 raters
15 internal & trained (NCSU librarians, faculty, students)
10 external & untrained (non-NCSU librarians)
raters code artifacts using rubrics
raters’ experiences captured on comment sheets
reliability statistically analyzed using Cohen’s kappa
validity statistically analyzed using a “gold standard” approach and Cohen’s kappa
This study employed a survey design methodology. The data for the study came from student responses to open-ended questions embedded in an online information literacy tutorial. This textual data was translated into quantitative terms through the use of a rubric. Using a rubric, raters coded student answers into pre-set categories, and these categories were assigned point values. The point values assigned to student responses were subjected to quantitative analysis in order to describe student performance, test for interrater reliability, and explore the validity of the rubric. According to Lincoln, this approach is called “discovery phase” or preliminary experimental design, and it is commonly employed in the development of new rubrics.
[1] Yvonna Lincoln. "Authentic Assessment and Research Methodology." to Megan Oakleaf

13. Kappa Index Kappa Statistic Strength of Agreement 0.81-1.00
Almost Perfect
Substantial
Moderate
Fair
Slight
<0.00
Poor

14. Average Kappa
Rank
Participant Group
Status
0.72 1
NCSU Librarian
Expert
0.69 2
Instructor
0.67 3
0.66 4
0.62 5
0.61 6
Non-Expert
0.59 7
0.58 8
Student
0.56 9
0.55 10
.055 11
0.54 12
0.52 13 14
0.43 15
External Instruction Librarian
0.32 16
External Reference Librarian
0.31 17 18
0.30 19 20
0.27 21
0.21 22
0.19 23
0.14 24
0.13 25
expert status does not appear to be correlated to educational background, experience, or position within the institution

15. Expert Kappa Statistics

16. Non-Expert Kappa Statistics

17. Expert Characteristics
focus on general features of artifact
adopt values of rubrics
revisit criteria while scoring
experience training

18. Non-Expert Characteristics
diverse outlooks or perspectives
prior knowledge or experiences
fatigue
mood
other barriers

19. Barrier 1 Difficulty Understanding an Outcomes-Based Approach
Many librarians are more familiar with inputs/outputs than outcomes.
Comments from raters:
using measurable outcomes to assess student learning focuses too much on specific skills—too much “science” and not enough “art.”
“While the rubric measures the presence of concepts…it doesn’t check to see if students understand [the] issues.”
“This rubric tests skills, not…real learning.”

20. Barrier 2 Tension between Analytic & Holistic Approaches
Some librarians are unfamiliar with analytical evaluation.
Comments from raters:
The rubric “was really simple. But I worried that I was being too simplistic…and not rating [student work] holistically.”
“The rubric is a good and a solid way to measure knowledge of a process but it does not allow for raters to assess the response as a whole.”

21. Analytic vs. Holistic Analytic Better for judging complex artifacts
Allow for separate evaluations of artifacts with multiple facets
Provide more detailed feedback
Take more time to create and use
Bottom line: Better for providing formative feedback
Holistic
Better for simple artifacts with few facets
Good for getting a “snapshot” of quality
Provide only limited feedback
Do not offer detailed analysis of strengths/weaknesses
Bottom line: Better for giving summative scores

22. Barrier 3 Failure to Comprehend Rubric
Some librarians may not understand all aspects of a rubric.
Comments from raters:
“I decided to use literally examples, indicators to mean that students needed to provide more than one.”
“The student might cite one example…but not…enough for me to consider it exemplary.”

23. Barrier 4 Disagreement with Assumptions of the Rubric
Some librarians may not agree with all assumptions and values espoused by a rubric.
Comments from raters:
The rubric “valued students’ ability to use particular words but does not measure their understanding of concepts.”

25. Barrier 5 Difficulties with Artifacts
Some librarians may be stymied by atypical artifacts.
Comments from raters:
I found myself “giving the more cryptic answers the benefit of the doubt.”
“If a student answer consists of a bulleted list of responses to the prompt, but no discussion or elaboration, does that fulfill the requirement?”
“It’s really hard…when students are asked to describe, explain, draw conclusions, etc. and some answer with one word.”

26. Barrier 6 Difficulties Understanding Library Context & Culture
Librarians need campus context to use rubrics well.

27. Training
Topics
Value & principles of outcomes-based analysis and evaluation
Theories that underlie rubrics
Advantages & disadvantages of rubric models
Structural issues that limit rubric reliability and validity (too general or specific, too long, focused on quantity not quality, etc)
Ways to eliminate disagreement about rubric assumptions
Methods for handling atypical artifacts

28. Future Research Investigate: attributes of expert raters
effects of different types and levels of rater training
non-instruction library artifacts
impact of diverse settings

29. Conclusion Are rubrics worth the time and energy?
This study confirmed the value of rubrics—nearly all participants stated that they could envision using rubrics to improve library instructional services.
Such feedback attests to the merit of rubrics as tools for effective evidence based decision-making practice.

30. American Library Association. 2000
American Library Association Information Literacy Competency Standards for Higher Education. 22 April 2005 <
Arter, Judith and Jay McTighe. Scoring Rubrics in the Classroom: Using Performance Criteria for Assessing and Improving Student Performance. Thousand Oaks, California: Corwin Press, 2000.
Bernier, Rosemarie. “Making Yourself Indispensible By Helping Teachers Create Rubrics.” CSLA Journal 27.2 (2004).
Bresciani, Marilee J., Carrie L. Zelna, and James A. Anderson. Assessing Student Learning and Development: A Handbook for Practitioners. Washington: National Association of Student Personnel Administrators, 2004.
Callison, Daniel. “Rubrics.” School Library Media Activities Monthly 17.2 (Oct 2000): 34.
Colton, Dean A., Xiaohong Gao, Deborah J. Harris, Michael J. Kolen, Dara Martinovich-Barhite, Tianyou Wang, and Catherine J. Welch. Reliability Issues with Performance Assessments: A Collection of Papers. ACT Research Report Series 97-3, 1997.
Gwet, Kilem. Handbook of Inter-Rater Reliability: How to Estimate the Level of Agreement between Two or Multiple Raters. Gaithersburg, Maryland: STATAXIS, 2001.
Hafner, John C. “Quantitative Analysis of the Rubric as an Assessment Tool: An Empirical Study of Student Peer-Group Rating.” International Journal of Science Education (2003).
Iannuzzi, Patricia. “We Are Teaching, But Are They Learning: Accountability, Productivity, and Assessment.” Journal of Academic Librarianship 25.4 (1999):
Landis, J. Richard and Gary G. Koch. “The Measure of Observer Agreement for Categorical Data.” Biometrics 33 (1977).
Lichtenstein, Art A. “Informed Instruction: Learning Theory and Information Literacy.” Journal of Educational Media and Library Sciences 38.1 (2000).
Mertler, Craig A. “Designing Scoring Rubrics For Your Classroom.” Practical Assessment, Research and Evaluation 7.25 (2001).
Moskal, Barbara M. “Scoring Rubrics: What, When, and How?” Practical Assessment, Research, and Evaluation 7.3 (2000).
Nitko, Anthony J. Educational Assessment of Students. Englewood Cliffs, New Jersey: Prentice Hall, 1996.
Popham, W. James. Test Better, Teach Better: The Instructional Role of Assessment. Alexandria, Virginia: Association for Supervision and Curriculum Development, 2003.
Prus, Joseph and Reid Johnson. “A Critical Review of Student Assessment Options.” New Directions for Community Colleges 88 (1994).
Smith, Kenneth R. New Roles and Responsibilities for the University Library: Advancing Student Learning through Outcomes Assessment. Association of Research Libraries, 2000.
Stevens, Dannielle D. and Antonia Levi. Introduction to Rubrics: An Assessment Tool to Save Grading Time, Convey Effective Feedback, and Promote Student Learning. Sterling, Virginia: Stylus, 2005.
Tierney, Robin and Marielle Simon. “What's Still Wrong With Rubrics: Focusing On the Consistency of Performance Criteria Across Scale Levels.” Practical Assessment, Research, and Evaluation 9.2 (2004).
Wiggins, Grant. “Creating Tests Worth Taking.” A Handbook for Student Performance in an Era of Restructuring. Eds. R. E. Blum and Judith Arter. Alexandria, Virginia: Association for Supervision and Curriculum Development 1996.
Wolfe, Edward W., Chi-Wen Kao, and Michael Ranney. “Cognitive Differences In Proficient and Nonproficient Essay Scorers.” Written Communication 15.4 (1998).

31. Questions?