The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger.

Slides:



Advertisements
Similar presentations
Information Literacy Instruction in Libraries
Advertisements

The Commissions Expectations for the Assessment of Student Learning and Institutional Effectiveness Beth Paul Interim Provost and Vice President for Academic.
Pathways for Assessing Scientific Literacy, Fluency in Technology, and Information Literacy Kathleen Warner, PhD Janet Zimmer, PhD Elizabeth Mulherrin,
Program Goals Just Arent Enough: Strategies for Putting Learning Outcomes into Words Dr. Jill L. Lane Research Associate/Program Manager Schreyer Institute.
March 2007 ULS Information Literacy and Assessment of Learning Program.
Measuring Up on College- Level Learning Margaret Miller, Project Director September 2003.
Comparison of Teacher-Centered and Learner-Centered Paradigms From Figure 1-2 in Huba and Freed, Learner-Centered Assessment on College Campuses: Shifting.
Bridging the Sophomore Gap: A Developmental Model of Information Literacy Shawn Bethke, Head of Library Public Services George Loveland, Library Director.
Information Literacy Defined A set of abilities that requires individuals: recognize what information is needed have the ability to locate, evaluate,
National Academic Reference Standards
Assessment of General Education Patricia E. Beeson Vice Provost for Graduate and Undergraduate Studies February 11, 2008 University of Pittsburgh.
Enhancing Critical Thinking Skills 2012 HBCU Library Alliance Leadership Institute Presented By: Violene Williams, MLIS Reference Librarian James Herbert.
Measuring Information Literacy from Third Grader to College Senior Carolyn Radcliff, M.L.S., SAILS Project Director Julie Gedeon, Ph.D., TRAILS Assessment.
Information Literacy Assessment: A Contextual Overview Keith Gresham Head of Information & Instruction Services University of Vermont Libraries VLA College.
BEST PRACTICES IN INFORMATION LITERACY ASSESSMENT Yvonne Mery, Vicki Mills, Jill Newby, University of Arizona Libraries February 11, 2009.
Core Competencies Student Focus Group, Nov. 20, 2008.
Information Literacy Demonstration – Partnership of Faculty and Library Gergana Georgieva Information Literacy Librarian August 25, 2008.
Alcorn State University Information L I N K J. D. Boyd Library.
One Library’s Approach Presented by Bruce Kocour Carson-Newman College
Educational Outcomes: The Role of Competencies and The Importance of Assessment.
From Standards to the Framework: Using Threshold Concepts to Rethink Instruction Ericka Raber, University of Iowa Libraries Leslie Ross, St. Ambrose University.
WASHBURNWASHBURN Friends of Mabee Library October 28, 2004 Standardized Assessment of Information Literacy Skills Presented by Judy Druse Martha Imparato.
Information Literacy: Lingo or Opportunity for Learning? Jim Stenerson and Sarah Burns Feyl May Faculty Institute, 2004 Pace University.
Developing An Online Information Literacy Course Nancy O’Hanlon Ohio State University Libraries Wuhan University, China March 2007.
USING STUDENT OUTCOMES WHEN INTEGRATING INFORMATION LITERACY SKILLS INTO COURSES Information Literacy Department Asa H. Gordon Library Savannah State University.
Joanne Muellenbach, MLS, AHIP The Commonwealth Medical College Scranton, Pennsylvania June 26, 2012.
Information literacy and Plagiarism: responsibilities responsibilities Annemarie Maritz.
Library Instruction in North America Library Orientation (before 1980) –Tour of library, instruction in using card catalog, print indexes, reference works.
Information Literacy Tutorial for the First Time in College Student St. Philip’s College LRC San Antonio, Texas Dr. Adele S. Dendy, Dean of Learning Resources.
Accreditation: An Opportunity to Promote Information Technology Fluency NLII Annual Meeting San Diego, CA January 26, 2004 Joan K. Lippincott CNI.
INFORMATION LITERACY AND COMPETENCY STANDARDS FOR HIGHER EDUCATION Seminar Presentation Presented by :- Sarita Rani Assistant Librarian Bhai Gurdas Library.
Information Literacy Standards Performing Arts Point Park University Library.
Accrediting Commission for Community and Junior Colleges of the Western Association of Schools and Colleges.
Transitions: Developing Information Literacy Skills Among STEM Students 2011 UK STEM Symposium Roundtable Sue Smith, UK Engineering Library Head Valerie.
Nina Koniuch RN,BScN, MScT Karen Halliday BA (Hon), MLIS
Planning and Assessing Instruction Sessions Using Learning Outcomes Texas Library Association Annual Conference Houston, TX – March 31, 2009.
Librarians Prepare for their Global Information Role in the 21 st Century Hannelore B. Rader University of Louisville Louisville, Kentucky, US January.
Pierce College CSUN-Pierce Paths Project Outcomes Report 2013.
Assessing Credit –bearing Information Literacy Class using ACRL Information Literacy Competency Standards Wenli Gao Instructional Services Librarian Morrisville.
1 Information Fluency: Literacy in the Digital Age Lisa Janicke Hinchliffe
Information Literacy and the University Curriculum A Workshop Sponsored by CETLA and the University Howard University June 2005.
Evidence of Student Learning Fall Faculty Seminar Office of Institutional Research and Assessment August 15, 2012.
The Role of Libraries Special Committee Robert Morris University
The Academy Experience: A Plan for Integrated Education.
Models and Standards Week 3.
Information Literacy in Academic Environments June 1, 2005 AMICAL Meeting No. 2 June 1-3, 2005 Delivered by Samira Rafidi Meghdessian Information Services.
Assignment Design I Katy Sullivan, Reference & Instruction Librarian Albin O. Kuhn Library & Gallery Spring 2004.
AASCB The Assurance of Learning AASCB Association to Advance Collegiate Schools of Business Marta Colón de Toro, SPHR Assessment Coordinator College of.
Information Literacy Workshop Association of Architecture School Librarians Houston, TX – April 27, 2008.
Bonnie Paller 2013 AALC Assessment Retreat.  The charge of the Task Force is to identify the abilities and intellectual traits that all students are.
Tackling the Complexities of Source Evaluation: Active Learning Exercises That Foster Students’ Critical Thinking Juliet Rumble & Toni Carter Auburn University.
Department of Information & Communications Online Information Literacy Audits: a longitudinal study Bob Glass & Jill Griffiths.
ACRL’s Information Literacy Competency Standards for Higher Education Why, Who, What? A Brief Overview Celita DeArmond, UTSA Library
Information Literacy Module for FYI Available to any FYI Tony Penny, Research Librarian – Goddard Library Research & Library Instruction Services We support.
Information Literacy Module for Majors Available to support any department Tony Penny, Research Librarian – Goddard Library Supporting the Architecture.
Knowledge building in the 21 st century at The Geelong College: Information-to-Knowledge Continuum “As we increasingly move toward an environment of instant.
Assessing Information Literacy with SAILS Juliet Rumble Reference & Instruction Librarian Auburn University.
Information Competency Also known as Information Literacy.
Research Assignment Design Kerri Carter – ext Diane VanderPol
Kenneth C. C. Yang The University of Texas at El Paso Presented at 2016 Sun Conference TEACHING INFORMATION LITERACY SKILLS IN COLLEGE CLASSROOMS: EMPIRICAL.
BAY AREA COMMUNITY COLLEGES INFORMATION COMPETENCY ASSESSMENT PROJECT (ICAP) BI Subcommittee Session MLA Memphis February 23, 2006 Donna Arnold.
An Introduction to Information Literacy for Secondary Education
Phyllis Lynch, PhD Director, Instruction, Assessment and Curriculum
Information Literacy Standards for Freshmen Seminars
NJCU College of Education
Assessment and Accreditation
Mapping the ACRL Framework and Nursing Professional
EDUCAUSE MARC 2004 E-Portfolios: Two Approaches for Transforming Curriculum & Promoting Student Learning Glenn Johnson Instructional Designer Penn State.
Student Learning Outcomes at CSUDH
Information Literacy: What is it and Why Should I Care?
Presentation transcript:

The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota The South Dakota Regental Information Literacy Exam : A Tool to Document and Assess Information Literacy Carol A. Leibiger, Ph.D. Head of Public Services and Reference, Information Literacy Coordinator, I.D. Weeks Library, University of South Dakota William Schweinle, Ph.D. Assistant Professor of Psychology Boise State University Irene Rockman (2005) there are homegrown, paper-and-pencil exams that have not been tested for validity, reliability, and psychometrics (587).  describing the development of the South Dakota Regental Information Literacy Exam, a “homegrown” IL exam that is valid, reliable and has unique and very useful psychometric properties. ACRL National Conference, Baltimore, March 31, 2007

Information Literacy and Life- long Learning The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Information Literacy and Life- long Learning An information literate person is able to: determine the extent of the information needed access the needed information effectively and efficiently evaluate information and its sources critically incorporate selected information into one’s knowledge base use information effectively to accomplish a specific purpose understand the economic, legal, and social issues surrounding the use of information, and access and use information ethically and legally American Library Association/Association of College and Research Libraries, Information Literacy Competency Standards for Higher Education (2000) ACRL National Conference, Baltimore, March 31, 2007

Rise of IL Instruction and Assessment The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Rise of IL Instruction and Assessment Higher education assessment movement Rise of strategic planning and Total Quality Management (TQM) in higher education Library instruction movement: change in focus from library skills to IL in academic libraries General education reform movement Inclusion of IL in accreditation standards Meulemans, Y. (2002); Rockman, I. (2002) IL assessment arose out of several movements in academia and academic libraries in the late 1980s and 1990s: Result = consensus on importance of IL and the need for its assessment: essential basic skill for student success in academic and personal matters lifelong learning skill Necessary to produce information literate citizens (liberal art) has joined the set of skills that are assessed by institutions of higher education ACRL National Conference, Baltimore, March 31, 2007

Other Documents Supporting IL The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Other Documents Supporting IL Reform on Campus (1972, and follow-up reports of the Carnegie Commission on Higher Education) Information Literacy Competency Standards for Higher Education (1989) SCANS Report (Secretary’s Commission on Achieving Necessary Skills, Dept. of Labor, 1991) Goals 2000: National Educate America Act (1994) Information Power National School Library IL Standards (1998) Greater Expectations: A New Vision for Learning as a Nation Goes to College (2002, Association of American Colleges and Universities) All of these emphasize the importance of the library and research in undergraduate higher education quality assignments, greater emphasis on essential skills, and lifelong learning the ability to acquire, evaluate, use, maintain, interpret, and communicate information as a necessary future job skill adult literacy and lifelong learning as necessary to compete in a global economy necessary skill to exercise the rights and responsibilities of citizenship. ACRL National Conference, Baltimore, March 31, 2007

Higher Learning Commission (2003) The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Higher Learning Commission (2003) Criterion 4, “Acquisition, discovery, and application of knowledge” 4a: The organization demonstrates, through the actions of its board, administrators, students, faculty, and staff, that it values a life of learning. 4b: The organization demonstrates that acquisition of a breadth of knowledge and skills and the exercise of intellectual inquiry are integral to its educational programs. 4c: The organization assesses the usefulness of its curricula to students who will live and work in a global, diverse, and technological society. 4d: The organization provides support to ensure that faculty, students, and staff acquire, discover, evaluate, and apply knowledge responsibly. Having IL in accreditation standards = recognition of its important as a learning outcome guarantees administrative support will probably get librarians into the classroom faster than all the other good reasons to integrate IL into instruction ACRL National Conference, Baltimore, March 31, 2007

Graduate Student IL Assessment The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Graduate Student IL Assessment Graduate students… often overestimate their information-finding skills operate on the principle of least effort tend to choose the easiest and most convenient resources don’t know enough - yet - about their disciplines to be effective searchers don’t have the critical skills to handle the information explosion George, C., Bright, A., Hurlbert, T., Linke, E. C., St. Clair, G., & Stein, J. (2006); Bellard, E.M. (2005); Chu, S. K.-W., & Law, N. (2003), Grant, M. & Bert, M., (2003). Like undergraduates, grad students -tend to try to expend the least amount of energy in finding information -are not yet knowledgeable about their discipline -do not possess adequate critical thinking skills to handle information glut ACRL National Conference, Baltimore, March 31, 2007

Graduate Student IL Assessment The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Graduate Student IL Assessment Demographic changes in the graduate student population average graduate student is female over 35 has been away from education for at least 2 years linguistically, ethnically diverse work and family responsibilities time management issues feelings of inadequacy technology anxiety Bellard, E.M. (2005); Gordon, C. (2002) Graduate student demographics have changed -no longer white twenty-something male -female -ethnically diverse -work/family responsibilities -time-management issues -feelings of inadequacy vis a vis technology and study ACRL National Conference, Baltimore, March 31, 2007

Graduate Student IL Assessment The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Graduate Student IL Assessment Graduate students can benefit from IL instruction because they are also: highly- and self-motivated learners cognitively mature understand their own learning styles, apply meta-cognitive strategies to their information seeking familiar with higher-quality search engines like Google Scholar an important conduit of IL instruction for undergraduate students Graduate students can benefit from IL instruction -for their own studies -for their students (TAs in gen. ed. courses) ACRL National Conference, Baltimore, March 31, 2007

Graduate Student IL Assessment The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Graduate Student IL Assessment no systematic IL assessment of advanced learners isolated suggestions and tools research paper required of all applicants to graduate programs IL skills audit or test required of entering graduate students (University of Missouri-Columbia, Boston College, and Australian National University) ACRL National Conference, Baltimore, March 31, 2007

Graduate Student IL Assessment The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Graduate Student IL Assessment SDILE can serve as an entrance assessment of IL for advanced learners skill set assumed for graduate students is congruent with that of undergraduate students who are information literate short yet valid and reliable instrument documenting and assessing IL allows identification of deficiencies in IL and formulation of appropriate remediation Because SDILE is an undergrad exit assessment, it can function as a grad entrance assessment -same skill set -allows early remediation ACRL National Conference, Baltimore, March 31, 2007

South Dakota Regental Universities The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota South Dakota Regental Universities Black Hills State University Dakota State University Northern State University South Dakota State University South Dakota School of Mines and Technology University of South Dakota (USD) In the fall of 2000, the six universities of the South Dakota Regental System implemented an Information Technology Literacy general education requirement. ACRL National Conference, Baltimore, March 31, 2007

South Dakota BOR ITL Requirement The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota South Dakota BOR ITL Requirement Information Technology Literacy was defined institution-specifically. All universities except USD defined ITL as IT. Definition affected how it was taught and assessed. Only USD taught and assessed it as IL ACRL National Conference, Baltimore, March 31, 2007

South Dakota System General Education Requirements (2005) The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota South Dakota System General Education Requirements (2005) Goal #1: Student will write effectively and responsibly and will understand and interpret the written expression of others. Goal #2: Student will communicate effectively and responsibly through listening and speaking. Goal #3: Student will understand the organization, potential, and diversity of the human community through study of the social sciences. Goal #4: Students will understand the diversity and complexity of the human experience through study of the arts and humanities. Goal #5: Students will understand and apply fundamental mathematical processes and reasoning. Goal #6: Students will understand the fundamental principles of the natural sciences and apply scientific methods of inquiry to investigate the natural world. Goal #7: Students will recognize when information is needed and have the ability to locate, organize, critically evaluate, and effectively use information from a variety of sources with intellectual integrity. early 2004: the SD BOR convened the Strategic Leadership Team a group of administrators, instructors, and librarians from the six state universities mandate to revise the system-wide general education goals and objectives implement the new goals in the fall of 2005. Fall 2004: 7 general education goals, with objectives and lists of courses that fulfilled the goals. ACRL National Conference, Baltimore, March 31, 2007

IL Student Learning Outcomes The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota IL Student Learning Outcomes Students will… determine the extent of information needed; access the needed information effectively and efficiently; evaluate information and its sources critically; use information effectively to accomplish a specific purpose; use information in an ethical and legal manner. (ALA/ACRL IL Competency Standards) ACRL National Conference, Baltimore, March 31, 2007

Seeking an IL Assessment: Problems With the ITL Exam The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Seeking an IL Assessment: Problems With the ITL Exam Characteristics of the ITL Exam WebCT 20 questions Multiple-choice questions Passing score 13/20 (65% correct) Problems Function = documentation of IL No assessment value (KR20 ≈ .30) Privileged students who passed SPCM 101 at USD Why not just use USD’s ITL Exam? ACRL National Conference, Baltimore, March 31, 2007

Seeking an IL Assessment: National Standardized Exams The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Seeking an IL Assessment: National Standardized Exams Project SAILS ETS Information and Communication Technology (ICT) Literacy Assessment James Madison Information Literacy Test (ILT) assessment instrument should be chosen or developed for the entire regental system. Project SAILS Length Analysis only goes to the student cohort, not the student ETS ICT Literacy Exam assesses both information literacy and technology literacy JMILT Like SAILS, only assesses IL standards 1-3 and 5 In addition, since our students need to pass an IL assessment by the end of their sophomore year, and they are already burdened with the lengthy CAAP Proficiency Test, it was desirable to find an IL assessment that was shorter, yet valid and reliable. ACRL National Conference, Baltimore, March 31, 2007

Seeking an IL Assessment The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Seeking an IL Assessment The BOR institutions’ needs: true assessment of IL (not ITL) short, yet valid and reliable, instrument student-level information ACRL National Conference, Baltimore, March 31, 2007

The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota IL Subcommittee Co-chaired by BOR assessment expert and a librarian (IL Coordinator, USD) 5 assessment experts (including one psychometrician, USD’s Director of Assessment) 5 library faculty 2 English instructors 1 Communication Studies instructor Charged with creating an assessment with special properties Since no suitable national standardized assessment tool was available, the SLT constituted the IL Subcommittee (consisting of five assessment directors, five librarians, two English instructors, and one Communication Studies instructor) in the fall of 2004. Mandate = Create IL Exam based on USD’s ITL Exam Create administrative guidelines ACRL National Conference, Baltimore, March 31, 2007

Special Properties Required of the SD IL Exam (SDILE) The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Special Properties Required of the SD IL Exam (SDILE) Brevity: 25 multiple-choice questions Online delivery Content valid vis-à-vis the Association of College & Research Libraries’ (ACRL) IL Competency Standards for Higher Education Discrete cutoff (proficiency threshold) Continuous (assessment) scores Both documents and assesses IL The IL Subcommittee met in Nov. 2004 to discuss administration and to create questions. exam would be online assessment using WebCT twenty-five multiple-choice questions one correct answer and three distracters. At this point, we asked the SLT exactly what they wanted the IL Exam to do: document IL or assess it? Expecting to be told one or the other function, we were somewhat surprised to be told to accomplish both. ACRL National Conference, Baltimore, March 31, 2007

IL Exam Questions The Solution: Two scoring methods The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota IL Exam Questions The Solution: Two scoring methods The documentation questions will have low and very similar item “difficulties” (locations in IRT terms). The assessment questions will be more difficult and be more varied in their “difficulty”. For each set of 5 questions 3 documentation: documents attainment of minimum level of Information Literacy, i.e., close to a fixed point 2 assessment: assesses levels of Information Literacy along a continuum, i.e., along a line. Will Documentation Items Assessment Items “Difficulty” Proficiency Point ACRL National Conference, Baltimore, March 31, 2007

The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota SLO 2: Documentation Question Access the needed information effectively and efficiently Why is Interlibrary Loan so valuable to a student’s research? a. It allows a student to visit and check out materials from a library that is not his/her local library. b. It allows a student to request materials from a library that is not his/her local library. c. It allows a student to access online materials at a library that is not his/her local library. d. It allows a student to purchase materials not located in his/her local library. Classical Difficulty = .87 ACRL National Conference, Baltimore, March 31, 2007

The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota SLO 2: Assessment Question Access the needed information effectively and efficiently Your instructor has given an assignment that requires the use of primary source materials. Which would you consult? a. a biography of someone involved in the issue with criticism b. a diary written by someone who was involved in the issue c. a textbook article about someone who was involved in the issue d. a journal articles about someone who was involved in the issue Classical Difficulty = .63 ACRL National Conference, Baltimore, March 31, 2007

The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Analyses Classical: average item difficulties across ALL item types is around 0.70, giving a pass rate of about 98% with a cut-score of 13/25. Two IRT analyses for the items relevant to each ACRL Standard (unidimensionality) 2PL (Rasch) to look for location (Θ) and discrimination (slopes) Nominal (Bock) ACRL National Conference, Baltimore, March 31, 2007

2PL (Rasch) Traces for Items The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota 2PL (Rasch) Traces for Items ACRL National Conference, Baltimore, March 31, 2007

“Partial Credit” Traces for SLO2 Documentation The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota “Partial Credit” Traces for SLO2 Documentation 0.2 0.4 0.6 0.8 1.0 -3 -2 -1 1 2 3 Nominal Response Model Item Characteristic Curve b a Probability c d Ability ACRL National Conference, Baltimore, March 31, 2007

The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Ongoing Test Revision Items are and will be added, revised, tested and dropped with each IL testing cycle. Biased (DIF) items will be removed. Gender Ethnicity Location Etc. ACRL National Conference, Baltimore, March 31, 2007

Classical Scores for the SD System The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Classical Scores for the SD System ACRL National Conference, Baltimore, March 31, 2007

Pass rates for the SD BOR Institutions The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Pass rates for the SD BOR Institutions ACRL National Conference, Baltimore, March 31, 2007

Nominal (Bock) Scores for the SD BOR Institutions The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Nominal (Bock) Scores for the SD BOR Institutions ACRL National Conference, Baltimore, March 31, 2007

The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Additional Evidence: Correlations Between Nominal and Classical Scores (n = 2171) NOM SLO2 NOM SLO3 NOM SLO4 NOM SLO5 # Correct >12 Correct NOM SLO1 .11 .08 -.02 -.01 .32 .17 .12 .04 .06 .43 .20 -.09 .40 .23 -.14 -.04 .19 .53 (Red correlations are not significant at p < .05) ACRL National Conference, Baltimore, March 31, 2007

SLO 4: Documentation Question (Old 4D2a) The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota SLO 4: Documentation Question (Old 4D2a) To best demonstrate the scope of a problem one should use... a. pictures. b. statistics. c. books. d. articles. ACRL National Conference, Baltimore, March 31, 2007

Item Characteristics (Old 4D2a) The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Item Characteristics (Old 4D2a) 0.2 0.4 0.6 0.8 1.0 -3 -2 -1 1 2 3 a b c d Ability Probability Bock Nominal Response Model ACRL National Conference, Baltimore, March 31, 2007

ACRL Student Learning Outcome #4 The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota ACRL Student Learning Outcome #4 An Information Literate student will use information effectively to accomplish a specific purpose. Problems: The question lacks a context? This SLO requires higher cognitive processing which is difficult to test with MC items? This SLO represents the intersection between research (taught by the library) and argumentation (taught in ENGL and SPCM). Should use of information in argumentation be more effectively taught in gen. ed. courses? What do you think? ACRL National Conference, Baltimore, March 31, 2007

4D2a Revised The best visual aid for a speech The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota 4D2a Revised The best visual aid for a speech comparing changes in the profits of two or three competing companies over a three-year period is... a. a spreadsheet. b. a market analysis. c. a line chart. d. a table. ACRL National Conference, Baltimore, March 31, 2007

Item Characteristics (New 4D2a)

Benefits of the SDILE A dual-purpose IL Exam Random item rotation The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Benefits of the SDILE A dual-purpose IL Exam Documentation Assessment Random item rotation Continuous improvement and refinement Low cost – WebCT administration ACRL National Conference, Baltimore, March 31, 2007

The End of the SDILE: Bureaucratic Blunder The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota The End of the SDILE: Bureaucratic Blunder Feb. 2005: The IL Exam questions (with answers) were posted to the BOR web site (www.sdbor.edu) 2006-2007: Conference presentations on the IL Exam actively sought beta-testing partners April 2007: Acclaimed presentation at ACRL, interest in beta-testing/cooperation from Project SAILS and 9 prestigious colleges and research universities Carol. ACRL National Conference, Baltimore, March 31, 2007

The End of the SDILE: Bureaucratic Blunder The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota The End of the SDILE: Bureaucratic Blunder April 2007: Student taking the IL Exam discovered the questions/answers online at the SD BOR web site (invalidated the pilot) Investigators discovered that the IL Exam questions had been downloaded 293 times (in-state, in-country, and abroad) Beta-testing partners were notified to stop the pilot ACRL National Conference, Baltimore, March 31, 2007

The End of the SDILE: The Vultures Gather… The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota The End of the SDILE: The Vultures Gather… A member regental university had been a “less than enthusiastic” participant Once the IL Exam pilot was invalidated, this university immediately proposed dropping the exam New BOR academic officer had also problematized the notion of an IL Exam May 2007: AAC/BOR persuaded to drop the IL Exam as a system IL requirement ACRL National Conference, Baltimore, March 31, 2007

The End of the SDILE: USD Drops the IL Exam The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota The End of the SDILE: USD Drops the IL Exam Spring 2007: USD decides to reconstitute the IL Exam as an institutional measure Summer 2007: Will leaves USD for Boise State U. USD refuses to hire Will as consultant USD claims IP ownership of the IL Exam ACRL National Conference, Baltimore, March 31, 2007

Lessons Learned Avoid unfunded mandates The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Lessons Learned Avoid unfunded mandates Get release time or some other tangible benefit, in writing, before beginning Get buy-in rather than imposing mandates Institutions should genuinely support the project Members should support the project, even if their institutions don’t Give creative teams room to work; don’t oversee or micro-manage ACRL National Conference, Baltimore, March 31, 2007

Lessons Learned Choose participants carefully. Members should… The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Lessons Learned Choose participants carefully. Members should… be chosen because of subject know-ledge and competence be competent to understand all members’ roles in the project promote the project’s agenda rather than their own or their institution’s agenda ACRL National Conference, Baltimore, March 31, 2007

Lessons Learned Clarify IP issues Protect secret information The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota Lessons Learned Clarify IP issues Work out IP ownership issues up-front BOR IP Institutions’ IP Individual members’ IP vs. work product Protect secret information Clear those who handle information Don’t post to unprotected sites Provide consequences for divulging IP Don’t expect kudos, gratitude, or apologies from administration during or after the project ACRL National Conference, Baltimore, March 31, 2007

For further information… The South Dakota Regental Information Literacy Exam: A Tool for Small and Medium-Sized Universities to Document and Assess Information Literacy Carol Leibiger and William Schweinle, University of South Dakota For further information… On the SDILE Carol Leibiger, Head of Public Services and Reference, Information Literacy Coordinator, University of South Dakota C.Leibiger@usd.edu William Schweinle, Assistant Professor of Psychology, Boise State University WillSchweinle@boisestate.edu On IRT Reise, S. P., Ainsworth, A. T., & Haviland, M. G. (2005). Item Response Theory: Fundamentals, applications, and promise in psychological research. Current Directions in Psychological Research, 14, 2, 95-101. ACRL National Conference, Baltimore, March 31, 2007