Assessment of Information Skills

There is more teaching going on around here than learning and you ought to do something about that. Graduating Senior King’s College, 1968

Assessment A systematic ongoing process of  setting goals or asking questions,  gathering information,  interpreting it, and  using it to improve student learning. Barbara D. Wright

Why not use grades? “A grade is an inadequate report of an inaccurate judgment by a biased and variable judge of the extent to which a student has attained an undefined level of mastery of an unknown proportion of an indefinite material.” Bill Boyle & Tom Christie Issues in Setting Standards Falmer Press, 1996

Grades may reflect many things besides mastery of course objectives: Verbal ability ParticipationCooperation Extra credit AttendanceEffort Criterion Performance vs. Value Added Myths regarding student evaluations

Pay me now or Pay me later

Align your Outcomes Institutional outcomes Program outcomes Course outcomes Class outcomes

Dimensions of Assessment  Formative - Process Feedback for improvement Feedback for improvement Corrective & diagnostic Corrective & diagnostic  Summative – Evaluative Judgmental Judgmental Decisional Decisional

Dimensions of Assessment  An individual  A cohort  A class  A course  A program  An institution

Quantitative Data  Numerical  Statistically reliable  Structured techniques  Objective  Assumes static reality  Usually can be generalized  Allows for comparison and correlation  Looks for facts and causes

Qualitative Data  Not numerical  Provides “richness”  Allows for ambiguities  Structured or semi-structured  Written, verbal, visual, etc  Often subjective  Can be exploratory  Assumes dynamic reality  Looks for motivations and points of view

AssumptionsVs.Assessment

Assessment  Enhance student learning and institutional effectiveness  Student learning rather than faculty evaluation  Assess the process, rather than the outcome, for the purpose of improving the outcome

Measure What Matters  Measure what matters, NOT what is measurable  Because what you measure becomes what you focus on

Assessment  Knowing what you are doing  Knowing why you are doing it  Knowing what students are learning as a result  Changing because of the information Debra Gilchrist

Indirect Evidence of Student Learning  Surveys, self-reports & journals  Focus groups & interviews  Alumni & employer surveys  Percent of students entering graduate or professional schools

Limitations of Indirect Measures  Provide data on factors that predict or mediate learning  DO NOT evaluate learning per se  DO NOT NECESSARILY imply that value- added learning has or has not occurred (e.g., enthusiasm or lack of interest may be caused by factors not related to the course) Oswald Ratteray

Direct Evidence of Student Learning  Student assignments  Standardized tests  Course embedded assessments  Portfolios of students’ work  Capstone experiences  Student performances & exhibits  Other observations of student behavior

Direct Evidence of Student Learning  Rubrics & exemplars  Concept maps  Juried/peer review of student projects  Performance on a case study or problem  Locally devised tests  Commercially produced tests & exams

Limitations of Direct Measures  Indicate: WHAT students learned WHAT students learned HOW MUCH they learned HOW MUCH they learned What they DID NOT learn What they DID NOT learn  DO NOT indicate: WHY students learn or did not learn WHY students learn or did not learn  DO NOT necessarily indicate: Whether VALUE-ADDED learning occurred Whether VALUE-ADDED learning occurred Oswald Ratteray

Develop Rubrics to Assess Work:  Levels of achievement  Criteria that distinguish good work from poor work  Descriptions of criteria at each level of achievement Peggy Maki

Basic Rubric Grid Format Title Scale Level 1 Scale Level 2 Scale Level 3 Dimension 1 Dimension 2 Dimension 3 Dimension 4 Task Description

Why use a Rubric?  Examine complex work efficiently  Clarifies faculty expectations  Communicates expectations  Improves students’ work  Criterion-referenced grades rather than normative-referenced  Facilitates course or program assessment

Examples

Portfolios  Teaching portfolio Documents teaching over time Documents teaching over time  Course portfolio Reflection & review of a single course Reflection & review of a single course  Student portfolio Highlights student learning Highlights student learning

Portfolio Elements  Course design, philosophy, rationale  Implementation  Results  Reflection

Portfolio Elements  Focus on the match between assessment & course goals  Use existing assessments & data, when possible  Include a variety of evidence types  Be purposeful when selecting evidence about learning

Assessment is not about us, it is about student learning. it is about student learning.

Assessment is like scotch, it is an acquired taste.

Questions?

Seniors’ Information Skills by Standard : Mean Scores/Percentage Correct - Fall 2004* GroupN Composite Scores Std 1 Std 2 Std 3 Std 4 Std A b B C a D E F G ab Refers to comparisons within column where the MEAN scores of group a are significantly (P<.05) higher than the MEAN scores of group b. *Please note that in some cases the small group size and number of questions per standard may preclude more meaningful statistical comparisons.

Seniors’ Information Skills by Standard : Mean Scores/Percentage Correct -Fall 2004* GroupN Composite Scores KnowledgeApplication Seniors A b b50.00 B C D E a a54.44 F G a a56.41 ab Refers to comparisons within column where the MEAN scores of group a are significantly (P<.05) higher than the MEAN scores of group b. *Please note that in some cases the small group size and number of questions per standard may preclude more meaningful statistical comparisons.

Information Skill of Seniors & Graduate Students from Three Northeast, Pennsylvania Colleges by Standard: Mean Scores/Percentage - Fall 2004 * CollegeN Composite Scores Std1Std2Std3Std4Std (Srs) b 2 (Grads) a 3 (Srs) a ab Refers to comparisons within column where the MEAN scores of group a are significantly (P<.05) higher than the MEAN scores of group b. *Please note that in some cases the small group size and number of questions per standard may preclude more meaningful statistical comparisons.

Information Skill of Seniors & Graduate Students from Three Northeast, Pennsylvania Colleges by Standard : Mean Scores/Percentage - Fall 2004* CollegeN Composite Scores KnowledgeApplication (Srs) (Grads) (Srs) ab Refers to comparisons within column where the MEAN scores of group a are significantly (P<.05) higher than the MEAN scores of group b. *Please note that in some cases the small group size and number of questions per standard may preclude more meaningful statistical comparisons.

Five Stages of Assessment (from Elisabeth Kubler-Ross) 1. Denial n “No, not me” 2. Anger or resentment n “Why me?” 3. Bargaining n “Yes, me, but…” 4. Depression n “Yes, me” 5. Acceptance