1. Assessment of Information Skills

2. There is more teaching going on around here than learning and you ought to do something about that. Graduating Senior King’s College, 1968

3. Assessment A systematic ongoing process of  setting goals or asking questions,  gathering information,  interpreting it, and  using it to improve student learning. Barbara D. Wright

4. Why not use grades? “A grade is an inadequate report of an inaccurate judgment by a biased and variable judge of the extent to which a student has attained an undefined level of mastery of an unknown proportion of an indefinite material.” Bill Boyle & Tom Christie Issues in Setting Standards Falmer Press, 1996

5. Grades may reflect many things besides mastery of course objectives: Verbal ability ParticipationCooperation Extra credit AttendanceEffort Criterion Performance vs. Value Added Myths regarding student evaluations

6. Pay me now or Pay me later

7. Align your Outcomes Institutional outcomes Program outcomes Course outcomes Class outcomes

8. Dimensions of Assessment  Formative - Process Feedback for improvement Feedback for improvement Corrective & diagnostic Corrective & diagnostic  Summative – Evaluative Judgmental Judgmental Decisional Decisional

9. Dimensions of Assessment  An individual  A cohort  A class  A course  A program  An institution

10. Quantitative Data  Numerical  Statistically reliable  Structured techniques  Objective  Assumes static reality  Usually can be generalized  Allows for comparison and correlation  Looks for facts and causes

11. Qualitative Data  Not numerical  Provides “richness”  Allows for ambiguities  Structured or semi-structured  Written, verbal, visual, etc  Often subjective  Can be exploratory  Assumes dynamic reality  Looks for motivations and points of view

12. AssumptionsVs.Assessment

13. Assessment  Enhance student learning and institutional effectiveness  Student learning rather than faculty evaluation  Assess the process, rather than the outcome, for the purpose of improving the outcome

14. Measure What Matters  Measure what matters, NOT what is measurable  Because what you measure becomes what you focus on

15. Assessment  Knowing what you are doing  Knowing why you are doing it  Knowing what students are learning as a result  Changing because of the information Debra Gilchrist

16. Indirect Evidence of Student Learning  Surveys, self-reports & journals  Focus groups & interviews  Alumni & employer surveys  Percent of students entering graduate or professional schools

17. Limitations of Indirect Measures  Provide data on factors that predict or mediate learning  DO NOT evaluate learning per se  DO NOT NECESSARILY imply that value- added learning has or has not occurred (e.g., enthusiasm or lack of interest may be caused by factors not related to the course) Oswald Ratteray

19. Direct Evidence of Student Learning  Student assignments  Standardized tests  Course embedded assessments  Portfolios of students’ work  Capstone experiences  Student performances & exhibits  Other observations of student behavior

20. Direct Evidence of Student Learning  Rubrics & exemplars  Concept maps  Juried/peer review of student projects  Performance on a case study or problem  Locally devised tests  Commercially produced tests & exams

21. Limitations of Direct Measures  Indicate: WHAT students learned WHAT students learned HOW MUCH they learned HOW MUCH they learned What they DID NOT learn What they DID NOT learn  DO NOT indicate: WHY students learn or did not learn WHY students learn or did not learn  DO NOT necessarily indicate: Whether VALUE-ADDED learning occurred Whether VALUE-ADDED learning occurred Oswald Ratteray

22. Develop Rubrics to Assess Work:  Levels of achievement  Criteria that distinguish good work from poor work  Descriptions of criteria at each level of achievement Peggy Maki

23. Basic Rubric Grid Format Title Scale Level 1 Scale Level 2 Scale Level 3 Dimension 1 Dimension 2 Dimension 3 Dimension 4 Task Description

24. Why use a Rubric?  Examine complex work efficiently  Clarifies faculty expectations  Communicates expectations  Improves students’ work  Criterion-referenced grades rather than normative-referenced  Facilitates course or program assessment

25. Examples

26. Portfolios  Teaching portfolio Documents teaching over time Documents teaching over time  Course portfolio Reflection & review of a single course Reflection & review of a single course  Student portfolio Highlights student learning Highlights student learning

27. Portfolio Elements  Course design, philosophy, rationale  Implementation  Results  Reflection

28. Portfolio Elements  Focus on the match between assessment & course goals  Use existing assessments & data, when possible  Include a variety of evidence types  Be purposeful when selecting evidence about learning

29. Assessment is not about us, it is about student learning. it is about student learning.

30. Assessment is like scotch, it is an acquired taste.

31. Questions?

32. Seniors’ Information Skills by Standard : Mean Scores/Percentage Correct - Fall 2004* GroupN Composite Scores Std 1 Std 2 Std 3 Std 4 Std 5 14257.3260.5656.6260.0058.4550.99 A2250.3652.7357.2752.73 43.64 b 45.45 B3856.0058.9549.4757.3761.5852.63 C2157.5255.2454.2961.90 65.71 a 50.48 D1558.6765.3357.3366.6753.3350.67 E 26 2660.3163.0865.3860.0062.31 50.77 50.77 F1161.0969.0956.3663.6460.0056.36 G964.0073.3364.4468.8960.0053.33 ab Refers to comparisons within column where the MEAN scores of group a are significantly (P<.05) higher than the MEAN scores of group b. *Please note that in some cases the small group size and number of questions per standard may preclude more meaningful statistical comparisons.

33. Seniors’ Information Skills by Standard : Mean Scores/Percentage Correct -Fall 2004* GroupN Composite Scores KnowledgeApplication Seniors14257.3261.8053.20 A2250.36 50.76 b 50.76 b50.00 B3856.0058.5553.64 C2157.5261.1154.21 D1558.6767.7850.26 E2660.31 66.67 a 66.67 a54.44 F1161.0968.1854.55 G964.00 72.22 a 72.22 a56.41 ab Refers to comparisons within column where the MEAN scores of group a are significantly (P<.05) higher than the MEAN scores of group b. *Please note that in some cases the small group size and number of questions per standard may preclude more meaningful statistical comparisons.

34. Information Skill of Seniors & Graduate Students from Three Northeast, Pennsylvania Colleges by Standard: Mean Scores/Percentage - Fall 2004 * CollegeN Composite Scores Std1Std2Std3Std4Std5 38857.2261.2454.5455.3158.0456.96 1 (Srs) 14257.3260.5656.6260.0058.45 50.99 b 2 (Grads) 2956.2863.4548.2847.5957.24 64.83 a 3 (Srs) 21757.2761.3854.0153.2757.88 59.82 a ab Refers to comparisons within column where the MEAN scores of group a are significantly (P<.05) higher than the MEAN scores of group b. *Please note that in some cases the small group size and number of questions per standard may preclude more meaningful statistical comparisons.

35. Information Skill of Seniors & Graduate Students from Three Northeast, Pennsylvania Colleges by Standard : Mean Scores/Percentage - Fall 2004* CollegeN Composite Scores KnowledgeApplication 38857.2258.9355.63 1 (Srs) 14257.3261.8053.20 2 (Grads) 2956.2857.1855.44 3 (Srs) 21757.2757.3057.25 ab Refers to comparisons within column where the MEAN scores of group a are significantly (P<.05) higher than the MEAN scores of group b. *Please note that in some cases the small group size and number of questions per standard may preclude more meaningful statistical comparisons.

36. Five Stages of Assessment (from Elisabeth Kubler-Ross) 1. Denial n “No, not me” 2. Anger or resentment n “Why me?” 3. Bargaining n “Yes, me, but…” 4. Depression n “Yes, me” 5. Acceptance