1. 1 Assessing for Learning Presented by Peggy L. Maki June 7, 2004 Seattle Pacific University PeggyMaki@aol.com Material from Maki, P. (2004). Assessing for Learning: Building a Sustainable Commitment Across the Institution. Stylus Publishing and AAHE.

2. 2 How Do You Learn? _______________________________ _______________________________ ________________________________ ________________________________

3. 3 Topics:  Research on Learning That Informs the Relationship among Teaching, Learning, and Assessment  Curricular and Co-curricular Coherence (Maps and Inventories)

4. 4 Development of Learning Outcome Statements Development of Learning Outcome Statements Alignment of Assessment Methods Alignment of Assessment Methods with Learning Outcome Statements and Collective Educational Practices with Learning Outcome Statements and Collective Educational Practices

5. 5 Terms: Learning Outcome Statements— sentences that describe what we expect our students to demonstrate, represent or produce as a result of how and what they learn. Learning Outcome Statements— sentences that describe what we expect our students to demonstrate, represent or produce as a result of how and what they learn. Coherence—degree to which the curriculum and co-curriculum intentionally offer multiple and varied opportunities for students to learn what the institution and its programs and services assert they teach Coherence—degree to which the curriculum and co-curriculum intentionally offer multiple and varied opportunities for students to learn what the institution and its programs and services assert they teach

6. 6 Maps and Inventories—strategies that enable us to ascertain how well we intentionally develop what we assert we expect students to learn Maps and Inventories—strategies that enable us to ascertain how well we intentionally develop what we assert we expect students to learn Alignment—degree to which learning outcome statements match how and what we teach; degree to which assessment methods methods match our collective educational practices Alignment—degree to which learning outcome statements match how and what we teach; degree to which assessment methods methods match our collective educational practices

7. 7 Research on Learning That Informs the Relationship among Teaching, Learning, and Assessment Learning is a complex process of interpretation--not a linear process Learning is a complex process of interpretation--not a linear process Learners create meaning as opposed to receive meaning Learners create meaning as opposed to receive meaning1.

8. 8 Knowledge is socially constructed (importance of peer-to-peer interaction ) Knowledge is socially constructed (importance of peer-to-peer interaction ) People learn differently—prefer certain ways of learning (learning inventories, such as Kolb or Vark) People learn differently—prefer certain ways of learning (learning inventories, such as Kolb or Vark)

9. 9 Deep learning occurs over time— transference Deep learning occurs over time— transference Meta-cognitive processes are a significant means of reinforcing learning (thinking about one’s thinking) Meta-cognitive processes are a significant means of reinforcing learning (thinking about one’s thinking)

10. 10 Learning involves creating relationships between short-term and long-term memory Learning involves creating relationships between short-term and long-term memory Transfer of new knowledge into different contexts is important to deepen understanding Transfer of new knowledge into different contexts is important to deepen understanding

11. 11 Practice in various contexts creates expertise Practice in various contexts creates expertise Surface Learning Surface Learning Deep Learning Deep Learning

12. 12 Curricular and Co-Curricular Coherence What do you do in your classes or in your programs or services to promote the kinds of learning or development that the institution seeks? What do you do in your classes or in your programs or services to promote the kinds of learning or development that the institution seeks? What do you expect your students to know and be able to do by the end of their education at your institution? What do you expect your students to know and be able to do by the end of their education at your institution?

13. 13 What do the curricula and other educational experiences “add up to?” What do the curricula and other educational experiences “add up to?” Which students benefit from specific teaching strategies or educational experiences? Which students benefit from specific teaching strategies or educational experiences? What educational processes are responsible for the intended student outcomes the institution seeks? What educational processes are responsible for the intended student outcomes the institution seeks?

14. 14 How can you help students make connections between classroom learning and experiences outside of the classroom? How can you help students make connections between classroom learning and experiences outside of the classroom? What pedagogies/educational experiences develop knowledge, abilities, habits of mind, ways of knowing/problem solving? What pedagogies/educational experiences develop knowledge, abilities, habits of mind, ways of knowing/problem solving?

15. 15 How are curricula, pedagogy, and educational experiences designed to develop knowledge, abilities, habits of mind, ways of knowing? How are curricula, pedagogy, and educational experiences designed to develop knowledge, abilities, habits of mind, ways of knowing? How do you intentionally build upon what each of you teaches or fosters to achieve programmatic and institutional objectives? How do you intentionally build upon what each of you teaches or fosters to achieve programmatic and institutional objectives?

16. 16 What methods of assessment capture desired student learning-- methods that align with pedagogy, content, and curricular design? What methods of assessment capture desired student learning-- methods that align with pedagogy, content, and curricular design?

17. 17 When a Student Becomes a Biologist, Psychologist, Engineer….. Cognitive AffectivePsychomotor

18. 18 Maps and Inventories Help us determine coherence among our educational practices that enable us to design appropriate assessment methods Help us determine coherence among our educational practices that enable us to design appropriate assessment methods Provide a visual representation of students’ journey Provide a visual representation of students’ journey

19. 19 Help students make meaning of the journey Help students make meaning of the journey Help students develop their own learning map Help students develop their own learning map

20. 20 Developing Learning Outcome Statements Institutional Outcomes Programmatic Outcomes Course Outcomes

21. 21 What Is an Outcome Statement? A sentence that describes what students should demonstrate, represent, or produce based on how and what they learn. A sentence that describes what students should demonstrate, represent, or produce based on how and what they learn. Emerges from what we value and how we teach or students learn; that is, it emerges from our educational practices and is developed through consensus. Emerges from what we value and how we teach or students learn; that is, it emerges from our educational practices and is developed through consensus.

22. 22 Relies on active verbs, such as create, compose, calculate, develop, build, evaluate, translate, etc., that target what we expect students to be able to demonstrate Relies on active verbs, such as create, compose, calculate, develop, build, evaluate, translate, etc., that target what we expect students to be able to demonstrate Is written for a course, program, or institution Is written for a course, program, or institution

23. 23 Is aligned with curriculum and co- curriculum Is aligned with curriculum and co- curriculum --varied and multiple ways to --varied and multiple ways to learn that desired outcome learn that desired outcome Is based on intentionality (established through dialogue) Is based on intentionality (established through dialogue)

24. 24 Helps students understand expectations and holds them accountable for their learning Helps students understand expectations and holds them accountable for their learning Can be assessed quantitatively or qualitatively Can be assessed quantitatively or qualitatively

25. 25 Example from ACRL ONE OUTCOME: Student examines and compares information from various sources in order to evaluate reliability, validity,accuracy, timeliness, and point of view or bias. ONE OUTCOME: Student examines and compares information from various sources in order to evaluate reliability, validity,accuracy, timeliness, and point of view or bias.

26. 26 Write one outcome statement that captures what a graduate from your institution or a program should be able to demonstrate, represent, or produce:

27. 27 How well does your outcome statement meet characteristics of a good statement? Describes learning desired within a context Describes learning desired within a context Relies on active verbs Relies on active verbs

28. 28 Emerges from your collective intentions Emerges from your collective intentions Can be mapped to curricular and co- curricular practices (multiple and varied opportunities to learn over time) Can be mapped to curricular and co- curricular practices (multiple and varied opportunities to learn over time) Can be assessed quantitatively or qualitatively during students’ undergraduate and graduate careers Can be assessed quantitatively or qualitatively during students’ undergraduate and graduate careers

29. 29 “The tasks to which students are asked to respond on an assessment are not arbitrary. They must be carefully designed to provide evidence that is linked to the cognitive model of learning and to support the kinds of inferences and decisions that will be based on the assessment results.” “The tasks to which students are asked to respond on an assessment are not arbitrary. They must be carefully designed to provide evidence that is linked to the cognitive model of learning and to support the kinds of inferences and decisions that will be based on the assessment results.” National Research Council. Knowing what students know: The science and design of educational assessment. Washington, D.C.: National Academy Press, 2001, p. 47. National Research Council. Knowing what students know: The science and design of educational assessment. Washington, D.C.: National Academy Press, 2001, p. 47.

30. 30 Assumptions Underlying Teaching Actual Practices Assumptions Underlying Assessment Tasks Actual Tasks

31. 31 Development of Assessment Methods Align methods with learning outcome statements and students’ educational experiences (verified through maps or inventories) Align methods with learning outcome statements and students’ educational experiences (verified through maps or inventories)

32. 32 When Will You Seek Evidence ? Formative—along the way? Formative—along the way? For example, to ascertain For example, to ascertain progress or development progress or development Summative—at the end? Summative—at the end? For example, to ascertain For example, to ascertain mastery level of achievement mastery level of achievement

33. 33 What Tasks Elicit Learning You Desire? Tasks that require students to select among possible answers (multiple choice test)? Tasks that require students to select among possible answers (multiple choice test)? Tasks that require students to construct answers (students’ problem-solving and thinking abilities)? Tasks that require students to construct answers (students’ problem-solving and thinking abilities)?

34. 34 Some Methods That Provide Direct Evidence Student work samples Student work samples Collections of student work (e.g. Portfolios) Collections of student work (e.g. Portfolios) Capstone projects Capstone projects

35. 35 Course-embedded assessment (derive examples; develop agreed upon prompt and ask students to respond to it in class or at a designated time) Course-embedded assessment (derive examples; develop agreed upon prompt and ask students to respond to it in class or at a designated time) Observations of student behavior Observations of student behavior Internal juried review of student projects Internal juried review of student projects

36. 36 External juried review of student projects External juried review of student projects Externally reviewed internship Externally reviewed internship Performance on a case study/problem Performance on a case study/problem Performance on case study accompanied with students’ analysis Performance on case study accompanied with students’ analysis

37. 37 Performance on national licensure examinations Performance on national licensure examinations Locally developed tests Locally developed tests Standardized tests Standardized tests Pre-and post-tests Pre-and post-tests

38. 38 Response to critical incident Response to critical incident Essay tests blind scored across units Essay tests blind scored across units Mapping Mapping Learning Logs or Journals Learning Logs or Journals Writing, Speaking, Visual Representation Writing, Speaking, Visual Representation

39. 39 Methods That Do Not Provide Direct Evidence but May Be Combined with Other Methods-Indirect Methods Faculty publications (unless students are involved) Faculty publications (unless students are involved) Courses selected or elected by students Courses selected or elected by students Faculty/Student ratios Faculty/Student ratios

40. 40 Percentage of students who study abroad Percentage of students who study abroad Enrollment trends Enrollment trends Percentage of students who graduate within five-six years Percentage of students who graduate within five-six years Diversity of student body Diversity of student body

41. 41 Focus group (representative of the population) Focus group (representative of the population) Interviews (representative of the population) Interviews (representative of the population) Surveys Surveys

42. 42 Other sources of information that contribute to your inference making: NSSE results, grades, participation rates or persistence in support services, course-taking patterns, majors Other sources of information that contribute to your inference making: NSSE results, grades, participation rates or persistence in support services, course-taking patterns, majors

43. 43 “What and how students learn depends to a major extent on how they think they will be assessed.” John Biggs, Teaching for Quality Learning at University: What The Student Does. Society for Research into Higher Education & Open University Press, 1999, p. 141.

44. 44 Works Cited Biggs, J. (1999). Teaching for Quality Learning at University: What The Student Does. Society for Research into Higher Education & Open University Press, 1999, p. 141. Biggs, J. (1999). Teaching for Quality Learning at University: What The Student Does. Society for Research into Higher Education & Open University Press, 1999, p. 141. Maki, P. (forthcoming, 2004., May). Assessing for Learning: Building a Sustainable Commitment Across the Institution. Sterling, VA: Stylus Publishing, LLC, and the American Association for Higher Education. National Research Council. 2001. Knowing What Students Know: The Science and Design of Educational Assessment. Washington, D.C.: National Academy Press