1. 1 Taking a Backward-Designed Collaborative Approach to Assessing Student Learning to Explore both Students' Learning Processes and Products Peggy Maki Education Consultant Specializing in Assessing Student Learning Assessment Editor and Writer Keynote Presented at NSEE October 4, 2012

2. How well do your students… Integrate Transfer Analyze Apply or re-apply Re-use Synthesize Restructure previous incorrect learning…. 2

3. Within a course or module or learning experience? Along the chronology of their studies and educational experiences? From one subject or topic or focus or context to another one such as from an exercise to a case study or internship? 3

4. Percent Students Forget When They Log off or Leave Your Learning Experience 4

5. Foci Research on Learning That Informs Teaching, Learning, and a Collaboratively Designed Chronological Approach to Assessing Student Learning A Backward Designed Problem-based Approach to Assess Students’ Enduring Learning 5

6. Research on Learning That Informs Teaching, Learning, and Assessing Student Learning 6

7. 7 Collaboration Technology Service Learning Work Life Dorm Life Services Learning Communities Courses/ Modules Internships Study Abroad Advising Learner Knowledge is Socially Constructed (importance of peer-to-peer interaction)

8. Research on….. Threshold Concepts: pathways central to the mastery of a subject or discipline that change the way students view a subject or discipline, prompting students to bring together various aspects of a subject that they heretofore did not view as related (Land, Meyer, Smith). 8

9. 9

10. What’s The Problem in Physics? Through the use of concept inventories, it is well documented that entry-level physics majors hold onto incorrect understanding of physics concepts—even ones that have been historically proven to be incorrect. 10

11. Despite all attempts to correct these incorrect concepts, such as incorporating more labs into early courses, students continue to draw on their incorrect understanding years after their early coursework often accounting for their weak performance as they progress in their studies or resulting in their dropping out of the major. 11

12. How to restructure incorrect understanding of physics concepts became the work of physics faculty at the University of Colorado (PhET project). 12

13. 13

14. Learning Progressions: knowledge- based, web-like interrelated actions or behaviors or ways of thinking, transitioning, self-monitoring. May not be developed successfully in linear progression thus necessitate formative assessment along the trajectory of learning. Movements towards increased understanding (Hess). 14

15. Writing beyond what is visually presented during a lecture Identifying clues to help organize information during a lecture Evaluating notes after class Reorganizing notes after class 15

16. Comparing note-taking methods with peers Using one’s own words while reading to make notes Evaluating one’s understanding while reading Consolidating reading and lecture notes 16

17. Sharing practices on how to organize, think, and memorize content Evaluating one’s own understanding Monitoring the effectiveness of note- taking practices Source: Calvin Y. Yu, Director of Cook/Douglass Learning Center, Rutgers University 17

18. 18

19. 19

20. Approaches to Learning… Surface Learning Deep Learning 20

21. NRC, 2002 21

22. Integrated Learning…. 22 Cognitive Affective Forms of Representation within Contexts Psychomotor

23. Collaboratively Agreed upon Learning Outcomes at Multiple Levels Foster Enduring Learning 23 Institution-level Outcomes (GE) Program- or Department-level Outcomes, including GE Course/Service Outcomes/Educational Opportunities Outcomes (including GE)

24. How Knowledge about Learning Shapes Teaching and Inquiry into Student Learning What do you expect your students to demonstrate, represent, or produce by the end of their program of study or by the end of their undergraduate or graduate studies? What chronological barriers or difficulties do students encounter as they learn--from the moment they matriculate? How well do we identify and discuss those barriers and then track students’ abilities to overcome them so that “more” students achieve our expectations? 24

25. Introduce (I) Reinforce (R) Emphasize (E) Intergroup Dialogues Internship Community Service Outcome 1: Speak to different audiences for a range of purposes (I) Group Activities involving Interpersonal Communication (E) Community Presentation (E) Collaborative Projects with Surrounding Community Outcome 2: 25 Mapping Multiple and Diverse Opportunities across Students’ Journey

26. 26 A Backward Designed Problem-based Approach to Assessment That Is Grounded in Research or Study Questions…

27. 27

28. Research or Study Questions 28

29. The Seeds of Research or Study Questions 29

30. A Taxonomy…. Identifies patterns of weakness in student work or performance: analysis, support of claims, quantitative reasoning, etc. Identifies context within which those obstacles occur. With students, attempts to identify “why” students are struggling with those obstacles. Chronologically alerts colleagues to weaknesses that still need to be addressed. 30

31. Some Examples of Research/Study Questions 31

32. 32

33. 33

34. 34

35. Identification or Design of Assessment Methods That Provide Evidence of Product and Process 35

36. Direct Assessment Methods  Scenarios—such as online simulations  Critical incidents  Mind mapping  Questions, problems, prompts  Data analysis or data conversion 36

37. Problem with solution: Any other solutions? Chronological use of case studies Chronological use of muddy problems Analysis of video Smaller problems that ready students for internships, capstone projects, applied learning opportunities 37

38. Self-reflective journals or self- reflection on strengths and weaknesses in one’s own work leading to acceptance of the need to improve Think aloud 38

39. Some Indirect Assessment Methods Student Assessment of Their Learning Gains: SALG Small Group Instructional Design: SGID NSSE/CCSSE Interviews/surveys 39

40. 1. Identify the outcome or outcomes you will assess. 1. Identify the outcome or outcomes you will assess. 5. Analyze and Interpret Students’ Work and Students’ Responses. 5. Analyze and Interpret Students’ Work and Students’ Responses. 4. Develop a Plan to Collect Direct and Indirect Assessment Results that Will Answer Your Question. 4. Develop a Plan to Collect Direct and Indirect Assessment Results that Will Answer Your Question. 2. State the Research or Study Question You Wish to Answer. 2. State the Research or Study Question You Wish to Answer. 3. Conduct a Literature Review about that Question. 3. Conduct a Literature Review about that Question. 6. Collaboratively Discuss Ways to Improve Pedagogy or Educational Practices 6. Collaboratively Discuss Ways to Improve Pedagogy or Educational Practices 7. Implement Agreed-upon Changes and Reassess. 7. Implement Agreed-upon Changes and Reassess. 8. Share Developments within and outside the Institution to Build Knowledge about Educational Practices. 8. Share Developments within and outside the Institution to Build Knowledge about Educational Practices. 40

41. Soft Times and Neutral Zones 41

42. Works Cited Hess, K. 2008. Developing and Using Learning Progressions as a Schema for Measuring Progress. National Center for Assessment, 2008. http://www.nciea.org/publications/CCSSO2_KH08.pdf Maki, P. 2010. 2 nd Ed. Assessing for Learning: Building a Sustainable Commitment Across the Institution. VA: Stylus Publishing, LLC Meyer, J.H., Land, R., and Baillie, C. 2010. Eds. Threshold Concepts and Transformational Learning. Rotterdam: Sense Publishers. National Research Council. 2002. Knowing What Students Know: The Science and Design of Educational Assessment. Washington, D.C. Yu, C. Y. “Learning Strategies Characteristic of Successful Students.” Maki, P. 2010. p. 139. 42