Doing Assessment as if Teaching & Learning Matter Most A Half-Day Workshop for Faculty and Guests Syracuse University Friday 14 October 2016 Tom Angelo

For all our efforts . . . Are we making any meaningful progress on promoting learning outcomes?

have at least a 4-year degree. More than 33% of US adults have at least a 4-year degree. About 42% have at least a 2-year degree. Have those gains in attainment led to a more civil, equitable and sustainable society? Notice I didn’t even bother mentioning the election.

Page 1 Middle [ I, III, VI ] Values Affirmation Jot down 2 or 3 educational/ professional values you hold that motivate you to invest time and energy in assessment efforts at Syracuse. _________________________________________

Which value(s) did you mention? Promoting equity Improving teaching effectiveness Improving student learning Promoting innovation Making assessment less burdensome When’s the coffee break? 5 5

Why Might We Assess Learning? - I Summative purposes To compare learners against each other To compare learning against criteria To certify competency To award qualifications To ration resources To provide accountability

Why Might We Assess Learning? - II Formative purposes To focus learners’ attention To illuminate and undermine misconceptions To increase motivation to learn To provide learners with feedback To improve performance To promote self-assessment & monitoring To develop independent, lifelong learning skills

Yet the greater part of our time and effort is spent on In other words, virtually all of our core lifelong learning values are expressed through formative assessment. Yet the greater part of our time and effort is spent on summative assessment.

To do assessment as if deep learning matters most, we must privilege formative purposes . . .

. . . and make our summative assessments seamlessly aligned means to achieve our most valued educational end – deep, meaningful, long-lasting learning.

Page 1 bottom [ I, III, VI ] Goal Ranking & Matching Exercise What are your Learning Goals and/or Burning Questions for this workshop? _________________________________________

My Intended Learning Outcomes (ILOs) for this Session By the end of this workshop, you will have: Identified at least two or three (2-3) strategies, guidelines or techniques to improve teaching, assessment, feedback, and/or grading in your work Identified at least two (2) useful resources and/or references for follow up Committed to experimenting with at least one small change in your practice, and 4. Talked with at least two folks you don’t normally get an opportunity to talk with about these topics

In other words, I’m hoping you’ll end this session with . . . Greater conceptual clarity Useful guidelines Useful strategies and techniques Useful references/resources Connections with colleagues 13 13

A First “Balcony” Question Could there be advantages to knowing something about your students’ goals? values? 14 14

Page 17 – [ III, IV, V ] Applications Card Ideas/Techniques Possible Applications

Background Knowledge Probe page 2 1. Of first-year students in 4Y US colleges in Fall 2015, what percentage graduated from: A. A traditional public school? ___ % B. A charter/magnet public school? ___% C. A private religious/parochial school? ___ % D. A private independent prep school? ___% E. Were home schooled? ___%

Background Knowledge Probe page 2 2. Of those same students, what %-age rated the following as “very important” reasons to go to college: A. To be able to get a better job? ___ % B. To learn more about things that interest me? ___% C. To get training for a specific career? ___ % D. To gain a general ed. & appreciation for ideas? ___% E. To be able to make more money? ___% F. To prepare myself for grad/prof school? ___% G. To make me a more cultured person ___%

Background Knowledge Probe page 2 3. Of those same students, what %-age thought they had a “very good chance of doing the following in college: A. Socializing w/different racial/ethnic person? ___ % B. Earning at least a ‘B’ average? ___% C. Communicating regularly with profs? ___ % D. Protesting or demonstrating? ___%

Background Knowledge Probe page 2 4. About what %-age of the 2015 FYs had: A. First parent with at least a 4-year degree? ___ % B. Second parent with at least a 4-year degree? ___%

Second “Balcony” Question If you followed directions: Are you more interested in learning the answers to questions on p. 2 than you would have been if I’d simply told that information to you? 20 20

Background Knowledge Probe page 2 1. Of FY students in 4Y US colleges in 2015, what percentage graduated from: A. A traditional public school? 74% B. A charter/magnet public school? 8% C. A private religious/parochial school? 10% D. A private independent prep school? 7% E. Were home schooled? 1%

Background Knowledge Probe page 2 2. Of those same students, what %-age rated the following as “very important” reasons to go to college: A. To be able to get a better job? 85% B. To learn more about things that interest me? 82% C. To get training for a specific career? 76 % D. To gain a general ed. & appreciation for ideas? 71% E. To be able to make more money? 70% F. To prepare myself for grad/prof school? 59% G. To make me a more cultured person 49%

Background Knowledge Probe page 2 3. Of those same students, what %-age thought they had a “very good chance of doing the following in college: A. Socializing w/different racial/ethnic person? 71% B. Earning at least a ‘B’ average? 68% C. Communicating regularly with profs? 44% D. Protesting or demonstrating? 9%* * Upon seeing these survey results, college presidents all over the US ordered their Admissions Offices to identify this 9%.

Background Knowledge Probe page 2 4. About what %-age of the 2015 FYs had: A. First parent with at least a 4-year degree? 61% B. Second parent with at least a 4-year degree? 60%

Effective Assessment is often just Gap Analysis Find the gaps Mind the gaps Narrow the gaps 25 25

More “Balcony” Questions Imagine that your first-year students had similar responses: Could you see any implications for: The courses you teach? For your program? For campus assessment efforts? 26 26

Six Dimensions of Higher Learning Outcomes Page 3 [ II, III, IV, VII] Six Dimensions of Higher Learning Outcomes % Your Past? % Their Future? ____ Factual Learning _____ ____ Conceptual Learning _____ ____ Procedural Learning _____ ____ Conditional Learning _____ ____ Metacognitive Learning _____ ____ Reflective Learning _____ 100% 100% 27

Balcony Question Which of those six dimensions needs and deserves the most focus if we aim to foster: Creative thinking? Problem-solving? Innovation? Life-long learning?

Metacognition involves . . . Self-Awareness Knowing that and when you are thinking Self-Monitoring Noticing the quality/intensity of your thinking Self-Regulation Directing/Correcting your thinking Reflection requires metacognition, but goes beyond it to affect/change values, beliefs, actions and/or habits 29

A Reflection-in-Action Exercise Rate your own learning approach thus far in the session: 1. Disengaged (I don’t see the point, or I already know all this.) 2. Semi-attentive (Checking in and out, per my interests.) 3. Engaged recipient (I’m taking it all in, but quietly.) 4. Active Participant (I’m contributing & cooperating.) 5. Reflective Synthesizer (I’m making connections to prior knowledge/experience and imagining how I might use this in my future work.) Adapted from: Guskey, T. R. (2000). Evaluating Professional Development. Thousand Oaks, CA: Corwin Press, p. 135. 30

can be promoted effectively through High-Impact Practices HOT HIPs! Page 4 - middle Higher Order Thinking can be promoted effectively through High-Impact Practices 31 31

HIPs – High-Impact Educational Practices First-Year Seminars and Experiences Learning Communities Collaborative Assignments and Projects Undergraduate Research Diversity/Global Learning Service Learning/Community-Based Learning Clinical Placements/Internships/Co-ops Capstone Courses and Projects Writing-Intensive Courses

What makes these HIPs so HOT? High expectations Explicit direct instruction Metacognitive scaffolding Deliberate practice Effective feedback Focused collaboration 33 33

“It’s not what we do, but what students do that’s the important thing Biggs, J. & Tang, C. (2007). Teaching for Quality Learning at University, 3rd Edition. Berkshire: McGraw-Hill, p. 19. 34

Backward Course (Re)Design – A Simple Self-Assessment 1 Program ILOs 2 Page 5 - Top Backward Course (Re)Design – A Simple Self-Assessment Step # Sequential Steps in an Ideal Backward Design Column 2 Observed Sequence Column 3 Preferred Sequence 1 Program ILOs 2 Course ILOs 3 Standards 4 Summative Assessment 5 Formative Assessment 6 Learning Activities, etc. 7 Teaching Strategies 8 Course/Program Eval.

“You can’t fix by analysis what you bungled by design. ” Light, R “You can’t fix by analysis what you bungled by design.” Light, R., Singer, J. & Willett, J. (1990). By Design. Cambridge, MA: Harvard U. Press.

Page 6 – Top – [ II, VII ] The Plus-Minus-Question Mark Technique Formative and summative assessment Backward design Constructive alignment Gap analysis Motivated reasoning Implicit bias Bus, Parrot & Parking Lot Tests Cognitive load Metacognition The Dance Floor & The Balcony Deliberate practice Novice-Expert differences 37

Bloom’s Taxonomy (revised) Anderson & Krathwohl, 2001 Create Design Invent Hypothesise Developed to explain what terms like “understanding” actually translate to in action (learning). Assess Explain Critique Decide Justify Evaluate Determine Induce Question Appraise Analyse Adapt Modify Verify Demonstrate Apply Arrange Categorise Convert Formulate Generalise Understand Identify Do simple procedure Remember

Page 7 – bottom [ II, IV, VI ] “Blooming” – Categorizing Questions 39 39

Page 7 Categorizing Questions by Bloom’s Taxonomy Levels (Blooming) By systematically varying the elements of the task, we can better assess and promote transfer and deep learning. 40 40

Page 8 - top The Bondy Scale 41 41

Bondy’s Clinical Performance (Skills) Rating Scale Independent plus capable of assessing, instructing and supporting [6. EXPERT] Safe, Very proficient, Requires no support or direction 5. INDEPENDENT Safe, Proficient, Requires only occasional support and direction 4. SUPERVISED Safe, Mostly proficient, Requires frequent support and direction 3. ASSISTED Safe only when supervised, Unskilled, Requires continuous support and direction 2. MARGINAL Unsafe, Unaware & Unskilled 1. DEPENDENT

A Detailed Assessment/Grading Rubric Page 9 – [ I, IV, V ] A Detailed Assessment/Grading Rubric What matters most in this course? How do you know? 43 43

“From the student’s point of view, the assessment is the curriculum.” Paul Ramsden 44

Making Groupwork Work Well A Checklist Approach Page 10 Making Groupwork Work Well A Checklist Approach Checklists are a simple way to manage cognitive load 45 45

Groupwork Feedback Form Page 11 Groupwork Feedback Form 46 46

Group Informal Feedback on Teaching Page 12 – Top The G.I.F.T. Group Informal Feedback on Teaching 47

A Course/Teaching Feedback Form Page 13 – [ I, V, VI ] A Course/Teaching Feedback Form What values do our course and teaching assessment systems convey to students? To faculty? 48 48

“The effectiveness of teaching is best evaluated by what the students do when the teacher is not present.”

Page 17 Applications Card – [ III, IV, V] Interesting Possible IDEAS/TECHNIQUES APPLICATIONS 50 50

The Parking Lot Test 51 51

What, Why and How Choose one of your possible applications. Prepare to answer the three questions below about that specific application: What is it? Why do you think it might be useful? How do you think you might use it? 52 52