1. Doing Assessment as if Teaching & Learning Matter Most
A Half-Day Workshop for Faculty and Guests
Syracuse University
Friday 14 October 2016
Tom Angelo

2. For all our efforts . . .
Are we making any
meaningful progress on promoting learning outcomes?

3. have at least a 4-year degree.
More than 33% of US adults
have at least a 4-year degree.
About 42% have at least a 2-year degree.
Have those gains in attainment led to a more civil, equitable and sustainable society?
Notice I didn’t even bother mentioning the election.

4. Page 1 Middle [ I, III, VI ] Values Affirmation
Jot down 2 or 3 educational/ professional values you hold that motivate you to invest time and energy in assessment efforts at Syracuse.
_________________________________________

5. Which value(s) did you mention?
Promoting equity
Improving teaching effectiveness
Improving student learning
Promoting innovation
Making assessment less burdensome
When’s the coffee break? 5 5

6. Why Might We Assess Learning? - I
Summative purposes
To compare learners against each other
To compare learning against criteria
To certify competency
To award qualifications
To ration resources
To provide accountability

7. Why Might We Assess Learning? - II
Formative purposes
To focus learners’ attention
To illuminate and undermine misconceptions
To increase motivation to learn
To provide learners with feedback
To improve performance
To promote self-assessment & monitoring
To develop independent, lifelong learning skills

8. Yet the greater part of our time and effort is spent on
In other words, virtually all of our core lifelong learning values are expressed through formative assessment.
Yet the greater part of our
time and effort is spent on
summative assessment.

9. To do assessment as if deep learning matters most, we must privilege formative purposes . . .

10. . . . and make our summative assessments seamlessly aligned means to achieve our most valued educational end – deep, meaningful, long-lasting learning.

11. Page 1 bottom [ I, III, VI ] Goal Ranking & Matching Exercise
What are your Learning Goals and/or Burning Questions for this workshop?
_________________________________________

12. My Intended Learning Outcomes (ILOs) for this Session
By the end of this workshop, you will have:
Identified at least two or three (2-3) strategies, guidelines or techniques to improve teaching, assessment, feedback, and/or grading in your work
Identified at least two (2) useful resources and/or references for follow up
Committed to experimenting with at least one small change in your practice, and
4. Talked with at least two folks you don’t normally get an opportunity to talk with about these topics

13. In other words, I’m hoping you’ll end this session with . . .
Greater conceptual clarity
Useful guidelines
Useful strategies and techniques
Useful references/resources
Connections with colleagues 13 13

14. A First “Balcony” Question
Could there be advantages to knowing something about your students’ goals? values? 14 14

15. Page 17 – [ III, IV, V ] Applications Card
Ideas/Techniques
Possible Applications

16. Background Knowledge Probe page 2
1. Of first-year students in 4Y US colleges in Fall 2015, what percentage graduated from:
A. A traditional public school? ___ %
B. A charter/magnet public school? ___%
C. A private religious/parochial school? ___ %
D. A private independent prep school? ___%
E. Were home schooled? ___%

17. Background Knowledge Probe page 2
2. Of those same students, what %-age rated the following as “very important” reasons to go to college:
A. To be able to get a better job? ___ %
B. To learn more about things that interest me? ___%
C. To get training for a specific career? ___ %
D. To gain a general ed. & appreciation for ideas? ___%
E. To be able to make more money? ___%
F. To prepare myself for grad/prof school? ___%
G. To make me a more cultured person ___%

18. Background Knowledge Probe page 2
3. Of those same students, what %-age thought they had a “very good chance of doing the following in college:
A. Socializing w/different racial/ethnic person? ___ %
B. Earning at least a ‘B’ average? ___%
C. Communicating regularly with profs? ___ %
D. Protesting or demonstrating? ___%

19. Background Knowledge Probe page 2
4. About what %-age of the 2015 FYs had:
A. First parent with at least a 4-year degree? ___ %
B. Second parent with at least a 4-year degree? ___%

20. Second “Balcony” Question If you followed directions:
Are you more interested in learning
the answers to questions on p. 2 than you would have been if I’d simply told that information to you? 20 20

21. Background Knowledge Probe page 2
1. Of FY students in 4Y US colleges in 2015, what percentage graduated from:
A. A traditional public school? 74%
B. A charter/magnet public school? 8%
C. A private religious/parochial school? 10%
D. A private independent prep school? 7%
E. Were home schooled? %

22. Background Knowledge Probe page 2
2. Of those same students, what %-age rated the following as “very important” reasons to go to college:
A. To be able to get a better job? %
B. To learn more about things that interest me? 82%
C. To get training for a specific career? %
D. To gain a general ed. & appreciation for ideas? 71%
E. To be able to make more money? 70%
F. To prepare myself for grad/prof school? %
G. To make me a more cultured person %

23. Background Knowledge Probe page 2
3. Of those same students, what %-age thought they had a “very good chance of doing the following in college:
A. Socializing w/different racial/ethnic person? 71%
B. Earning at least a ‘B’ average? 68%
C. Communicating regularly with profs? 44%
D. Protesting or demonstrating? 9%*
* Upon seeing these survey results, college presidents all over the US ordered their Admissions Offices to identify this 9%.

24. Background Knowledge Probe page 2
4. About what %-age of the 2015 FYs had:
A. First parent with at least a 4-year degree? 61%
B. Second parent with at least a 4-year degree? 60%

25. Effective Assessment is often just Gap Analysis
Find the gaps
Mind the gaps
Narrow the gaps 25 25

26. More “Balcony” Questions
Imagine that your first-year students had similar responses:
Could you see any implications for:
The courses you teach?
For your program?
For campus assessment efforts? 26 26

27. Six Dimensions of Higher Learning Outcomes
Page 3 [ II, III, IV, VII]
Six Dimensions of Higher Learning Outcomes
% Your Past? % Their Future?
____ Factual Learning _____
____ Conceptual Learning _____
____ Procedural Learning _____
____ Conditional Learning _____
____ Metacognitive Learning _____
____ Reflective Learning _____
100% % 27

28. Balcony Question Which of those six dimensions needs and deserves the most focus if we aim to foster: Creative thinking? Problem-solving? Innovation? Life-long learning?

29. Metacognition involves . . .
Self-Awareness
Knowing that and when you are thinking
Self-Monitoring
Noticing the quality/intensity of your thinking
Self-Regulation
Directing/Correcting your thinking
Reflection requires metacognition,
but goes beyond it to affect/change values, beliefs, actions and/or habits 29

30. A Reflection-in-Action Exercise
Rate your own learning approach thus far in the session:
1. Disengaged (I don’t see the point, or I already know all this.)
2. Semi-attentive (Checking in and out, per my interests.)
3. Engaged recipient (I’m taking it all in, but quietly.)
4. Active Participant (I’m contributing & cooperating.)
5. Reflective Synthesizer (I’m making connections to prior knowledge/experience and imagining how I might use this in my future work.)
Adapted from: Guskey, T. R. (2000). Evaluating Professional Development. Thousand Oaks, CA: Corwin Press, p. 135. 30

31. can be promoted effectively through High-Impact Practices
HOT HIPs!
Page 4 - middle
Higher Order Thinking
can be promoted effectively through
High-Impact Practices 31 31

32. HIPs – High-Impact Educational Practices
First-Year Seminars and Experiences
Learning Communities
Collaborative Assignments and Projects
Undergraduate Research
Diversity/Global Learning
Service Learning/Community-Based Learning
Clinical Placements/Internships/Co-ops
Capstone Courses and Projects
Writing-Intensive Courses

33. What makes these HIPs so HOT?
High expectations
Explicit direct instruction
Metacognitive scaffolding
Deliberate practice
Effective feedback
Focused collaboration 33 33

34. “It’s not what we do, but what students do that’s the important thing
Biggs, J. & Tang, C. (2007). Teaching for Quality Learning at University, 3rd Edition. Berkshire: McGraw-Hill, p. 19. 34

35. Backward Course (Re)Design – A Simple Self-Assessment 1 Program ILOs 2
Page 5 - Top
Backward Course (Re)Design – A Simple Self-Assessment
Step #
Sequential Steps in an
Ideal Backward Design
Column 2 Observed Sequence
Column 3 Preferred
Sequence 1
Program ILOs 2
Course ILOs 3
Standards 4
Summative Assessment 5
Formative Assessment 6
Learning Activities, etc. 7
Teaching Strategies 8
Course/Program Eval.

36. “You can’t fix by analysis what you bungled by design. ” Light, R
“You can’t fix by analysis what you bungled by design.” Light, R., Singer, J. & Willett, J. (1990). By Design Cambridge, MA: Harvard U. Press.

37. Page 6 – Top – [ II, VII ] The Plus-Minus-Question Mark Technique
Formative and summative assessment
Backward design
Constructive alignment
Gap analysis
Motivated reasoning
Implicit bias
Bus, Parrot & Parking Lot Tests
Cognitive load
Metacognition
The Dance Floor & The Balcony
Deliberate practice
Novice-Expert differences 37

38. Bloom’s Taxonomy (revised)
Anderson & Krathwohl, 2001
Create
Design
Invent
Hypothesise
Developed to explain what terms like “understanding” actually translate to in action (learning).
Assess
Explain
Critique
Decide
Justify
Evaluate
Determine
Induce
Question
Appraise
Analyse
Adapt
Modify
Verify
Demonstrate
Apply
Arrange
Categorise
Convert
Formulate
Generalise
Understand
Identify
Do simple
procedure
Remember

39. Page 7 – bottom [ II, IV, VI ] “Blooming” – Categorizing Questions39 39

40. Page 7 Categorizing Questions by Bloom’s Taxonomy Levels (Blooming)
By systematically varying the elements of the task, we can better assess and promote transfer and deep learning. 40 40

41. Page 8 - top The Bondy Scale41 41

42. Bondy’s Clinical Performance (Skills) Rating Scale
Independent plus capable of assessing, instructing and supporting
[6. EXPERT]
Safe,
Very proficient,
Requires no support or direction
5. INDEPENDENT
Safe,
Proficient,
Requires only occasional support and direction
4. SUPERVISED
Safe,
Mostly proficient,
Requires frequent support and direction
3. ASSISTED
Safe only when supervised, Unskilled,
Requires continuous support and direction
2. MARGINAL
Unsafe,
Unaware &
Unskilled
1. DEPENDENT

43. A Detailed Assessment/Grading Rubric
Page 9 – [ I, IV, V ]
A Detailed Assessment/Grading Rubric
What matters most in this course?
How do you know? 43 43

44. “From the student’s point of view, the assessment is the curriculum.”
Paul Ramsden 44

45. Making Groupwork Work Well A Checklist Approach
Page 10
Making Groupwork Work Well
A Checklist Approach
Checklists are a simple way
to manage cognitive load 45 45

46. Groupwork Feedback Form
Page 11
Groupwork Feedback Form 46 46

47. Group Informal Feedback on Teaching
Page 12 – Top
The G.I.F.T.
Group Informal Feedback on Teaching 47

48. A Course/Teaching Feedback Form
Page 13 – [ I, V, VI ]
A Course/Teaching Feedback Form
What values do our course and teaching
assessment systems convey to students?
To faculty? 48 48

49. “The effectiveness of teaching is best evaluated by what the students do when the teacher is not present.”

50. Page 17 Applications Card – [ III, IV, V]
Interesting Possible
IDEAS/TECHNIQUES APPLICATIONS 50 50

51. The Parking Lot Test 51 51

52. What, Why and How Choose one of your possible applications.
Prepare to answer the three questions below about that specific application:
What is it?
Why do you think it might be useful?
How do you think you might use it? 52 52