1. Imagine the possibilities Peg Balachowski Everett Community College

2. Using Improvement Science to Improve Student Outcomes
Anthony Bryk is the current president of the Carnegie Foundation for the Advancement of Teaching, where he is leading work on transforming educational research and development, more closely joining researchers and practitioners to improve teaching and learning.
“There is no blueprint. We are inventing this together as we go along. “
Anthony Bryk

3. Thanks to Lawrence Morales, Associate for Improvement Science and Jane Muhich, Managing Director for Community College Programs, Director of Productive Persistence for the use of their work.

4. got persistence?

5. Outline Improvement Science The Model for Improvement
Tools in Improvement Science
Driver Diagrams
PDSA Cycles
Run Charts

6. Carnegie’s Networked Improvement Community
Language &
Literacy
Productive
Persistence
Improvement
Science
Advancing
Teaching
Analytics

7. Carnegie’s Pathways
Addressing the Developmental Math challenge with a Pathways approach:
Create a high quality Instructional System
Attend to Language and Literacy issues
Advance Quality Teaching
Sustain a Networked Improvement Community
Promote Productive Persistence

8. Improvement Science
A field designed to provide a framework for research that is focused on improvement.
Poses questions about a change being tested
Proposes predictions about how the test will go
Uses a testing mechanism to answer questions and verify predictions
Collects data to support conclusions
Analyzes data to inform later iterations of tests

9. Langley, G. J., et al (2009). The improvement guide: a practical approach to enhancing organizational performance (2nd ed.). San Francisco: Jossey-Bass.

10. The Model for Improvement (MFI)
The Aim
The Measures
The “Change Ideas”

11. Three Questions for Guiding Work
What are we trying to accomplish?
How will we know that a change is an improvement?
What changes can we make that will result in an improvement?
One Method for Testing Change
The Plan-Do-Study-Act Cycle (PDSAs)

12. Tools in Improvement Science
1. Driver Diagrams 2. PDSA Cycles 3. Run Charts

13. Productive Persistence
The tenacity plus good strategies that math students need in order to be successful.

14. Tool 1: The Driver Diagram
AIM
What are we trying to do?
Aim
What?
Who?
By When?
Measures:
All three essential questions are on the driver diagram, a clear framework.
MEASURES
How will we know a change is an improvement?

15. How will we know a change is an improvement?
The Driver Diagram
AIM
What are we trying to do?
Driver 1
Driver 2
Driver 3
Primary
Secondary
Driver A
Driver B
Driver C
Driver D
Driver E
Aim
What?
Who?
By When?
Measures:
MEASURES
How will we know a change is an improvement?

16. The Driver Diagram CHANGE What changes can we test? AIM
What are we trying to do?
Driver 1
Driver 2
Driver 3
Primary
Secondary
Driver A
Driver B
Driver C
Driver D
Driver E
Change Ideas
Change Idea
Aim
What?
Who?
By When?
Measures:
MEASURES
How will we know a change is an improvement?

17. Secondary Drivers Skills & Habits Secondary Drivers Mindsets
Primary Drivers 
Secondary Drivers
Aim:
Students continue to put forth effort during challenges and when they do so they use effective strategies.
Skills & Habits 
Secondary Drivers
Mindsets
Secondary Drivers
Value 
Secondary Drivers
Social Ties
Secondary Drivers
We have the five primary drivers of productive persistence
Not meant to exhaustive
High leverage drivers, if we got traction in that area we would make a difference in our desired outcome
Faculty
Secondary Drivers

18. Students believe they are capable of learning math. Mindsets
Primary Drivers 
Aim:
Students continue to put forth effort during challenges and when they do so they use effective strategies.
Students have skills, habits and know-how to succeed in college setting.
Skills and Habits
Students believe they are capable of learning math.
Mindsets
Students believe the course has value.
Value
Students feel socially tied to peers, faculty, and the course.
Social Ties
Faculty and college support students’ skills and mindsets.
Faculty

19. Students believe they are capable of learning math.
Primary Drivers 
Secondary Drivers
Have accurate knowledge about succeeding in the course and navigating The Health Care Data Guide: Learning from Data for Improvement institution.
Aim:
Students continue to put forth effort during challenges and when they do so they use effective strategies.
Students have skills, habits and know-how to succeed in college setting.
Have the know-how and self-discipline to set and prioritize long and short-term goals over short-term desires and distractions
Use learning strategies that are appropriate for the academic challenge they are facing.
Have strategies for regulating anxiety.
See that math isn’t just a set of algorithms to be memorized but a connected set of concepts that can be understood and applied.
Students believe they are capable of learning math.
Believe they can actively grow their math ability with effort, help, and good strategies.
View math success as something “people like them” do, and not something “other people” do.
Students believe the course has value.
Students see how completion of this course is relevant to goals for degree/certificate completion.
Students believe the knowledge from the course is relevant to a personal or socially-valued goal.
Students feel as though they are completing academic tasks for personal reasons.
Students feel socially tied to peers, faculty, and the course.
Students feel that the professor cares that they, personally, succeed in the course and in college.
Possible Measures:
Attendance
Time on task
Strategy use
Help-seeking
Revising work
Challenge-seeking
Students feel comfortable asking questions
Students do not feel stigmatized due to membership in a negatively stereotyped group.
Students do not question whether they belong.
Faculty and college support students’ skills and mindsets.
Faculty believe students can succeed if they develop more productive skills and mindsets.
Faculty know how to promote productive skills and mindsets.
Faculty see helping their students to productively persist as part their role as an instructor.
Faculty integrate PP principles in how they talk to students and in the curriculum they assign.

20. Secondary Drivers Skills & Habits Secondary Drivers Mindsets
Primary Drivers 
Secondary Drivers
Aim:
Students continue to put forth effort during challenges and when they do so they use effective strategies.
Skills & Habits 
Secondary Drivers
Mindsets
Secondary Drivers
Value 
Secondary Drivers
Social Ties
Secondary Drivers
Let’s look specifically at the social ties driver
Faculty
Secondary Drivers

21. Social Ties Students feel the professor cares that they succeed
Primary Driver 
Secondary Drivers
Change Ideas
Students feel the professor cares that they succeed
Social Ties
Students feel comfortable asking questions
Students do not doubt their belonging in the course

22. Primary Driver 
Secondary Drivers
Change Ideas
Students feel the professor cares that they succeed
routines
Use of names routines
Your ideas?
Social Ties
Students feel comfortable asking questions
Students do not doubt their belonging in the course

23. Your ideas? Social Ties Your ideas?
Primary Driver 
Secondary Drivers
Change Ideas
Students feel the professor cares that they succeed
routines
Use of names routines
Your ideas?
Social Ties
Students feel comfortable asking questions
Answering routines
Data collection routine
Your ideas?
Students do not doubt their belonging in the course

24. How do we know these work?
Primary Driver 
Secondary Drivers
Change Ideas
Students feel the professor cares that they succeed
routines
Use of names routines
Your ideas?
Social Ties
Students feel comfortable asking questions
Answering routines
Data collection routine
Your ideas?
Students do not doubt their belonging in the course
Group role routine
Group noticing routine
Your ideas?
CHANGE
How do we know these work?

25. Tool 2: PDSA Cycles
“A way to turn change ideas into action and connect action to learning.”*
A disciplined way to test ideas.
Avoids implementing too soon.
*Langley GL, Nolan KM, Nolan TW, Norman CL, Provost LP. The Improvement Guide: A Practical Approach to Enhancing Organizational Performance (2nd edition). San Francisco: Jossey-Bass Publishers; 2009.

26. What changes can we test?
PDSA Cycles
CHANGE
What changes can we test?
PLAN
What’s the change?
What’s your prediction?
Plan to conduct test DO
Execute test
Collect data,
document observations
STUDY
Compare results to prediction
What did you learn?
ACT
Next steps: Adapt, adopt, abandon
“A way to turn change ideas into action and connect action to learning.”

27. PDSA Cycles in Productive Persistence
Aim: To measure student sense of belonging, develop an efficient way to gather data on the number of students asking questions
Cycle 1: Feedback from ONE student.
Cycle 2: Test with ONE student in class.
Cycle 3: Test with MORE students in class.
Cycles 4-6: Further testing with more students, more than one student at a time, and regular implementation.
Cycle 7: Inform whole class more fully.
Next Step?

28. References
Provost, L. P., & Murray, S. K. (2011). The health care data guide learning from data for improvement. San Francisco, CA: Jossey-Bass.

29. Tool 3: Run Charts
A graphical display of data to allow you to track outcomes over time.

30. Run Chart Scenario
Around the 4th week of the term, an instructor is noticing a lot of students are missing class. So she brainstorms and eventually has an idea for how to reduce absences. Her plan is to implement her idea in Week 8 of the semester and then compare data from Week 4 and Week 11 of the semester to see if her idea worked.
Here are her data….
Week 4
Data
Week 8
Implement
Week 11
Data

31. Did this change lead to an improvement?
Questions to Ponder:
Did this change lead to an improvement?
What questions do you have about the data?
What other factors might be at play?
Week 8
Implementation

32. Group Activity
1) Look at the following 5 Run Charts: Note: Each one has the same data for Weeks 4 & 11 as the bar chart. 2) Decide what conclusions you can make for each case: Did the change lead to improvement? Why or why not?

33. Case 1 Conclusion? Week 4 Data Week 8 Implement Week 11
The change appears to have had no effect. Improvement was happening steadily and the data reflect this.
Conclusion?

34. Case 2
Week 4 appears to be an outlier, otherwise there is normal variation.
Conclusion?

35. Case 3
The improvement appears to have taken place in Week 5. What happened? Change in Week 8 did not lead to improvement.
Conclusion?

36. Case 4
The change appears to have led to an improvement, but after Week 11 the improvement seems to have faded.
Conclusion?

37. Case 5
The change led to an improvement that was sustained but below the median.
Conclusion?

38. How do we know there’s improvement?
In all of these Cases, we need to consider the question:
“When do the data give us a signal that we have improved on our outcome measure?”
Signal is detected through use of one of 4 basic rules. If you have a signal, tells you you have change not attributed to normal variation.
These 6 different cases show that you can have the same two data points for week 4 and 11, but they may represent completely different stories.
With a run chart, we are able to more accurately whether a change leads to sustained improvement. Data in a bar graph (as in slide 2) could appear to show improvement, but they do not account for the variable possibilities demonstrated through these run chart cases.

39. Does this count as an improvement?
Case 5 Revisited
How do we know that the pattern in the data can be attributed to the change and confirm that it is an improvement?
Held their gains for a while, but is this enough data points to make the conclusion?
Does this count as an improvement?

40. Run Chart Rules for Signals
Shift: Six or more consecutive points all above or all below the median. (Skip points on the median.) Trend: Five or more consecutive points all going up (increasing) or all going down (decreasing). Astronomical Point: An obvious value different than others in the data set.

41. Example 1: Are there signals?
Note: 3 of 4 of the following examples contain real data and the y-axis units have changed.

42. Example 1: Analysis Shifts? Trends? Astronomical Points? None
No Signals 
There is no signals of improvement:
The number of runs (6) is within the limits for charts (13 points not on the median has a range of 4 to 11 runs). It is thus considered normal variation around the median
There are no shifts, trends, or astronomical points

43. Example 2

44. Example 2
*First two weeks: Trend downwards from Jan 1 to just past Jan 8, 5 consecutive points decreasing (+1 point which is same as previous data point)
Middle of the month (1/8 to after 1/15): shift with 6 consecutive points below the median line
Through rest of January: Trend upwards, 5 consecutive points increasing

45. Example 3
Median = 0.83
The shift here is later in the data set. In the context of a community college classroom, this might be a usual pattern faculty sense in their classes, as this occurs well enough into the term that students may be tired and attendance drops off. According to the data alone though, the shift indicates that this is a non-random pattern. Having actual data helps back up these intuitions on attendance patterns, and help us think about how you might reverse that tendency and what things you could try.

46. Example 3
Median = 0.83
The shift here is later in the data set. In the context of a community college classroom, this might be a usual pattern faculty sense in their classes, as this occurs well enough into the term that students may be tired and attendance drops off. According to the data alone though, the shift indicates that this is a non-random pattern. Having actual data helps back up these intuitions on attendance patterns, and help us think about how you might reverse that tendency and what things you could try.

47. Example 4
Median = 0.78

48. Example 4
Median = 0.78
In Faculty E’s run chart, there is a shift in the most recent data from10/1 until 10/8. We see that the shift happens after exam, which might lead you to think about possible changes to test before/after the next exam to address a possible drop in attendance. If you tested a change and continued to collect this data, you can compare attendance around that exam with the one on the run chart here.
Even if you are not collecting data, you would probably sense this lower attendance in your class. When you have actual data to back this up, it allows you to think about what to do next.

49. Reflections from users
“From hearing the group discussions in the last couple modules—they were teaching themselves. The amount that I had to come to the group[s] and make sure they were on task went down. They were becoming self-learners…. It was wonderful to see.”
- Productive Persistence Faculty
“It helped me focus on what was going on with the students, particularly on who was engaged in class. I enjoyed doing the Improvement work and I feel like it really helped me focus on teaching in a different way than I did before.”
Another possible quote
“Writing up the [PDSA] cycles was helpful for me. I reflected more deeply on what I was trying to do and it helped with planning for how I would modify it, what I would expect out of it and what I would hope to see,” (aaron)

50. Thanks for your time! Peg Balachowski mbalachowski@everettcc.edu
Questions?
Thanks for your time!
Peg Balachowski