1. Answering the Questions that Count
PCG Education
1/28/2020
Answering the Questions that Count
Elementary Schools
Advancing Improvement in Education Annual Conference September 24, 2013

2. Introduction SUPPORTING DOING 2 Key Questions:
How do you increase the use of data within your school?
How do you use data to improve student outcomes?
SUPPORTING
DOING

3. Our Agenda Topic Goals Introduction
PCG Education
1/28/2020
Our Agenda
Topic
Goals
Introduction
Introduction to the Data Use Theory of Action
Using Questions to Initiate Inquiry
Learn a protocol to identify pressing questions to focus inquiry
Analyzing Data
Learn a protocol for Collaborative Data Analysis to refine questions and sharpen focus
Engaging Others
Learn how data can be a vehicle to engage others in inquiry and root cause analysis
Prepare a Data Overview to Take Data-Driven Action
Apply tools and protocols to implement an inquiry driven engagement effort that will result in action

4. Data Use Theory of Action
Increased Student Achievement
Focused
Results
Policy
Programs
School Improvement Planning, PD, Time
Program Evaluation, Curriculum Gaps
Data-Driven
Actions
Practice
Placement
Sharing Data, Observing Practice
Differentiating Instruc-tion, Flexible Grouping
Quality
Capacity
Culture
Conditions for Data Use
Accurate
Timely
Relevant
Complete
Data Integration
Analysis Tools
Skills
Structures
Commitment
Beliefs
Collaboration
Leadership
© 2010 Public Consulting Group. Used with permission.

5. Assessing Your Data Quality, Capacity, and Culture
Using the Data Quality, Capacity, and Culture Self Assessment, rate each area with a check mark in the red, yellow, blue, or green column. Red means you feel your campus is seriously lacking in this area. Green means you feel your campus is doing well in this area. Think about each statement in the context of your whole school practice.
As a group, use the discussion questions associated with the QC2 Self Assessment in your Participant’s Packet on page 4 to share your observations with each other.

6. Taking Note 1 Strengths & Limiting Factors
Take a moment to make a note of your top strength(s) and your most limiting factor(s) in each area:
Strengths
Limiting Factors
Data Quality
Data Capacity
Data Culture
With whom at my school/board do I need to share this information?

7. Using Questions to Initiate Inquiry

8. TAIS Continuous Improvement Process
PCG Education
1/28/2020
TAIS Continuous Improvement Process
Data Analysis
Needs Assessment
Improvement Plan
Implement & Monitor

9. Equity, Accountability, and Continuous Improvement
Questions About…
Context
Student Characteristics
Special Education
English Language Learners
Transfers
Educational Programs
Context
Achievement
Engage-ment
Achievement
Assessment Performance Within a Subject
Assessment Performance Across Subjects
Course Grades
Credit Attainment
Other Assessment Performance
Equity, Accountability, and Continuous Improvement
Engagement
Attendance
Graduation
Retention
Persistence

10. When Data Analysis Goes Bad…
Infer
“Something is wrong with our reading curriculum. 70% of our 6th graders scored Below Proficient on our most recent assessment.”
Assume
“We obviously don’t have the right focus in our grade 6 reading curriculum.”
Speculate
“Wouldn’t we get better results if we revamped our grade 6 reading curriculum?”
Act
impulsively
“Let’s revamp our grade 6 reading curriculum.”

11. Types of Questions Throughout the Inquiry Process
PCG Education
1/28/2020
Types of Questions Throughout the Inquiry Process
Pose a
Question About
Your Students
Take Action
Determine Student
Learning Problem
Determine Appropriate Action
to Fix the Root Cause
Determine Root Cause
(Problem of Practice)
Discovery:
What’s happening?
Doing:
What do we do about it?
How did our students do in…?
What percentage of students…?
Which students…?
What groups of students…?
In which subject areas…?
How do we fix…?
What do we do about…?
Should we…?
Do we need…?
Diagnosis:
Why is it happening?
Why did ours students…?
How do our teachers…?

12. Asking the Right Questions Protocol
PCG Education
Asking the Right Questions Protocol
1/28/2020
Purpose
To formulate a few specific questions from a broad starting point.
Total Time
About 30 minutes
Step
Description
Time
Identify
Write an issue in your school or family of schools you wish to address on the top of a piece of chart paper. It can be formulated as a statement or question. Your issue/question should be related to student outcomes.
5 min
Brainstorm
As a group, brainstorm questions that stem from the original question/statement. Write the questions as stated on the chart paper. All items must be phrased as questions. Your questions should be DISCOVERY questions.
10 min
Prioritize
From this group of questions, identify the top three. Out of these three, identify the top priority question.
2 min
Branch
Write the top priority question on the top of a new piece of chart paper. Brainstorm a new set of questions about this question, following the same rules as before. Remember: DISCOVERY questions.
From this list of questions, again identify your top three questions. On a new piece of chart paper, write your initial issue/question, then write these top three questions and post on the wall.

13. Taking Note 2 Asking the Right Questions
Our top three questions are:

14. PCG Education
1/28/2020
Analyzing Data

15. Analyzing Data Data Analysis Needs Assessment Implement & Monitor
PCG Education
1/28/2020
Analyzing Data
Data Analysis
Needs Assessment
Implement & Monitor
Improvement Plan

16. PCG Education
1/28/2020
Preparing to Analyze
Select the question you will use to start your inquiry.
Identify the data that will be needed to answer the question.
Where does the data you need reside?
Which report will you need?
Turn to page 15 in your handbook to get started.

17. You have 45 minutes to work
PCG Education
1/28/2020
Data Analysis
You have 45 minutes to work

18. Taking Note 3 Data Analysis
What possible conclusions did you draw from your data?
What new questions did you ask?
Who in your campus would you bring in to continue this analysis?

19. PCG Education
1/28/2020
Engaging Others

20. Data Overview Defined da·ta o·ver·view [dā'-tə, dä'-tə ō'vər-vyōō']
PCG Education
1/28/2020
Data Overview Defined
da·ta o·ver·view [dā'-tə, dä'-tə ō'vər-vyōō']
Etymology: City, Boudett, & Murnane
Date: 2005
-noun
1 : collection of graphical displays that highlight important patterns
2 : presentation that tells a story using data
3 : a meeting to engage faculty in constructive conversations about the meaning of data
-usage
“Golly, that Data Overview meeting was stimulating and fun, and I just can’t wait to dig deeper into the data to learn even more about our students!”

21. When have you shared data before?
PCG Education
1/28/2020
When have you shared data before?
Pair/Share:
What are some of the highlights (and lowlights) of times when you have shared data at faculty meetings?

22. What are the Roles in a Data Overview?
PCG Education
1/28/2020
What are the Roles in a Data Overview?
Presenter(s) Principal, Department Head, Data Team Leader, etc. Responsible for presentation of the Data Overview.
Data Team Members
Work with small groups during brainstorm and analysis work as well as prioritization of questions.
Data Guru
Helps prepare for meeting—analyzes data, builds data tables, builds data displays.
Participants
Faculty and staff. Participate in brainstorming. Help identify questions for Digging into Data step.

23. What Happens in a Data Overview?
PCG Education
1/28/2020
What Happens in a Data Overview?
Set the Meeting Agenda
Set Next Steps for Digging into the Data
Why are We Here?
State the Purpose
What are Our Next Steps?
Prioritize Questions
What is the Story?
What are Our Most Important Questions?
Present the Data
Record Questions
What do You Observe?
What Questions do the Data Elicit?
Analyze and Brainstorm

24. Storyboard your Meeting
PCG Education
1/28/2020
Storyboard your Meeting
Used with permission from Education Direction, LLC

25. Hidden Valley School District
Data Overview Presentation, September 2009
Presented by the District Data Team

26. Agenda Purpose of this presentation
Presentation of relevant data displays
Collaboratively brainstorm to:
Identify and prioritize a problem evident in the data
Formulate hypotheses to explain the problem
Generate clarifying questions to direct further inquiry
Identify additional data needed and potential data sources
Discuss next steps

27. The Issues
The academic performance of English language learners (ELLs) in the Hidden Valley School District is lower than the statewide population of ELLs at several grade levels on 2009 state English Language Arts (ELA) tests.
The academic performance of ELLs in the district lags behind their native English-speaking peers.
The academic performance of ELLs in the district falls below established performance and improvement targets.

28. Purpose
To begin the process of collaborative inquiry that will address the focusing questions:
What does the performance of ELLs in the district look like over time on recent ELA tests?
How does the performance of ELLs in the district compare to that of ELLs state-wide on the 2009 ELA test?
Why is the performance of ELLs in some grades closer to the state average than others on the 2009 ELA test?
What is the performance of ELLs in targeted grades in district schools on the 2009 ELA test?
How does the performance of ELLs in the district compare to the state on the 2009 MEPA test?

29. Tests Administered as values
2006 
2007 
2008 
2009 
2006–2009 Change 3
A/P
25.3%
23.3%
31.5%
27.1%
1.8% NI
51.6%
47.8%
41.6%
38.6%
-13.0%
W/F
23.2%
28.9%
27.0%
34.3%
11.1% 4
3.7%
11.5%
11.8%
14.6%
10.9%
53.2%
39.7%
38.2%
43.9%
-9.3%
43.1%
48.7%
50.0%
41.5%
-1.6% 5
2.6%
8.0%
1.7%
9.6% 7%
40.2%
27.6%
14.8%
33.7%
-6.5%
57.3%
64.4%
83.5%
56.6%
-0.7% 6
5.0%
6.3%
15.9%
6.5%
26.7%
33.3%
36.5%
32.8%
6.1%
68.3%
60.3%
47.6%
55.7%
-12.6% 7
10.0%
1.5%
9.4%
9.7%
-0.3%
20.0%
27.3%
25.0%
29.2%
9.2%
70.0%
71.2%
65.6%
61.1%
-8.9% 8
4.5%
7.8%
7.4%
3.3%
22.7%
18.9%
20.7%
-2.0%
72.7%
73.3%
67.4%
71.6%
-1.1% 10
16.3%
4.7%
5.9%
3.0%
-13.3%
46.5%
31.3%
35.3%
19.7%
-26.8%
37.2%
64.1%
58.8%
77.3%
40.1%
What does the performance of ELLs in the district look like over time on ELA tests?

30. Tests Administered as values
District
State
Difference
Tests Administered as values
2009  3
(N = 70)
A/P
27.1%
26.2%
0.9% NI
38.6%
40.7%
-2.1%
W/F
34.3%
33.0%
1.3% 4
(N = 82)
14.6%
18.0%
-3.4%
43.9%
46.8%
-2.9%
41.5%
35.2%
6.3% 5
(N = 83)
9.6%
16.3%
-6.7%
33.7%
40.5%
-6.8%
56.6%
43.2%
13.4% 6
(N = 61)
11.5%
18.7%
-7.2%
32.8%
-1.5%
55.7%
47.0%
8.7% 7
(N = 72)
9.7%
13.1%
29.2%
34.7%
-5.5%
61.1%
52.2%
8.9% 8
(N = 116)
7.8%
12.9%
-5.1%
20.7%
27.0%
-6.3%
71.6%
60.1% 10
(N = 66)
3.0%
21.8%
-18.8%
19.7%
36.8%
-17.1%
77.3%
41.4%
35.9%
How does the performance of ELLs in the district compare to that of ELLs state-wide on the 2009 ELA test? 30

31. Tests Administered as values
District
State
Difference
Tests Administered as values
2009  3
(N = 70)
A/P
27.1%
26.2%
0.9% NI
38.6%
40.7%
-2.1%
W/F
34.3%
33.0%
1.3% 4
(N = 82)
14.6%
18.0%
-3.4%
43.9%
46.8%
-2.9%
41.5%
35.2%
6.3% 5
(N = 83)
9.6%
16.3%
-6.7%
33.7%
40.5%
-6.8%
56.6%
43.2%
13.4% 6
(N = 61)
11.5%
18.7%
-7.2%
32.8%
-1.5%
55.7%
47.0%
8.7% 7
(N = 72)
9.7%
13.1%
29.2%
34.7%
-5.5%
61.1%
52.2%
8.9% 8
(N = 116)
7.8%
12.9%
-5.1%
20.7%
27.0%
-6.3%
71.6%
60.1% 10
(N = 66)
3.0%
21.8%
-18.8%
19.7%
36.8%
-17.1%
77.3%
41.4%
35.9%
Why is the performance of ELLs in some grades closer to the state average than others on the 2009 ELA test? 31

32. Tests Administered as Values
What is the performance of ELLs in targeted grades in district schools on the 2009 ELA test?*
Canyon ES
Twin Peaks ES
Golden Meadow ES
Hidden River ES
Mesa ES
Tests Administered as Values
2009  3 N 4 21 32
A/P
100.0%
23.8%
21.9% NI
0.0%
56.3%
W/F
52.4% 6 2 49
16.7%
18.4%
83.3%
34.7%
46.9% 5 15 42
2.4%
60.0%
31.0%
40.0%
66.7% 32

33. Tests Administered as Values
What is the performance of ELLs in targeted grades in district schools on the 2009 ELA test?*
Mesa MS
Canyon MS
Tests Administered as Values
2009  6 N 53 8
A/P
13.2% NI
34.0%
25.0%
W/F
52.8%
75.0% 7 70 2
10.0%
30.0%
60.0%
100.0% 95 18
2.1%
27.8%
20.0%
22.2%
77.9%
50.0% 33

34. Tests Administered as values
What is the performance of ELLs in targeted grades in district schools on the 2009 ELA test?*
Hidden Valley HS
Tests Administered as values
2009  10 N 66
A/P
3.0% NI
19.7%
W/F
77.3% 34

35. Percentage of Students at Each Performance Level
Spring 2009 MEPA Results:
Percentage of Students at Each Performance Level
by Years of Enrollment in the State and Hidden Valley

36. Brainstorm Groups Group A Group B Group C Group D Jerry Jill Colleen
Kerry
Maria
Saleem
Michelle
Alex
Jose
Jeff
Clarence
Jordan
Roger
Patricia
Cindy
Ann

37. Brainstorm!
Purpose
To collaboratively investigate the focusing question.
What you will do?
Using the Data Overview Brainstorming Protocol, you will make observations, identify problems, form hypotheses, and craft clarifying questions related to the focusing question.
Estimated Time
About 30 minutes.

38. Next Steps What additional data do we need to collect?
Who will collect the data?
How will the data be collected?
When will we have collected the data?
Who will analyze the data?
What materials should we have for the next meeting?
Next meeting date and time?

39. Activity—Preparing a Data Overview
PCG Education
Activity—Preparing a Data Overview
1/28/2020
Preparing Your Data Overview
Turn to page 22 in your Participant’s Packet and review the steps to create your data overview.
Use the questions you generated earlier to identify the questions and data that you will present in your data overview.
Use the storyboard template in your Participant's Packet to plan the presentation that will be used in your data overview meeting.

40. Wrapping Up

41. Today we… Topic Goals Introduction
PCG Education
1/28/2020
Today we…
Topic
Goals
Introduction
Introduction to the Data Use Theory of Action
Using Questions to Initiate Inquiry
Learn a protocol to identify pressing questions to focus inquiry
Analyzing Data
Learn a protocol for Collaborative Data Analysis to refine questions and sharpen focus
Engaging Others
Learn how data can be a vehicle to engage others in inquiry and root cause analysis
Prepare a Data Overview to Take Data-Driven Action
Apply tools and protocols to implement an inquiry driven engagement effort that will result in action

42. Taking Note 4 —Next Steps
What are some “big ideas” that you want to make sure to remember from today? With whom in your school or district do you need to share this with in order to take your next steps toward greater levels of data use?
Big Ideas
People to Share With 1 2 3

43. Thank You Robb Geier Linda Wurzbach
PCG Education
1/28/2020
Thank You
Robb Geier
PCG Education Director of Data Services
Linda Wurzbach
Resources for Learning President , ext. 1

44. Public Consulting Group, Inc.
PCG Education
1/28/2020
Public Consulting Group, Inc.
148 State Street, Tenth Floor, Boston, Massachusetts 02109
(617) ,