1. Principles and Strategies of KEYS 2.0 Data Analysis and Interpretation GAE Training January 1009 Jacques Nacson Senior Policy Analyst NEA New Products and Programs jnacson@nea.org

2. Objectives  To examine how KEYS fits within the context of continuous school improvement  To review key statistical terms  To review basic research concepts  To consider basic principles of data analysis  To examine and learn to interpret the KEYS 2.0 school and district reports

3. Problem or Opportunity Identified Data Collection Diagnosis & Refinement of Problem Action Planning Action Taking Implementa tion & Evaluation

4. Statistical Terms  Mean, Median, Mode  90 th Percentile  Standard deviation  Factor analysis  Regression analysis  Correlation analysis  Statistical significance (.05 level,.01 level)

5. Data Collection: “How much information do we need?”  at the MOST --  Is the information collected compelling enough to convince any skeptic?  at the LEAST --  Will the information collected at least create cognitive dissonance for the resistors?

6. Triangulation of Data  Compensates for imperfections of data- gathering instruments.  When multiple measures yield same results, it increases confidence in results.  When multiple measures fail to yield same results, it raises important follow-up questions

7. Data Collection: “What must we keep in mind?” 1.Do the instruments and methods we plan to use measure what we claim they do? (Validity) 2. Do the instruments and methods we plan to use accurately measure the phenomena we are studying? (Reliability)

8. Data Gathering Techniques  Interviews  Checklists  Diaries  Logs  Questionnaires  Audio or video tapes  Photographs  Consultative advice

9. General Principles of Data Interpretation Before Looking at Your KEYS 2.0 Results

10. General Principles to Consider Specifically Related to Your KEYS 2.0 Data

11. Horizontal Axis Measure of quality: 5 point scale Left side Disagree (low value) Right side Agree (high value) Vertical Axis Indicators (groups of questions that measure the same concept) KEY 1. Shared Understanding and Commitment To High Goals Respondents Provide Direct Instruction to Students Understanding the Graph for each KEY

12. School AverageAll Schools Average 90 th Percentile Score Data Points School average (Black) All schools average (Red) 90 th percentile score (Yellow) Length of the horizontal bar (Blue/Purple) 1 standard deviation above and one below the school average (measure of agreement or consensus) Standard Deviation Understanding the Graphs for each KEY The Goals for your school in terms of continuous improvement for each indicator The school average moving continuously toward the right side (agree – high value of quality for that indicator) At the same time, reduce the standard deviation (narrow the length of the horizontal bar, meaning greater agreement among respondents)

13. Hierarchical organization – Keys—Indicators--items Rating - the degree to which the respondents believe that the indicator accurately describes the school: average of all respondents on the questions that make up each indicator  1 = strong belief that indicator does not describe the school  2 = some belief that indicator does not describe the school  3 = neutral  4 = some belief that the indicator describes the school  5 = strong belief that the indicator describes the school KEYS Vocabulary

14. KEYS Vocabulary (Con’d)  Consensus - level of agreement among respondents on the rating for each indicator Low consensus = wide band High consensus = narrow band

15. Demo School Report with Links to Resources

17. Demo School Report with Links to Resources (Drilling Down)

19. Demo School Report with Links to Resources (Drilling Down further)

20. Example of a District Report (Aggregate Scores for Key 6)

21. Example of District Report Distribution of School Scores for Indicator 6.5 (Interventions Example of District Report Distribution of School Scores for Indicator 6.5 (Interventions)

22. Example of a District Report Aggregate Scores for Key 5 (Resources)

23. Example of District Report: Distribution of School Scores for Indicator 5.4 (Safe & Healthy Learning Environment)

24. PROCESSING THE DATA Looking at your groups ’ assigned Key Graph answer these questions: On which indicators is there the greatest agreement (the shorter bar)? On which indicators is there the least agreement (the longer bar)? Why might that variability of perspective exist?

25. PROCESSING THE DATA – Cont ’ d How might you come to greater consensus on this? Which of the KEYS indicators have both a high mean score and also have a shorter bar? (Strengths) Which of the indicators have both a low mean score and a wide degree of variability? (Areas for Improvement) Which items have the highest correlation with student achievement?

26. Analyzing and Interpreting KEYS 2.0 Data - The Logic Sequence 1. Are results surprising or do they confirm what you know or believe? 2. If the results are surprising, what might have caused the difference? 3. How might you go about checking further to determine the validity of the findings? 4. If you determine that the findings are valid and the issue is important and relevant to your particular situation, what might be the reasons for such findings? 5. What is the most likely cause? The underlying or root cause? How did you arrive at this conclusion?

27. Analyzing and Interpreting KEYS 2.0 Data -The Logic Sequence – Cont ’ d 6. What are some possible actions/solutions that you and your colleagues might take to alter or reverse and improve the condition? 7. What is the best possible solution given your particular context or situation? 8. What are the steps that you must plan and take in order to implement the best possible solution? 9. What are the resources and the skills needed to implement the solution successfully? 10. How would you know if your actions/solutions were successful in ameliorating the condition?

28. Focus group Questions: Do the KEYS data support or refute other data previously collected? What professional development opportunities are indicated by the data? What barriers to achieving previously stated goals surfaced during your discussion? What other learning does your focus group wish to share with the larger group?

29. Steps a School Might Take Once KEYS Preliminary Analyses are Completed 1. GAPS: Decide on one indicator or a group of indicators where gap (s) exist. 2. RELEVANCE: Reflect with the “team” on the relevance, importance and priority of the selection. (Get feedback from stakeholders) 3. DATA COLLECTION-VALIDATION: Consider the need to collect additional data to validate the KEYS findings. 4. DATA COLLECTION-DIAGNOSIS AND REFINEMENT: Examine what is the “root” cause of the problem.

30. Steps a School Might Take Once KEYS Preliminary Analyses are Completed 5. THEORY OF ACTION: Identify and select the most appropriate solution for your context. (Get Feedback from stakeholders) 6. ACTION PLANNING: Set SMART goals and develop specific action/project plans. (Get feedback and commitments from stakeholders) 7. IMPLEMENTATION: Action plans must be implemented for improvement to occur. 8. DATA COLLECTION-EVALUATION: Both process and product evaluations are necessary for learning to happen. 9. BACK TO STEP 1: Repeat the cycle