1. Fixing a Broken Model How to Use Significance Testing to Assess for Disproportional Representation

2. Presenters Clifford V. Hatt, Ed.D., ABPP Clifford V. Hatt, Ed.D., ABPP Coordinator, Psychological Services, Coordinator, Psychological Services, Virginia Beach City Public Schools, Virginia Jon C. Thompson, Psy.D., HSPP Jon C. Thompson, Psy.D., HSPP Pediatric Neuropsychologist Pediatric Neuropsychologist St. Vincent’s Hospital, Indianapolis, Indiana

3. What’s “Broken”? No way to tell if two different percentages are really different No way to tell if two different percentages are really different Using the “white” student population as the comparison group Using the “white” student population as the comparison group Using “adjustment” amounts to determine if the percentages are meaningful Using “adjustment” amounts to determine if the percentages are meaningful Considering a disproportion instead of incidence of a disorder Considering a disproportion instead of incidence of a disorder

4. The Questions and Issues The What and The How of Disproportionate Representation

5. Proportional Concept Assumes same ethnic proportions will/should be reflected in all educational situations Assumes same ethnic proportions will/should be reflected in all educational situations Special Education Classifications Special Education Classifications Referrals for educational assistance Referrals for educational assistance Discipline referrals Discipline referrals AP classes AP classes Drop out rates Drop out rates Seems simple and self-evident Seems simple and self-evident

6. Disproportion vs. Incidence Do disabilities and illnesses occur in the general population and in subpopulations at the same rate? Do disabilities and illnesses occur in the general population and in subpopulations at the same rate? If the incidence of mental retardation occurs in 1% of the general school age population (based on USDOE data), can we expect it to occur at that same rate in specific school-aged ethnic groups? If the incidence of mental retardation occurs in 1% of the general school age population (based on USDOE data), can we expect it to occur at that same rate in specific school-aged ethnic groups?

7. 2004: % of Students Age 6-21 Served Under IDEA, Part B AllLDSLIMREDOther United States 9.164.241.730.840.731.62 Virginia9.624.121.450.800.782.47

8. Incidence Rates Reported in DSM-IV-TR Learning Disability: Approximately 5% of students in public schools are diagnosed with a learning disability (reading, written language, math) Mental Retardation: Approximately 1% of general population is diagnosed with mental retardation

9. Is Special Education Data Consistent? MRLD United States 2004 IDEA Data 0.84%4.24% Virginia 2004 IDEA Data 0.80%4.12% DSM-IV-TR Reported Incidence Rate 1.0%5.0%

10. Consistency of Comparative Data If Special Education Data is consistent, which incidence rate should be used for comparison? If Special Education Data is consistent, which incidence rate should be used for comparison? United States Data United States Data Virginia Data Virginia Data DSM-IV-TR DSM-IV-TR

11. For every complex problem there is an answer that is clear, simple, and wrong. H. L. Mencken (1880-1956)

12. Problems with the Methods Risk Index Risk Index Number of students from a given ethnic background identified with a disability/Total number of that ethnicity in the general student population Number of students from a given ethnic background identified with a disability/Total number of that ethnicity in the general student population Similar to incidence rate Similar to incidence rate Only descriptive analysis Only descriptive analysis

13. Problems with the Methods Odds Ratio (Relative Risk Index) Odds Ratio (Relative Risk Index) Uses the Risk Index of an ethnic minority as the nominator and the Risk Index of whites as the denominator Uses the Risk Index of an ethnic minority as the nominator and the Risk Index of whites as the denominator Gives simple comparison of groups with whites serving as the control group Gives simple comparison of groups with whites serving as the control group Can determine whether a particular ethnic group will be identified as having a disability more or less often compared to whites Can determine whether a particular ethnic group will be identified as having a disability more or less often compared to whites

14. Problems with the Methods Odds Ratio (Relative Risk Index) Odds Ratio (Relative Risk Index) No way to evaluate for significance No way to evaluate for significance Assumes whites are the majority and are representative of a disability group as a whole Assumes whites are the majority and are representative of a disability group as a whole Assumes all disabilities are equally represented across ethnic groups Assumes all disabilities are equally represented across ethnic groups

15. Problems with the Methods Composition Index Composition Index Calculated by dividing the # of students of a racial or ethnic group in a category by the total # of students in that ethnic or racial group in the total student population. Calculated by dividing the # of students of a racial or ethnic group in a category by the total # of students in that ethnic or racial group in the total student population. Provides composition for a pre-determined population so that when all compositions of the different ethnic groups are added together, total equals 100% Provides composition for a pre-determined population so that when all compositions of the different ethnic groups are added together, total equals 100%

16. Problems with the Methods Composition Index Composition Index Composition of the identified category is then compared to composition of the population as a whole Composition of the identified category is then compared to composition of the population as a whole No way to evaluate for significance No way to evaluate for significance Highly susceptible to interpretation flaws when population under consideration is small Highly susceptible to interpretation flaws when population under consideration is small Assumes all disabilities occur with equal frequency across all populations Assumes all disabilities occur with equal frequency across all populations

17. Arbitrary Numbers It may be recognized that composition ratios are not very useful when comparing subgroups. It may be recognized that composition ratios are not very useful when comparing subgroups. Some type of “statistical” adjustment is often suggested without any clear rationale. Some type of “statistical” adjustment is often suggested without any clear rationale. Some states may use “+/- 20 percentage points” to suggest significant disproportionality (Coutinho & Oswald, 2004). Some states may use “+/- 20 percentage points” to suggest significant disproportionality (Coutinho & Oswald, 2004).

18. A “Fix” for the Problems of a Broken Model What if you could determine if two percentages (proportions) were really different? What if you could determine if two percentages (proportions) were really different? What if you could compare local with state data and state data with federal data using the same method? What if you could compare local with state data and state data with federal data using the same method? What if you could track progress made after interventions were implemented? What if you could track progress made after interventions were implemented?

19. A “Fix” for the Problems of a Broken Model What if you actually saw measurable differences in what you did instead of always never quite measuring up? What if you actually saw measurable differences in what you did instead of always never quite measuring up? What if this all made sense?!!! What if this all made sense?!!!

20. One Solution Determine the significance of the difference between two proportions (percentages are just a special type of proportion) Determine the significance of the difference between two proportions (percentages are just a special type of proportion) Determine if real statistical differences are found Determine if real statistical differences are found Use a z-score as the common measure Use a z-score as the common measure

21. Standard normal distribution -2 -1 0 1 2

22. Within 2* SDs (* really 1.96) -2 -1 0 1 2 95%

23. Formula Key Population Population proportion Samplesize # ID Sp Ed Sampleproportion 1 p1p1p1p1 n1n1n1n1 X1X1X1X1 2 p2p2p2p2 n2n2n2n2 X2X2X2X2

24. Formula for z

25. Formula in Special Ed Terms z = (% min spec ed - % min students)/ sqrt[(x * 1-x)*(1/n1 + 1/n2)] z = (% min spec ed - % min students)/ sqrt[(x * 1-x)*(1/n1 + 1/n2)] x = (# min students + # spec ed min students)/(n1 + n2) x = (# min students + # spec ed min students)/(n1 + n2) n1 = total # minority students n1 = total # minority students n2 = total # spec ed students n2 = total # spec ed students

26. A Simple Way to Do It Excel spreadsheet with the formula imbedded Excel compared to state population Excel spreadsheet with the formula imbedded Excel compared to state population Excel compared to state population Excel compared to state population Enter the number of minority students in special ed and the number of minority students in the local student population Enter the number of minority students in special ed and the number of minority students in the local student population Select a comparison group: state data or national data Select a comparison group: state data or national data Enter the respective information Enter the respective information

27. Results for Virginia Current Method New Method # Districts Demonstrating Disproportionality Amer. Indian 2325 Asian49 Black9647 Hawaiian79 Hispanic372 White5817

28. Implications Different choices in the use of the comparison group make a difference! Different choices in the use of the comparison group make a difference! The method you chose to compare data makes a difference! The method you chose to compare data makes a difference! Disproportionality is a problem regardless of method. Disproportionality is a problem regardless of method. More specific methodology allows for more appropriate, targeted remedies. More specific methodology allows for more appropriate, targeted remedies. Can be easily calculated in an Excel spreadsheet format. Can be easily calculated in an Excel spreadsheet format.

29. Implications Modified z-score method provides a way to determine whether the differences between predicted and obtained proportions are meaningful. Modified z-score method provides a way to determine whether the differences between predicted and obtained proportions are meaningful. More appropriately identifies districts that need to have corrective action. More appropriately identifies districts that need to have corrective action. Also provides a way to determine progress or change following interventions. Also provides a way to determine progress or change following interventions. Accounts for differences in size or number of students under consideration. Accounts for differences in size or number of students under consideration.

30. A Little Comic Wisdom Politics is the art of looking for trouble, finding it everywhere, diagnosing it incorrectly and applying the wrong remedies. Politics is the art of looking for trouble, finding it everywhere, diagnosing it incorrectly and applying the wrong remedies. Groucho Marx (1890-1977)