1. 1 Assessment, Accountability and Adaptation April 13, 2006 copies of this presentation can be found at www.business.duq.edu/faculty/davies

2. 2 Education is a product. Economics provides tools for adequately addressing assessment, accountability, and adaptation, but to use the tools, we must first admit that education, while occupying a place of great importance, is fundamentally a product.

3. 3 How does the market judge products, hold producers accountable, and adapt to consumers’ needs? Assessment AccountabilityAdaptation

4. 4 The cycle begins with the consumer who develops a mental list of important attributes and a acceptable tradeoffs. Together, these form the basis of the consumer’s assessment of the product. Assessment AccountabilityAdaptation Attributes Tradeoffs

5. 5 Consider grocery stores… The Consumer Has a List of Important Attributes Store is located in a safe and convenient location. The produce is fresh. There is a large selection of brands. The store has a deli. Prices are reasonable. The Consumer Is Aware of Acceptable Tradeoffs Convenient location is more important than fresh produce. Lower price is more important than having a deli.

6. 6 The consumer assesses the performance of the store by comparing the store’s attributes to the consumer’s list of important attributes and acceptable tradeoffs. Assessment Attributes Tradeoffs

7. 7 The consumer holds the store accountable via repeat purchase. If the store’s attributes are acceptable to consumers, the store’s sales rise. If not, the store’s sales fall. Assessment Accountability Attributes Tradeoffs

8. 8 If sales fall, the store will mimic and/or improve upon attributes of stores with rising sales. If the store’s sales rise, competing stores will mimic and/or improve upon this store’s sales. This is adaptation. Assessment AccountabilityAdaptation Attributes Tradeoffs

9. 9 The consumer now re-assesses the store in light of the store’s new attributes and any changes in the consumer’s attributes / tradeoffs. This is outcomes assessment. Assessment AccountabilityAdaptation Attributes Tradeoffs

10. 10 In the market place, what might cause the AAA cycle to break down? Attributes Tradeoffs Assessment Adaptation Accountability

11. 11 Break in the Cycle #1: Consumer is unidentified Attributes Tradeoffs Assessment AccountabilityAdaptation If the consumer is unidentified, questions arise: 1.What attributes are important? 2.What tradeoffs are acceptable? With an unidentified consumer:  Appropriate assessment becomes unclear.  Source of accountability becomes unclear.  Appropriate adaptation becomes unclear.

12. 12 Assessment AccountabilityAdaptation Break in the Cycle #2: Accountability not Possible 1.Repeat purchase – the consumer must have a need to purchase again so as to signal pleasure; 2.Mobility – the consumer must be able to purchase elsewhere so as to signal displeasure; 3.Short-run assessment – the consumer must be able to assess product quality prior to the repeat purchase decision. If accountability is not possible  Appropriate adaptation becomes unclear.  Incentive for adaptation disappears. Possibility of accountability requires: Attributes Tradeoffs

13. 13 How does the AAA cycle play out for primary and secondary education? Assessment Adaptation Accountability Attributes Tradeoffs First: Can we identify the consumer? Second: On what is assessment based? Third: Is accountability possible? Fourth: What adaptations result?

14. 14 First: Can We Identify the Consumer? Student? Attributes: friends, sports, extracurricular activities. Parents? Attributes: safety, education, sports, future opportunities. Colleges? Attributes: critical thinking skills, math skills, communication skills, personality, drive. Employers? Attributes: work ethic, communication skills, ability to function as part of a team. Society? Attributes: likelihood of incarceration, likelihood of becoming a welfare recipient.

15. 15 Second: On What is Assessment Based? If Consumer = Student Assessment based on the student’s social environment. If Consumer = Parents Assessment based on parents’ satisfaction with perceived educational environment. If Consumer = Colleges Assessment based on students’ college preparation. If Consumer = Employers Assessment based on students’ job skills. If Consumer = Society Assessment based on crime and poverty measures.

16. 16 Only three definitions of “consumer” permit accountability. Third: Is Accountability Possible?

17. 17 How we define “the consumer” ultimately determines how education will adapt. Definition of “consumer” determines attributes and tradeoffs, which determine assessment, which determines accountability, which determines adaptation. Fourth: What Adaptations Result?

18. 18 Assessment AccountabilityAdaptation Assessment is college preparation. Accountability is imposed via acceptance rates. Schools adapt by making students more attractive to colleges so as to boost acceptance rate. Fourth: What Adaptations Result when Consumer = Colleges?

19. 19 Assessment AccountabilityAdaptation Assessment is college preparation. Accountability is imposed via acceptance rates. Schools adapt by making students more attractive to colleges so as to boost acceptance rate. Unintended Consequence Schools will encourage students who are technically, rather than academically, talented to go to college rather than to prepare for the job market. Fourth: What Adaptations Result when Consumer = Colleges?

20. 20 Assessment AccountabilityAdaptation Assessment is job skills. Accountability is imposed via employment and wage rates. Schools adapt by making students more attractive to employers so as to boost employment and wage rates. Fourth: What Adaptations Result when Consumer = Employers?

21. 21 Assessment AccountabilityAdaptation Assessment is job skills. Accountability is imposed via employment and wage rates. Schools adapt by making students more attractive to employers so as to boost employment and wage rates. Unintended Consequence Schools will encourage students who are academically, rather than technically, talented to prepare for the job market rather than to go to college. Fourth: What Adaptations Result when Consumer = Employers?

22. 22 Assessment AccountabilityAdaptation Assessment is educational environment. Accountability is imposed via changing schools. Schools adapt by making themselves more attractive to parents so as to boost enrollment and retention. Fourth: What Adaptations Result when Consumer = Parents?

23. 23 Assessment AccountabilityAdaptation Assessment is educational environment. Accountability is imposed via changing schools. Schools adapt by making themselves more attractive to parents so as to boost enrollment and retention. Unintended Consequence Schools will develop educational environments that conform to parents’ perceptions of quality education. Fourth: What Adaptations Result when Consumer = Parents?

24. 24 Let’s assume that: 1.We can skip the step of identifying the consumer, and 2.Standardized Testing is the appropriate assessment tool,  What is the impact of assessment on adaptation? Can We Assess Education Without Identifying the Consumer?

25. 25 Assessment AccountabilityAdaptation Assessment is Standardized Testing. Accountability is imposed via funding (“high-stakes testing”). Schools adapt by making students better able to pass the test so as to boost funding. Fourth: What Adaptations Result when Consumer = Undefined?

26. 26 Assessment AccountabilityAdaptation Assessment is Standardized Testing. Schools adapt by making students better able to pass the test so as to boost funding. Unintended Consequence Schools will prepare students to pass the Standardized Test (“teaching to the test”). Accountability is imposed via funding (“high-stakes testing”). Fourth: What Adaptations Result when Consumer = Undefined?

27. 27 Let’s assume that: 1.We can skip the step of identifying the consumer, and 2.Standardized Testing is the appropriate assessment tool,  What attributes do school districts control? Fourth: What Adaptations Result when Consumer = Undefined?

28. 28 An oft-cited important attribute is the Student-Teacher ratio This strategic plan stresses the need for strategies aimed at keeping more teachers in the classroom. – Strategic Plan, Texas Higher Education Coordinating Board, 2002. Teacher recruitment is the key to the drive to reduce class size. – Congresswoman Eddie Bernice Johnson, (D-TX), 2005. Utah will undertake several initiatives aimed at keeping more teachers in the classroom. – Utah Governor John Huntsman (R), 2005. Growing numbers of students in our nation’s schools…mean that our need for teachers increases each year. – Meeting the Challenges of Recruitment and Retention, NEA, 2005.

29. 29 Student-Teacher Ratio has no apparent impact on NAEP scores. Source: National Center for Education Statistics, U.S. Department of Education Each dot represents one state. Reported scores are sums of state averages for math and reading.

30. 30 School size has no apparent impact on NAEP scores. Source: National Center for Education Statistics, U.S. Department of Education

31. 31 Spending per Pupil has no apparent impact on NAEP scores. Source: National Center for Education Statistics, U.S. Department of Education

32. 32 Spending per Pupil has no apparent impact on NAEP scores. Source: National Center for Education Statistics, U.S. Department of Education

33. 33 Source: National Center for Education Statistics, U.S. Department of Education Spending per Pupil has no apparent impact on NAEP scores.

34. 34 Median Family Income has no apparent impact on NAEP scores. Source: National Center for Education Statistics, U.S. Department of Education

35. 35 Perhaps standardized tests are inadequate for assessing the quality of education. What about dropout rate? This measure comes closer to reflecting parent/student choice-to-purchase.

36. 36 Student-Teacher Ratio has no apparent impact on dropout rate. Source: National Center for Education Statistics, U.S. Department of Education

37. 37 School Size has no apparent impact on dropout rate. Source: National Center for Education Statistics, U.S. Department of Education

38. 38 Spending per Pupil has no apparent impact on dropout rate. Source: National Center for Education Statistics, U.S. Department of Education

39. 39 Source: National Center for Education Statistics, U.S. Department of Education Spending per Pupil has no apparent impact on dropout rate.

40. 40 Median Family Income has no apparent impact on dropout rate. Source: National Center for Education Statistics, U.S. Department of Education

41. 41 Does anything predict Standardized Test performance or Dropout Rate?

42. 42 For 8 th grade, past NAEP scores predict future NAEP scores. Source: National Center for Education Statistics, U.S. Department of Education

43. 43 Source: National Center for Education Statistics, U.S. Department of Education For 4 th grade, past NAEP scores predict future NAEP scores.

44. 44 4 th grade NAEP scores predict 8 th grade NAEP scores. Source: National Center for Education Statistics, U.S. Department of Education

45. 45 Conclusion Attributes that conventional wisdom suggests are appropriate have little or no impact on Standardized Test Results or Dropout Rates. But, school performance is not random – states that perform well in the future are those that have performed well in the past. Supposition As most of the attributes employed by school districts are systemic, evidence suggests that educational quality may be better influenced via individual attributes.

46. 46 Assessment, Accountability and Adaptation April 13, 2006 copies of this presentation can be found at www.business.duq.edu/faculty/davies