1. Scientific and Economic Value of the Metrological Point of View William P. Fisher, Jr. University of California, Berkeley Pacific Rim Objective Measurement Symposium 6-9 August 2012 Jiaxing, China

2. Overview Some basic economic principles shared by science and commerce Three points of view on measurement in education The kinds of markets created by the three approaches to measurement A plan for the future

3. Economic Principles Shared By Science and Commerce Separate local economies – Different currencies – Different weights and measures – Higher costs of exchange – Less efficient, harder to compare values Unified regional and global economies – Same currency – Same weights and measures – Lower costs of exchange – More efficient, easier to compare values

4. Example 1 of Scientific Market Biochemistry – Equipment calibrated in universal reference standard metrics – Test results always reported in common units – Measures available on the spot – Easy to coordinate research across labs – Result: SARS virus sequenced in weeks by network of labs, vaccine successfully synthesized

5. Example 2 of a Scientific Market Custom tailored suits – Tape measures calibrated in universal reference standard metric – Results always reported in common units – Measures available on the spot – Easy to coordinate across tailors – Result: measures can be sent around the world and a well fitting suit obtained with little trouble

6. Example 3 of Scientific Market Education – Tests typically not calibrated at all – If they are calibrated, they are in local units – Test results are usually reported in unique units – Measures available only after costly data analysis – Very difficult to compare outcomes outside of special contexts – Result: Improvement efforts repeatedly fail, quality uncontrolled, costs spiral higher

7. The Ideal Efficient Market Cost of estimating value is very low Cost of comparing value for price is very low Supply and demand easily match up Low value for price: cannot compete High value for price: rewarded Improved value easy to recognize Improved value pushes out old value

8. Readily available high quality information on product or service High CostLow Cost Hard to match supply and demand Easy to match supply and demand Market Efficiency Basic Economics Hard to know how to improve quality Easy to know how to improve quality Quality Improvement Hard for customers to find quality Easy for customers to find quality Customer Quality-Seeking

9. Three Points of View on How to Present Information on Educational Outcomes True Score Theory Measurement Theory Metrological Traceability

10. True Score Theory Disconnected Scores and Tests School 1 – Student A has a score of 22 on a reading test. – This classroom averages a score of 24. School 2 – Student Z has a score of 18 on a reading test. – This classroom averages a score of 26.

11. True Score Theory Disconnected Scores and Tests Who has more reading ability, A or Z? ?? What can one student read that the other cannot? ?? Which classroom reads better on average? ?? Which student is more on track for college readiness? ??

12. True Score Theory Disconnected Scores and Tests School 1 – Student A’s reading scores on 2 tests are 22 & 32. – The classroom average score goes from 24 to 30. School 2 – Student Z’s reading scores on 2 tests are 18 & 32. – The classroom average score goes from 26 to 40.

13. True Score Theory Disconnected Scores and Tests Who gained more in reading ability, A or Z? ?? What new texts can A and Z read? ?? Which classroom improves more? ?? Are both students on track for college readiness? ?? Result: – Very high cost, almost useless information

14. Disorganized, uncontrolled, decaying

15. Measurement Theory Connected Measures and Tests School 1 – Student A has a measure of 22 (+/- 2) on a reading test. – This classroom averages a measure of 24 (+/- 1). School 2 – Student Z has a measure of 18 (+/- 2) on a reading test. – This classroom averages a measure of 26 (+/- 1).

16. Measurement Theory Connected Measures and Tests Who has more reading ability, A or Z? A What can one student read that the other cannot? – Text with measures between 18 and 22. Which classroom reads better on average? 2 Which student is more on track for college readiness? ??

17. School 1 – Student A’s measures on 2 tests are 22 & 32 (+/- 2). – The classroom average goes from 24 to 30 (+/- 1). School 2 – Student Z’s measures on 2 tests are 18 & 32 (+/- 2). – The classroom average goes from 26 to 40 (+/- 1). Measurement Theory Connected Measures and Tests

18. Who gained more in reading ability, A or Z? Z What new texts can Z read? – Those with measures between 18 and 32. Which classroom improves more? 2 Are both students on track for college readiness? ?? Result: – Very high cost, incomplete, but useful information Measurement Theory Connected Measures and Tests

19. Organized, expressive, preserved

20. Metrologically Traceable Measures School 1 – Student A’s measure (22, +/- 2) is inferred when 73% of the items built into a reading assignment targeted at 22 are answered correctly. – This classroom averages a measure of 24 (+/- 1). School 2 – Student Z’s measure (18, +/- 2) is inferred when 76% of the items built into a reading assignment targeted at 18 are answered correctly. – This classroom averages a measure of 26 (+/- 1).

21. Metrologically Traceable Measures Who has more reading ability, A or Z? A What can one student read that the other cannot? – Text with measures between 18 and 22. Which classroom reads better on average? 2 Is one student more on track for college readiness? Yes, A

23. School 1 – Student A’s measures on 2 tests are 22 & 32 (+/- 2). – The classroom average goes from 24 to 30 (+/- 1). School 2 – Student Z’s measures on 2 tests are 18 & 32 (+/- 2). – The classroom average goes from 26 to 40 (+/- 1). Metrologically Traceable Connected Measures and Tests

24. Who gained more in reading ability, A or Z? Z What new texts can Z read? – Those with measures between 18 and 32. Which classroom improves more? 2 Are both students on track for college readiness? No, but A is Result: – Very low cost, complete and useful information Metrologically Traceable Connected Measures and Tests

25. Coordinated, harmonized, growing

26. What to choose? True Score Theory Economics School 1 Average Grade 7 End of Year Teacher’ Quiz Reading Score = 89% Average Gain in 7 th Grade Reading as measured by in-class quizzes and tests: ?? Annual tuition = US$5,000 Cost of average gain in reading scores = US$?? School 2 Average Grade 7 End of Year Teacher’ Quiz Reading Score = 94% Average Gain in 7 th Grade Reading as measured by in-class quizzes and tests: ?? Annual tuition = US$1,000 Cost of average gain in reading scores = US$?? Not enough information to decide! Simulated data

27. What to choose? Measurement Theory Economics School 1 Average Grade 7 End of Year Statewide Reading Measure = 32 (+/- 6) Adjusted average gain in 7 th Grade Reading Measures = 10 (+/- 4) Cost of adjusted average gain in reading measures = US$5,000.00 School 2 Average Grade 7 End of Year Statewide Reading Measure = 34 (+/- 5) Adjusted average gain in 7 th Grade Reading Measures = 11 (+/- 3) Cost of adjusted average gain in reading measures = US$1,000.00 Best buy But do you really want to buy the average gain? Simulated data

28. My 7 th grader’s gain – US$1,000 for 6 units – US$166.67 per unit gain Your 7 th grader’s gain – US$1,000 for 9 units – US$111.11 per unit gain What to choose? Measurement Theory Economics 50% greater cost!

29. What to choose? Measurement Theory Economics Reading Ability Scale

30. What to choose? Metrology Economics Best buy We might repeat the Measurement Theory outcomes… School 1 Average Grade 7 End of Year Statewide Reading Measure = 32 (+/- 6) Adjusted average gain in 7 th Grade Reading Measures = 10 (+/- 4) Cost of adjusted average gain in reading measures = US$5,000.00 School 2 Average Grade 7 End of Year Statewide Reading Measure = 34 (+/- 5) Adjusted average gain in 7 th Grade Reading Measures = 11 (+/- 3) Cost of adjusted average gain in reading measures = US$1,000.00 Simulated data

32. My 7 th grader’s gain – US$833.40 for 6 units – US$138.90 per unit gain Your 7 th grader’s gain – US$1,250.10 for 9 units – US$138.90 per unit gain What’s a parent to choose? Metrology Economics Same per unit cost! Simulated data

33. Readily available high quality information on product or service High CostLow Cost Hard for customers to find quality Easy for customers to find quality Customer Quality-Seeking Basic Economics High stakes measurement theory cost per test item: > US$3,000.00 Routine theory-informed metrologically traceable cost per test item: < US$0.01

34. What’s a teacher to choose? Metrology Economics Cost per unit gain: US$180 Simulated data Cost per unit gain: US$620

35. What’s a principal to choose? Metrology Economics Three schools Twelve months each A | B | C Cost per unit gained US$458 US$208US$116 Simulated data Better Reading Outcomes 

36. Basic Shop Floor Questions What is variation trying to tell us? (Deming) Which variations are due to common causes, and which are due to special causes? (Shewhart) How far can educational outcomes be maximized, and unwanted variation reduced? Can variation in outcomes be reduced by bringing all students to the highest levels?

37. What’s needed? System of distributed units Instruments measuring in uniform metrics Predictive construct theories to bring down costs Low cost items and administration Immediate results Continuous Quality Improvement (CQI) training and tools A culture that rewards innovation

38. What’s needed? We need commitment to a long range vision of quality education. But vision is not enough; we also need: – Skills – Incentives – Resources – Plans

39. What’s needed? Vision+Skills+Incentives+Resources+Plan= Sustainable Change +Skills+Incentives+Resources+Plan=Confusion Vision++Incentives+Resources+Plan=Anxiety Vision+Skills++Resources+Plan=Resistance Vision+Skills+Incentives++Plan=Frustration Vision+Skills+Incentives+Resources+=Treadmill Adapted from Knoster, T. P., Villa, R. A., & Thousand, J. S. (2000). A framework for thinking about systems change. In R. A. Villa & J. S. Thousand (Eds.), Restructuring for caring and effective education: Piecing the puzzle together, 2 nd Ed (pp. 93-128). Baltimore: Paul H. Brookes.

40. Disorganized, uncontrolled, decaying

41. Organized, expressive, preserved

42. Coordinated, harmonized, growing

43. Thank you