1. Measuring education - 1 What to measure? International comparisons

2. 1 – Inputs, outputs, outcomes The National Accounts framework

3. National accounting framework One aim of National Accounts = value production in the economy as a whole Production = transformation of inputs (L, K) into outputs (goods & services: G & S) Measuring production begins with identifying all the units which produce the G & S in question. Next, consider which of the transactions of these producers need to be identified in measuring production in the National Accounts Then, data on these transactions need to be obtained and aggregated

4. Education as a non market service In the National Accounts, producers are allocated to industries according to a classification system of industries twinned with a classification system of products. International Standard Industrial Classification (ISIC), linked with the Central Product Classification (CPC) Education is Division 85

5. Education as a non market service Education (Division 85) –851 Pre-primary and primary education –852 Secondary education 8521 General secondary 8522 Technical and vocational secondary –853 Higher secondary –854 Other education –855 Educational support activities Source: International Standard Industrial Classification, revision 4

6. Education as a non market service Education and health services: government provisions them free of charge or at not economically significant prices = non-market output. Price is “not economically significant” if it covers less than half of the costs of the service Price paid cannot be the basis for valuing it Valuation for NA = sum of the costs incurred in its production

7. Education as a non market service Costs = sum of: –Intermediate consumption (the goods and services used up in producing the service) –Compensation of employees (costs of teachers, doctors, nurses, etc.) –Consumption of fixed capital (depreciation of school and buildings, of medical equipment) –Other taxes, less subsidies, on production

8. Education as a non market service How to measure the output of education & health? (1) Use costs in place of prices (2) Impute some value for the relative valuation by the consumer to generate aggregation weights.

9. Solution (1) Assume that, on average, the relative valuation by the consumer equals the relative cost incurred by producers ? Government producers: Argument = in a democracy, via the electoral process, consumers have influence over the production decision of governments so that the provision of non-market services is socially optimal. relative costs = relative utility / willingness to pay

10. Solution (2) Impute some value for the valuation by the consumer ? (2) Impute some value for the relative valuation by the consumer to generate aggregation weights. implies a fully developed consumer or welfare perspective in the estimation of the volumes of health and education services. = controversial –from a NA perspective if total value of health or education services  total expenditure –if the construction of weights entails delicate questions such as the necessity to put a monetary value on human life

11. Back to the pb of aggregation How to aggregate the production of 300 cars and 10,000 salads? Answer for market products = market price Idea that it reflects the relative utility of G & S

12. Back to the pb of aggregation But how do you aggregate 300 heart surgery operations and 10,000 eye treatments with no market price? To describe the volume or price of a product category, you have to assume all goods in that category fulfill the same need + the price deflator implies an evaluation of the increases in quality

13. Output in value / in volume GDP at current prices is not so interesting as growth of GDP and its components in volume Growth in volume excludes changes in price levels between two periods Only elements of the National Accounts that can be disaggregated in terms of prices and volumes are useful in analyzing economic growth, productivity and inflation

14. Previous methods and problems Previous methodologies : measuring the volumes or the prices of inputs to obtain a measure of the volume or price of outputs Pb: Such input-based methods cannot capture any productivity change Productivity growth exists when more or better output can be produced with the same resources. Quality of output does change in education services  ignoring productivity changes = foregoing important information and measuring volume growth inaccurately

15. Reminder: volume and price indexes Remember that in principle, deflating sales to consumers by a consumer price index = calculating directly a volume index by weighting together quantity indices which represent the volume of each homogeneous service The amount spent on each service – a function of its price - provide the weights to aggregate these quantity indices No difference in concept to construct a price index and then deflates sales or to construct a volume index directly

16. Reminder: volume and price indexes With no quality change, the price index for education services would be a weighted average of the price indices of teaching service activities This price index can then be used to obtain a volume measure of output by deflation

17. The issue of quality adjustments Require the identification of characteristics such as speed, engine size for a car or the processor speed for a computer Idea = isolate price changes due to changes in characteristics from those price changes that constitute ‘inflation’

18. The issue of quality adjustments One possibility to measure quality = to put together several characteristics into a single indicator that reflects the contribution of the product to outcome. Ex: price indices for health care. Triplett (1998) suggests quality-adjusted life years (QALYs) as a single dimensional measure could be used for quality-adjustment of different treatments within a product group

19. The issue of quality adjustments Should not be affected by any other factors that influence consumer outcome (e.g., socioeconomic background of students or lifestyle of patients) and Should reflect the normal / average /expected effect of the activity on “state of knowledge” of the consumer. ‘Normal’, ‘average’ or ‘expected’ because to the extent possible, measures of service production should not be influenced by the individual capacity of the consumer to make use of these services

20. The issue of quality adjustments Discussion on the necessity to invoke utility and outcome Characteristics that matter for consumers have to be identified for quality adjustment “glance over the national accounts production boundary” to deal with quality change in education services The quality of services cannot be identified without considering indicators falling outside the production boundary OECD conference in June 2007: current issue!

21. In-, output, outcome, utility

22. General links between output and outcome Hill services and result of the production process Peter Hill (1975) says the production of services is inseparable from their use or consumption: two economic units are simultaneously involved in service production whereas production of a good takes place within a single producer unit.  Output of services should be viewed as the attributable contribution of the producer to the outcome. Unit ProductProducerConsumer GoodOutput ServiceOutcome

23. Definitions: education output Eurostat handbook on prices and volume in NA “the quantity of teaching received by the students, adjusted to allow for the qualities of the services provided, for each type of education” “In the area of education, the output can be defined as the quantity of teaching (that is, the transfer of knowledge, successfully or not) for a pupil, whereas the related outcomes are the skill and knowledge that a pupil achieves.” ISCED definition “organised and sustained communication designed to bring about learning”, where “Communication” involves the transfer of information (messages, ideas, knowledge, strategies, etc.) between two or more persons; […] knowledge “Learning” is taken as any change in behaviour, information, knowledge, understanding, attitudes, skills, or capabilities which can be retained and cannot be ascribed to physical growth or to the development of inherited behaviour patterns

24. Definitions: education output (2) OECD (2007) proposal  “expected transfer of knowledge and skills provided by an education unit, for each type of level of education / curriculum”  Transfer of knowledge and skills can be seen as the true quantity indicator, and change in academic scores due to school could provide a right measurement.  Expected = « average », means that each pupil receiving the same teaching is supposed to consume the same output, although some pupils will assimilate more than others (ES handbook: « successfully or not »). This difference in outcome is supposed due to the pupil/consumer.

25. Basic formula I volume = I quantity (number of enrolled pupils) x I quality ed (average transfer of knowledge) x I quality non ed (total expenditures / core educational) Without information on the transfer of knowledge, we find the number of pupils, 2 nd best quantity indicator for Eurostat. If we assume that the transfer is continuous and proportional to time of teaching, we get the “pupil x hours”, 1 st best quantity indicator for Eurostat.

26. Overview of inputs, output, outcomes in volume

27. Education output = average transfer of knowledge and skills, some remarks No accounting of the distribution of individual levels of knowledge and skills, although “reduction of inequalities” is usually one of the targets of the education policy: consistent with the idea of “individual services” in NA No accounting of healthy, wealthy, well-rounded people / citizens, which are sometimes difficult to measure and which are more “indirect” outcomes. Education status (outcome) is measured at the end of a curriculum, rarely at the beginning  we need a model (role of the pupil / consumer) in order to estimate the part of the output in the outcome.

28. Other outcome-based quality indicators OECD: academic scores as first best direct education outcomes But some curricula could be better estimated by obtainment of a degree or professional insertion (indirect outcome). Professional insertion: employment rate (if the right qualification if possible) and real earnings can be used (multiplied) Coherent with human capital theory (discounted sum of future real earnings as return on “GFCF” in education), Pb of evaluation: what is obtained after the curriculum minus what would have been obtained without, if we want to measure the “output”.

29. 2 – International comparisons: historical landmarks

30. A little bit of history Until after WW2, educational systems were evaluated without any indicators Inspectors judged what went on in classes according to what they had learnt to be “good teaching” 1) The 1970s OECD, 1973: “a system of indicators aiming at orienting policy”. Goal: increase social well-being. Social scientists  A set of 47 indicators…

31. A little bit of history … that were never calculated because there was no data to do so and governments were not willing to spend money to collect it  Failure 2) The 1980s 1983: A Nation at Risk (the US). Strong criticism 1986: Education Indicators. A guide for policymakers = a period when the focus is on the quality of teaching (pedagogy). Heated debates on indicators

32. A little bit of history 3) The 1990s 1991: OECD conference Making Education Counts: Developing and using international indicators. At first, nothing on students’ achievement Only the educational system: costs, enrolments = a management approach: reducing costs, optimizing resources First volumes of Education at a Glance: 1992

33. A little bit of history 4) The 2000s Strong focus on students attainment. “educ at a glance”: -A: output -B: resources -C: access -D: learning environment, organization of schools Chapter on output was last until 2002: moved to 1 st position. From 5 indicators of students attainments to 14 = a dramatic change in perspective Psychometric approach, cognitive sciences have become dominant 1995: International Adult Literacy Survey / 2000: PISA

34. A little bit of history The Lisbon criteria (2000): “knowledge economy” Reduce to 10 % the proportion of early school leavers (indicator 23). Reduce by at least 20% the proportion of students with bad reading results (indicator 5). At least 85 % of youth should complete secondary education (indicator 4). Increqe by at least 15 % the number of graduates from scientific fields, while simultaneously reduging the gender gap (indicator 12). Participation rate to continuing education: 12,5% of the adult population (indicator 19).

35. 3 – ISCED The International Standard Classification of EDucation

36. The ISCED classification 1) A little bit of history (again) UNESCO, 1975-1978  ISCED 1978 From the start, goal = int. comparisons 1990: need for change More adult education, more distance learning, more various education providers On-the-job training (Germany) not taken into account  ISCED 1997

37. The ISCED classification 2) What makes building an international classification so difficult? Definitions must be precise but also work in very different contexts –Ex: “completed” education when there is no exam/ diploma? –Ex: what is “basic education”? –Ex: “Vocational” training: theoretical or actual ? Risk = countries including very different things in a given category  meaningless comparisons

38. The ISCED classification 2) What makes building an international classification so difficult? The right number of items –30 is not usable, 4 is too little… –Usually: 2 to 6 different levels, hierarchical Being understandable across the world: translation problems –Ex ISCED: 6 official languages + 100 countries using it as a tool in a foreign language

39. The ISCED classification 2) What makes building an international classification so difficult? Accepting differences within a category –2 exact same programs do not exist = need to define what is the more important criteria. –ISCED: level rather than length or organisation of education –Ex: primary education may last 4 to 8 years … but clearly defining boundaries –Implies spending much time discussing borderline, infrequent cases

40. ISCED-97 levels of education Level 0Pre-primary education Level 1Primary education or first stage of basic education Level 2Lower secondary or second stage of basic education Level 3(Upper) secondary education Level 4Post-secondary non-tertiary education Level 5First stage of tertiary education Level 6Second stage of tertiary education

41. ISIC rev. 4 and ISCED 97 ISIC rev 4 classesISCED-97 levels of education 8510Pre-primary and primary education Levels 0 and 1 8521General secondary educationLevels 2 and 3 oriented general 8522Technical and vocational secondary education Levels 2 and 3 oriented vocational and technical 8530Higher educationLevels 4, 5 and 6 8541Sports and recr. education Not classified in ISCED-97 levels of education 8542Cultural education 8549Other education n.e.c. 8550Educational support activitiesNot explicitly mentioned in ISCED-97 levels of education Formal education (our focus) informal Supportactivities But no R&D or adm., like in COFOG

42. The ISCED classification 3) The basics of the ISCED classification 0 to 6: relies on belief in a hierarchy of skills: from basics to more elaborated Cannot simply be based on number of years? No: number of hours, curriculum, goals of one year very different from one country to another –Ex: more time, same goals for students with disabilities : same level in more years –Ex: selective tracks that go faster: less years, more skills

43. The ISCED classification 3) The basics of the ISCED classification 0 to 6: relies on belief in a hierarchy of skills: from basics to more elaborated OK but hierarchy must be made explicit –Ex: theoretical knowledge higher than practical skills? Definition of “education”: the object of the classification

44. The ISCED classification : defining education “ISCED does not intend to provide a comprehensive definition of education, still less to impose an internationally standardized concept of the philosophy, aims or content of education, or to reflect its cultural aspects. Indeed, for any given country the interplay of cultural traditions, local customs, socio-economic conditions, at the very least, will have resulted in a concept of education in many ways unique to that country, and any attempt to impose a common definition would not be productive. However, for the purposes of ISCED, it is necessary to prescribe the scope and coverage of the educational activities to be covered by the classification.”

45. The ISCED classification: defining education “Within the framework of ISCED, the term education is thus taken to comprise all deliberate and systematic activities designed to meet learning needs. This includes what in some countries is referred to as cultural activities or training. Whatever the name given to it, education is understood to involve organized and sustained communication designed to bring about learning. The key words in this formulation are to be understood as follows:” - organized - sustained - communication - learning

46. The ISCED classification 4) Ex of problems: level 4 “Post-secondary non-tertiary education” Was created to put short programs that were classified as secondary or as tertiary depending on countries  made comparisons invalid A category that was created to get rid of those programs, making comparisons between the rest relevant But not understood, not used as was expected 5) Ex of debate: cut-off btw 5 & 6? Has to be well defined because countries will always want to put more in the highest level 6 does not include research Masters, only PhD

47. 4 – What outcome at what level?

48. What are the expected outcomes of Preschool ? –Not easily measured. Ex: evaluation of Perry School Project, Abecedarian Project, Head Start… –Range from grade repetition before 15, to wage at 25, to number of arrests before 40…

49. What are the expected outcomes of Primary education ? –“basic knowledge”: literacy and numeracy. But when do you measure them (at what age?) and how (  next class) Middle and high school ? –Goal of international comparisons  not possible to rely on “what students know” (ex: history, geography, literature… national, hence not comparable) –Define a number of “skills” + measure them without cultural bias? (  next class) These = goals of PISA

50. What are the expected outcomes of Universities ?

51. 5 – International rankings of higher education institutions (“universities”)

52. ARWU Since when does it exist? - answer: 2003 Why such a success? - Context: focus on education and research as investment - Internet: more widespread information  Increased international competition

53. Who? Which institutions should be included? - ARWU: all institutions with Nobel Prizes, Fields Medals or frequently cited researchers - Times: all institutions quoted by the 5010 experts - Financial times: Masters programs - Leiden: universities only

54. What should count? How do you calculate a score? - ARWU: awards, citations (researchers, articles, top 20 reviews) - Times: experts quote 30 establishments they consider as excellent in their field + survey of recruiters + citations + staff/students ratio - Financial times: Experts + indicators on international openness - Leiden: bibliometrics

55. What do these rankings do? Bibliometrics  important size effect: More researchers, more papers Countries where research done outside of teaching establishments (“pure” research) : not counted  disadvantaged The difficulty lies in the many aspects of higher education –Teaching (basics, research or business oriented?) –Research (applied, fundamental?) –Implementation (patents, consulting…) Put all on the same scale implies weighting

56. What do these rankings do? ARWU stresses research only, and excellence in research (Nobel Prizes). Leaves students, their careers, apart Times: research = 20% CC: Depends on who the ranking is aimed at: - Students? Can they be Misleading? - University managers? - Policy makers?

57. Consequences These rankings have massive institutional, practical consequences –Ex of France: quest for “Critical size” (Paristech, PSE…) –Ex on research practice: pressure to publish under this or that affiliation Pb: indicators, rankings are quickly translated into straightforward evaluation: “x is better than y (in general)”. Nevermind  sizes, means, goals… Trade-off between simplicity and completeness

58. 6 – A few important figures

59. A few important figures OECD avg Share of GDP spent on initial education: OECD

60. A few important figures The share of educ/GDP is the result of 3 factors: -Size of the school-age population -Percentage going to school -Spending per pupil

61. A few important figures Yearly spending per pupil, primary to higher education, ppp$, 2005 (OECD)

62. A few important figures Yearly spending per pupil, primary education, ppp$, 2005 (OECD)

63. A few important figures Yearly spending per pupil, higher education, ppp$, 2005 (OECD)

64. A few important figures Spending per pupil in sec and higher, relative to spending on primary education (=100) 2005 (OECD)

65. A few important figures Number of science graduates for 100 000 25-34 year- olds in the labor force « science » here includes biology, physics, maths, statistics, computer sciences, engineering and technical professions, making and transformation, building and architecture MenWmenTotal

66. A few important figures Spending on higher education institutions, GDP% (2005) Grey = private / red = government

67. A few important figures Proportion of 15-yr-olds having repeated a grade at least once (2003)

68. Lessons to be learnt Output / outcome Indicators on processes / on outcomes: what is “evaluation”? Indicators sometimes shape the debate… but sometimes don’t (ex: debate on education in Fr since 1990) Rankings of universities on a global scale: recent. Competition within/ between countries. Importance of the design of classifications (ex: ESEC for occupations)