1. Approaches to evidence based policy making in education Professor Geeta Kingdon

2. Congratulations CBSE CAER commitment to evidence based decision-making Centre will help to improve learning by promoting reliable and valid assessments also help to create capacity do robust evaluations Thus bring assessment-led reform in Indian education Splendid example of a PPP

3. Talk is on evidence based policy, focus on macro Current landscape Pitiably poor learning levels (PISA, EI, NAS, ASER, NFHS, SchoolTELLS; NAS showed that <50% of class V kids could answer “how much greater is 555 than 198”? ) Measuring learning can be disheartening, when much effort & large resources expended (SSA reforms, RTE, NCF, 6 Pay Comm, TET, NCFTE, NCSL) & still outcomes are poor; denial Encouraging that Planning Comm’s 12 th FYP (GOI, 2013, p49) “will place the greatest emphasis on improving learning outcomes at all levels”

5. Steps for evidence based decision-making Obtain the evidence (e.g. measure learning levels) Evidence has to be credible (e.g. measure learning with validity and reliability, with proper sampling) Embrace & analyse results to reach diagnoses Formulating policies to take remedial action Pilot test new policies, to see their impacts on learning Scale up policies that have strong impact at lowest cost

6. Evidence from learning assessments – can empower parents An imp potential use of evidence from A – is parental info Has potential to increase school accountability World university rankings, but not school rankings Fear they may reflect socio-econ background but there are ways of reducing that In India a debate needed

7. Evidence from assessment of teachers One potential reason for poor learning - teachers themselves lack competence Teachers rarely tested in large scale way Evidence from T tests in India showed poor competence Led to the decision to bring in the TET

8. SchoolTELLS survey The teacher tests were graded by SCERT Bihar staff Assessment tasks for teachers aligned with standard teaching tasks that teachers in primary school would be required to do in the classroom routinely. Language tool helped to understand teacher’s ability in the following: Do you know: e.g. meaning of difficult words in a grade 4 level text Can you explain : e.g. explain difficult words in simple language or summarize a Std 4 story text effectively Can you spot common mistakes: e.g spelling and grammar mistakes Maths tool also helped to understand the teacher’s ability in the following : Do you know: e.g. solve problems Std 4 or 5 level Can you explain: e.g. explain problem solving in simple steps Can you spot and analyze common mistakes : e.g in arithmetic operations

9. TASK 1: VOCABULARY RELATED TASKS Grading done on 3 criteria:  Was the word meaning “meaningful” ?  Was language used “easy to understand” ?  Were there any spelling mistakes ? From Std 3 onwards, the vocabulary in language textbooks becomes difficult. So, teachers need to be able to explain difficult words in simple language.

10. TASK 2 : SUMMARIZING TEXTS

11. TASK 3: SPOTTING SPELLING & GRAMMATICAL MISTAKES

12. Deficits in teachers literacy skills 43% of word meanings correct; 57% wrong 45% of summaries were meaningful; 55% wrong 40% teachers did not have spelling mistakes (in a 2- sentence write up) 35% had 1-2 mistakes; 25% had >=3 mistakes Only 50% of teachers could spot >3 mistakes in a write up in which we had deliberately introduced 6 mistakes.

13. TASK 4 : ARITHMETIC PROBLEM : Percentages Percentage problem There are 38 children enrolled in a class. Of these 23 children are present today. What percent of children are absent today? or A class has 55 children. Of these 32 have books. What percent of children do not have books?

14. Area problem To plant a leechi tree you need 25 sq meters. Ramesh has a field that is 80 meters long and 70 meters wide. What is the maximum number of trees that he can plant in his field ? TASK 5: ARITHMETIC : AREA PROBLEM

15. TASK 6 : SPOTTING & UNDERSTANDING COMMON MISTAKES IN ARITHMETIC Samples of children’s work shown. Teachers asked to choose…

16. Deficits in teachers’ numeracy skills 78% could spot correct one when presented a sample of three simple division sums 24.5% could do a percentage sum; 75% not 27.9% could do an area sum; 72% not About 20% said they never had problems in addressing the maths queries of their pupils

17. How this evidence and TET evidence helps School-TELLS (2008) & ‘Inside Classrooms’ study (2011) highlighted deficits in T knowledge & ability to teach This evidence contributed to decision to introduce the TET TET objective to vet applicants, ensure competence Abysmal pass rate - 0.4 to 3.7% pass rate

18. Evidence in this T assessment is extremely valuable The evidence in this T assessment is extremely valuable It helps to identify the training needs of teachers. It can inform policy makers who decide training curricula But have states used this evidence in this way? States need the desire to analyse; make use of this evidence; a transparent approach Congratulate the CAER for analysing C-TET – this is evidence based policy making (the policy maker CBSE sought evidence – gave data)

19. Is this evidence relevant only for govt and rural private schools? Is this problem of low cognitive skills of Ts confined to government primary teachers ? Clearly more generic problem – Even highly paid govt T have major deficits in skills TET evidence Private schools cannot be complacent on this Testing T can help to assess the training needs of each T

20. The importance of evidence There are many initiatives to improve education NFE (1982)OBB (1986) TLC (1988)MDM (1982) SK (1987)LJ (1988) DPEP (1993)SSA (2003) Aadhar, ABL, MLE, Nallikali, Nai Disha, Read India, RTE Have these programs had impact? – little evidence to judge Efforts to improve more successful, when based on evidence

21. Culture of seeking evidence Why base decisions on evidence? While poor quality schooling does not threaten lives, it seriously affects people’s quality of life, and even longevity. In medicine, its unthinkable without thorough testing by experts, & the use of most robust, expensive randomised control trials; But in Education, Ministers freely make policies without consulting evidence / experts In good educ systems, and ideally, policy will be not made on supposition, ideology or political expediency. Govt seeks evidence

22. UK policy makers’ use of evidence from impact evaluations Early intervention - quality & effectiveness of pre-school experience in securing better long term outcomes was used to justify more investment (Sammons et al, 2006) Class size - evidence of no stat difference was used to justify not extending the policy into later years (Blatchford et al, 2002) Formative assessment - effect sizes in attainment (alongside pupil & teacher perspectives) led to inclusion in national policy (Wiliam et al, 2004) Education Maintenance Allowance - evidence from pilot study of post 16 retention led to national roll out, though long-term sustainability was worse in pilot areas (Middleton et al, 2005)

23. Policies could be better, if evidence based The narrative in the 6 th Pay Comm for across-the-board doubling teacher salaries (without increased accountability) was : it will motivate teachers. Was this effective? Did it raise teacher effort? No one checked. [next slide] A state govt recently announced it would regularise 176,000 para teachers, in the name of quality of education; it did not look at the relative effectiveness of regular and para teachers (3 papers) Under RTE it has been made mandatory for teachers to have B.Ed. Certification, reduced PTR to 30, and many inputs mandatory; where is the evidence for this? no pilot testing If there were garnering of evidence on the impact of policies, then ineffective policies could be weeded out

24. UP Reg.ParaPriv. Salary/month, 2008 (today after 6 th Pay Comm.) 12,017 (27,000) 3,000 (3,500) 940 (1400) Absence rate 24.612.017.4 % time teaching 75.383.389.0 SchoolTELLS survey (2008) Higher resources, lower effort Structure of accountability matters more than resources

25. What kind of evidence? What kind of evidence is useful / acceptable? Distinguishing correlation and causation The importance of methodology Using the force of the federal chequebook to nudge researchers to use robust methods capable of yielding causal inferences.

26. Interest in evidence of impact Field of medicine long interested in evidence of impact Recent upsurge of interest in impact evaluation in many fields nutrition, labour, governance, rural development, education, poverty ‘International Initiative for Impact Evaluation’ (3ie) established 2008 Network of Networks on Impact Evaluation (NONIE), comprised of: the OECD/DAC Evaluation Network (DACEN), the UN Evaluation Group (UNEG), the Evaluation Cooperation Group (ECG), and the International Organization for Cooperation in Evaluation (IOCE) International Development Evaluations Association (IDEAS) Development IMpact Evaluation (DIME) Spanish Impact Evaluation Fund (SIEF)

27. What is impact evaluation? Measuring outcomes (eg learning) is difficult But measuring outcomes is NOT impact evaluation measurement of net effects of a program on the outcomes of interest (e.g. on learning) In IE, it is important to appreciate the difference between correlation and causation

28. Correlation vs Causation As ice cream sales increase, number of drowning deaths increases sharply Therefore, ice cream causes drowning Sleeping with one's shoes on is strongly correlated with waking up with a headache Therefore, sleeping with one's shoes on causes headache. Taller people have higher earnings Therefore, higher height causes higher earnings

29. Salary and Height

30. Salary and Height?

31. Spurious Relationships This relationship is said to be “spurious” When we did the bivariate relationship, we said, In reality, things look like this: SalaryHeight SalaryHeight Gender Think of Sarva Shiksha Abhiyan and its impact

32. Correlation vs Causation In achievement equation with class-size, the coeff on class-size var is -0.46, i.e. an increase in class-size by 10 is associated with a 4.6 point lower ach mark Can we say from this that: class-size is negatively related to achievement? reducing class-size will lead student achievement to rise? what is the diff between these? what else determines ach? where is that included in the way the reg. equation is written

33. Quantitative impact evaluation methods OLS production function (not causal relation) Instrumental Variables Panel data Randomised experiment Quasi-experimental Propensity score matching method Each method has strengths & drawbacks RCT, PSM, Panel, IV

34. Evidence on impact of policy interventions Performance related pay Duflo and Hanna, RCT Muralidharan/Sundararaman, RCT Contract teachers Atherton & Kingdon, panel data Muralidharan et al, RCT Goyal & Pandey, OLS, school FE DPEP Jalan and Glinskaya, PSM Schmid, IV Private schools Desai et al, panel French/Kingdon, panel Mid Day Meal Afridi, 2010, panel Union membership Kingdon Teal, panel

35. Greater emphasis on evidence NCERT evaluated impact of programs in 4 states, under TCF Data becoming available – EI, ASER, DISE, SEMIS, NAS What’s imp is the quality of the data / studies, i.e. robustness of methods/ designs; degree to which they tease out causal effect Long term investment needed in capacity dev MHRD / JRM approval for estb of National Assessment & Evaluation Centre

36. Some important considerations Even evidence showing reliably what works is insufficient Policy makers need to: seek, read such evidence discern good/bad evidence act on evidence, i.e. make evidence-based policy Freedom from political interference Independence from funding body, hence PPP better

37. Good evidence does not always permit good decisions Even when we have good evidence a policy has capacity to substantially improve outcomes, there can be powerful political economy barriers to the implementation of policies. E.g. Duflo and Hanna (Rajasthan) say : “Although this study suggests that a system of automatic monitoring with enforcement by physically remote agents who are prepared to enforce the rules is technically feasible and indeed provides better incentives for teachers, a later effort to introduce this system with higher-skilled, higher-status, and more politically powerful health-care workers ran into strong political obstacles (Banerjee et al. 2007b).” Another e.g. para T in UP We need to understand the political economy constraints, and how can they be eased

38. Thanks

39. TASK WITH DIFFICULT WORDS DIFFICULT WORDS Bihar govt. schools UP govt. schools Bihar/ ALL UP Reg. Para 05 Para 06Reg.ParaPrivateAll Word meaning Not attempted8.810.810.25.08.312.59.4 Wrong meaning35.533.435.427.229.332.032.2 Partial meaning12.414.817.316.913.515.114.9 Full meaning42.441.037.150.848.940.443.4 Four difficult words are given. Please write their meaning using simple words VOCABULARY TASKS : DO TEACHERS KNOW WORD MEANINGS ?

40. SUMMARIZING TEXTS : EXAMPLES FROM TEACHERS

41. TASK WITH PASSAGE Bihar govt. schools UP govt. schools Bihar/ ALL UP Reg. Para 05 Para 06Reg.ParaPrivateAll Gave meaningful summary? Not attempted 3.52.45.63.32.68.64.4 Irrelevant/wrong 25.427.040.528.640.237.933.0 Partially meaningful 25.415.922.59.916.216.417.9 Fully meaningful 45.654.831.558.241.037.144.7 SUMMARIZING TEXTS : CAN TEACHERS SUMMARIZE ?

42. TASK WITH PASSAGE TASK WITH PASSAGE Bihar govt. schools UP govt. schools Bihar/ ALL UP Reg. Para 05 Para 06Reg.ParaPrivateAll Summary easy to understand Easy to Understand71.269.678.679.381.373.875.0 Are there any spelling errors? No spelling error32.029.128.254.748.246.739.6 1-2 Spelling errors44.338.525.630.235.534.435.3 >=3 spelling errors23.732.546.215.116.418.925.1 SUMMARIZING TEXTS : CAN TEACHER SUMMARIZE USING SIMPLE LANGUAGE Of those writing meaningful summary

43. BiharUPAll TASK: PERCENTAGE WORD PROBLEM Reg.Para 05 Para 06 Priv.Reg.ParaPriv. Not attempted 14.412.026.437.016.723.528.620.6 Incomplete 32.748.846.225.940.0 54.642.6 Wrong steps 5.86.45.511.110.03.51.35.7 Correct steps wrong answer 3.96.43.33.74.47.01.34.6 Correct answer no steps 0.01.63.30.01.14.42.62.1 Solved correctly 43.324.815.422.227.821.711.724.5 TASK 4 : PERCENTAGE PROBLEM : Findings

44. BiharUPAll AREA PROBLEM TASK Reg. Para 05 Para 06-07Priv.Reg.ParaPriv. No attempt 27.928.838.551.930.048.741.636.6 Incomplete 19.225.626.47.418.919.126.021.8 Wrong steps & A 5.84.01.13.77.83.52.64.1 Correct steps, wrong A 3.93.28.80.04.41.75.24.1 Only correct A, no steps 4.85.63.30.08.94.49.15.5 Solved correctly 38.532.822.037.030.022.615.627.9 TASK 5 : ARITHMETIC : AREA PROBLEM

45. BIHARUP Fully agree Parti ally agree Some what agree Dis agree Fully agree Parti ally agree Some what agree Dis agree Govt. school teachers24.511.046.817.715.218.343.122.3 Private school teachers16.712.545.825.016.918.536.927.7 % teachers who agree with the statement “Sometime I have difficulties in addressing mathematical queries and problems of my students” 80% primary school teachers have difficulties in teaching maths Only about 20% of govt. school teachers believe they don’t face problems. About 80% admit to have difficulties sometimes. This suggests possible interest in in-service training to upgrade maths skills