1. Standing on the Shoulders of Chess Masters: Using RCTs to Produce Evidence about Teaching Chess in Italian Elementary Schools Gianluca Argentin Alberto Martini Barbara Romano romanob@gse.upenn.edu

2. Why should chess be taught in school? According to the European Parliament for several reasons

3. The EU Parliament claims that “Chess can improve children’s concentration, patience and persistence and can develop sense of creativity, intuition, memory, analytic and decision- making skills. Chess also teaches determination, motivation and sportsmanship”

4. The evidence on the effects of learning chess on cognitive outcomes is literally scattered all over the world: from chess association websites to academic journals in a variety of disciplines. However Rarely this evidence is based on rigorous evaluation design such as randomized controlled trials

5. We will present: 1. the results of a simple RCT to measure the effects of learning chess on math achievement (SAM) 2. the design of a more complex RCT to answer several questions that arise when incorporating chess in the regular curriculum (SC3) Our true long run objective is promoting the use of random assignment in evaluation and education research in Italy.

6. PROGETTO SAM Scacchi e Apprendimento della Matematica

7. We recruited 33 schools that had been involved in chess teaching before (a non probabilistic sample, yet nicely spread all over Italy)

8. Each school agreed to exclude at least one 3° grade class, chosen randomly All the other 3° grade classes were taught 30 hours of chess by professional instructors during 2009-10 school year

9. A total of 113 classes - about 2000 pupils - participated either as treatment or control All pupils were pre-tested and post- tested using modified versions of INVALSI tests High level of compliance to the design

10. Clearly something is happening to the experimentals Impact Pre-test Score distribution Post-test Score distribution

11. Estimated effect on math scores (in standard deviations) ITT Mod. 0: nessun controllo, tutti i casi 0.28 p value.01*** Effect size0.34 P-value.01*** A third of a standard deviation is an exceptionally good result This makes people (including us) doubt it

12. Is it because you included questions too favorable for chess players? a. Draw the snail path b. Which shape corresponds to the snail path?

13. We found an effect on almost each item on the test Differential probability of correct response for each item

14. IT MUST BE THE EFFECT OF CHEATING!

15. CHEATING SHOULD SHOW UP HERE, BUT IT’S EMPTY

16. We need to prove it with an RCT based on a probabilistic sample of schools Chess does work!

17. Sc3 S PERIMENTAZIONE C ONTROLLATA DI S CACCHI A S CUOLA (Springer = Knight in German)

18. Does learning chess at school (at some point between 3 rd and 4 th grade) affects achievement in mathematics and reading measured in 5 th grade? How persistent are the effects of learning chess? Does any beneficial effect remain one or two years after training? Which fraction of pupils continue to play chess and possibly benefit from it, after it is no longer taught in class? Does chess taught by professional instructors obtain better or similar results than when based on web teaching tools with the support of regular teachers? Which solution is more cost-effective and sustainable at scale?

19. First question: does learning chess improve achievement in math and reading, as measured by INVALSI standardized tests 80 control schools 160 treated schools Fifth grade INVALSI National test Fifth grade INVALSI National test Second grade INVALSI National test Second grade INVALSI National test 240 schools selected with probability prop. to size out of 18,000 primary schools in Italy RA

20. The second question: External instructor or regular teacher+web? 80 schools “teaching by external instructor” Fifth grade INVALSI National test Fifth grade INVALSI National test Second grade INVALSI National test Second grade INVALSI National test 80 schools “web-based with support by regular teachers” RA

21. The third question: Do school-based activities increase persistency of the effect of learning chess? 80 schools promote chess in 5° grade Test of knowledge of chess and frequency of use (late In fifth grade) Test of knowledge of chess and frequency of use (late In fifth grade) 80 schools do not promote chess in 5° grade Test of knowledge of chess and frequency of use (early in third grade) Test of knowledge of chess and frequency of use (early in third grade) RA

22. The two dimensions crossed represent a factorial design

23. We are looking for sponsor/supporter And for other countries willing to “play (it again) SAM” Anyone interested?