1. Netherlands Organisation for Scientific Research NWO / PROO Dutch Programme Council for Educational Research Instruments for measuring literacy development of at-risk adolescents Ron Oostdam, Amos van Gelderen, Roel van Steensel University of Amsterdam

2. Netherlands Organisation for Scientific Research Literacy development of at-risk students in multilingual contexts: A tale of three cities International project in cooperation with the Ontario Institute for Studies in Education (OISE) and Service de la recherche en éducation (SRED) Main research questions:  What are the most important socio-cultural, educational and individual variables associated with literacy development of at- risk adolecents (grades 7 to 9) in multilingual contexts?  Do the different variables play similar roles in predicting at-risk and non-at-risk adolescents’ literacy?

3. Netherlands Organisation for Scientific Research Factors influencing students’ literacy

4. Netherlands Organisation for Scientific Research Design of the research project groups of variables School environment (part- project 1) 1) nature of literacy tasks 2) embeddedness of literacy in curriculum 3) Instruction selfregulatory literacy strategies 4) connections with home-environment 5) social context of teaching & learning home and community environment (part-project 2) 1) nature of literacy tasks 2) embedding of literacy 3) use of self regulatory literacy strategies 4) connections with school-environment 5) social context 6) personal backgrounds individual attributes (part-project 3) 1) knowledge of vocabulary and grammar 2) metacognition reading and writing 3) fluency of word and sentence processing 4) non-verbal intelligence 5) attitudes towards literacy

5. Netherlands Organisation for Scientific Research Integrative part project 4  Part project 4 is directed at the measurement of literacy  The whole research project consists of a longitudinal part (3 years) and a large scale cross-sectional part  Longitudinal approach consists of an in-depth study of a focused sample of 40 at-risk adolescents  The part-projects monitor the participants on the different groups of variables  The cross-sectional study takes place in part-project 4 only  Results from the part-studies 1-3 are integrated  In the cross-sectional study predictions about the relations between the variables and literacy are tested

6. Netherlands Organisation for Scientific Research cross-sectional study  Allows testing with substantial statistical power for hypotheses derived from the multiple case-studies  Allows analyzing interactions between socio-cultural, educational and individual variables in a statistically reliable way  Allows comparison of literacy skills between at-risk and non-at-risk students

7. Netherlands Organisation for Scientific Research International Literacy Test  Literacy skills defined in a broad way: the degree in which students are able to read and write in order to participate in all kinds of functional and cultural exchange in the surrounding society  The PISA definition of reading is adopted (the ability to understand, use and reflect on written tasks in order to participate effectively in life) and extended to writing  Reading and writing tasks were constructed in close cooperation with Toronto and Geneva

8. Netherlands Organisation for Scientific Research Criteria for measuring literacy  Criterium of socio-cultural validity: measurement of literacy takes place with reading and writing tasks grounded in the lives of students  Test has to take into account differences in literacy skills  For international comparison the tasks need to be designed in such a way that they approach international, socio-cultural equivalence

9. Netherlands Organisation for Scientific Research Taxonomy literacy test Four types of text types: narrative, expository, argumentative, instructive (reading and writing) Four types of media: (school)books, newspapers and magazines, ‘official’ documents, internet (reading) Distinction made between continuous and discontinuous texts (reading) Three different types of questions: retrieving, interpreting, reflecting (reading) Two answer formats: multiple choice and open (reading)

10. Netherlands Organisation for Scientific Research Reading tasks pilot study (* = discontinuous) Narrative (2 tasks) Expository (7 tasks) Argumentative (2 tasks) Instructive (3 tasks) Internet (2 tasks) WikipediaWebsite camping (School) Books (3 tasks) King clothesBasilisk Vietnam Newspapers & Magazines (8 tasks) The boardBokito TV-guide* Newsletter* Aroma Jockey Cell phone ban Bio-fuel Recipe* Official documents (1 task) Regulations youth hostel*

11. Netherlands Organisation for Scientific Research Example reading task HOUSE RULES 1.Smoking and lighting fires are strictly prohibited on our premises! 2.For safety and hygienic reasons hot dishes and drinks must not be prepared in living rooms or bedrooms. 3.During the night (from 10:00 p.m. until 6:00 a.m.) respect the rights of other guests and residents to a good night’s sleep. Showers should not be used and noise should be avoided at these times. We expressly point out that in the event of disturbance caused by noise, measures can be taken against the person(s) who caused the noise, including the termination of the contract for accommodation. 4.Guests are requested to avoid waste and to be economical in using electricity, heating, and water. Kindly deposit any garbage that cannot be avoided in the appropriate containers for recycling or disposal. 5.No visitors from outside are allowed in the bedrooms. Visitors can be received in the cafeteria daily from 10:00 a.m. until 10:00 p.m. Persons who are not occupants of our hostel cannot stay overnight in the hostel.

12. Netherlands Organisation for Scientific Research Retrieving 5.No visitors from outside are allowed in the bedrooms. Visitors can be received in the cafeteria daily from 10:00 a.m. until 10:00 p.m. Persons who are not occupants of our hostel cannot stay overnight in the hostel. What do the rules say about visitors from outside? a)There are no restrictions with respect to receiving visitors from outside. b)Visitors from outside are not allowed. c)Visitors from outside are only allowed at certain times and in a certain place. d)Visitors from outside are only allowed after the hotel management has given permission.

13. Netherlands Organisation for Scientific Research Interpreting 3.During the night (from 10:00 p.m. until 6:00 a.m.) respect the rights of other guests and residents to a good night’s sleep. Showers should not be used and noise should be avoided at these times. We expressly point out that in the event of disturbance caused by noise, measures can be taken against the person(s) who caused the noise, including the termination of the contract for accommodation. Why can’t guests take showers during the night? a)Because of safety and hygienic reasons. b)Because the hostel wants guests to be economical in the use of water. c)Because the hostel does not provide hot water at night. d)Because taking a shower could disturb other guests.

14. Netherlands Organisation for Scientific Research Reflecting 9. The hostel depends on a helping hand from our guests. We need your assistance with cleaning and tidying up the equipment, rooms and objects you have used during your stay as well as setting and clearing tables in the cafeteria. What is the purpose of rule 9? a)To offer a complaint. b)To give an order. c)To give advice. d)To make a request.

15. Netherlands Organisation for Scientific Research Writing tasks pilot study Narrative (2 tasks) Expository (1 task) Argumentative (2 tasks) Instructive (1 tasks) King clothes (finish the story; last sentence given) Bokito (letter to a friend) Cell phone ban (article for schoolpaper) Melissa (write a story; first and last sentence given) Yummi Yummi Candybar (letter to producer chocolate bars) Day out (exchange project between schools)

16. Netherlands Organisation for Scientific Research Example writing task SAVE FOR TWO FREE CINEMA TICKETS!!! This is what you must do: On each Yummy Yummy candy bar wrapper there is 1 point. Collect 10 points and send these in a sufficiently stamped envelope to: Yummy Yummy Candy Bars Points Offer, PO Box 3333, 1273 KB Etten-Leur, the Netherlands. Also include € 0.39 in stamps to cover postage. Clearly write your name, address and postal code, and the free (FREE!) cinema tickets will be sent to your home as soon as possible. This offer is open until April 15. It is April 7. You have collected a total of 8 points, but you cannot find anymore bars with points. The bars in the shops have no points on the wrappers, but it is still not April 15. Thus, you cannot get your 10 points together. Nevertheless, you wish to receive the two cinema tickets. Therefore, you send your 8 points along with two complete wrappers without points. Write a letter to enclose with the points and wrappers. Explain why you are unable to send ten points. Convince Yummy Yummy Candy Bars that you want to receive the two cinema tickets and that there was nothing you could do to get ten points. Make sure that they send you the cinema tickets! Then address the envelope.

17. Netherlands Organisation for Scientific Research Pilot study Pilot of reading tasks  14 reading tasks (internet tasks not admistered in Switserland)  408 Dutch students (grade 7 and 9)  298 Swiss students (grade 7, 8 and 9)  Interviews with 40 Dutch students Pilot of writing tasks  6 writing tasks  80 Dutch students (grade 7 and 9)  24 Swiss students (grade 7, 8 and 9)  Interviews with 4 Dutch students

18. Netherlands Organisation for Scientific Research Final selection tasks Reading tasks  Contribution of items to the overall reliability of the literacy test  Difficulty level of the items  Distribution of questions over formats (retrieving, interpreting, reflecting; open or closed)  Distribution of tasks over text types and media  Differences between grades  Opinion of students about the tasks Writing tasks  Global scoring on text quality and text length  Global scoring of written texts on text type and content  Opinion of students about the tasks Practical constraints  Maximum administration time (about 3 hours)  Internet tasks (and writing tasks) demand computer facilities

19. Netherlands Organisation for Scientific Research Results reading tasks  Overall test reliability for reading tasks sufficient:.75 (Geneva, without internet tasks) and.77 (Amsterdam, including internet tasks)  Total number of items: 102: Retrieving (45), Interpreting (31), Reflecting(26)  One task showed a ceiling effect in Geneva as well as Amsterdam  A relatively small number of items with a item rest correlation below.10: Geneva (13); Amsterdam (19)  All tasks, except for one, show a sufficient difference between grades  Most students evaluate the tasks as attractive and enjoyable to read, with the exception of ‘House Rules’ and ‘TV guide’

20. Netherlands Organisation for Scientific Research Results writing tasks  The narrative assignments seem to work quite well variability in content, quality and text length In one task students ‘squeeze in’ the elements from the final sentence right at the end of their texts  The expository assignment is problematic Requested text type (a letter) often not delivered Little variability in content  The argumentative assignments differ Yummi Yummi candy bar evokes good writing products Other task evokes texts without relevant content  The instructive assignment works well, although students find it too much work to process all the given information

21. Netherlands Organisation for Scientific Research Final remarks –In the next 2-3 weeks a final selection made for 8 reading and 3-4 writing tasks –Items with a low item rest correlation removed or adjusted –Final data collection (first measurent) done in april- may

22. Netherlands Organisation for Scientific Research For further information: Dr. R.J. Oostdam Faculty of Social and Behavioral Sciences SCO Kohnstamm Institute for Educational Research University of Amsterdam P.O. Box 94208 NL-1090 GE Amsterdam phone: ** 31 (0)20 525 1330 fax: ** 31 (0)20 525 1200 E-mail: R.J.Oostdam@uva.nl

23. Netherlands Organisation for Scientific Research Effect sizes comparison 7th and 9th grade AmsterdamSwitzerlandDifference Bokito0.30.00 TV guide0.410.32.09 Recipe0.560.26.30 Aroma jockey0.810.34.47 Basilisk0.100.04.06 Youth hostel0.370.21.16 Rouler0.510.46.05 King Clothes0.370.46.09 Phone ban0.340.40.06 Newsletter0.770.26.51 Vietnam0.370.46.09 The board0.270.71.44