1. IATEFL/TEASIG, Innsbruck 2011 READING LITERACY OF AUSTRIAN SCHOOL LEAVERS: BETWEEN PISA AND "MATURA” Irene Thelen-Schaefer, BIFIE Wien

2. IATEFL/TEASIG, Innsbruck 2011 Presentation overview  The study  Theoretical background  The research questions  The instruments  Operationalisation  Results  Findings  Limitations  Future research

3. IATEFL/TEASIG, Innsbruck 2011 Background to the study  An interest in researching the predictive potential in PISA reading tests

4. IATEFL/TEASIG, Innsbruck 2011 Cross-cultural assessment and translation PISA = Programme for international student assessment  In order to make a comparison of competence possible  Adaptation of tests across borders and cultures  Set the specific psychological construct into the right context  Exclude cultural bias

5. IATEFL/TEASIG, Innsbruck 2011 The Study  Background test: Reading in German (L1) German translation of PISA reading literacy tasks 2009  Reading in English (L2) English version PISA reading literacy tasks 2009  Austrian standardised school leaving exam (“Matura”)  Research focus the comparability of items the predictability of L2 reading proficiency

6. IATEFL/TEASIG, Innsbruck 2011 Theoretical background 1: The skill reading L1 reading and L2 reading  Are we testing the same thing?  Same basic cognitive comprehension process across L1 and L2 contexts BUT distinctive features in L2 reading:  Language proficiency (“language threshold“)  L2 processing skills  Background knowledge

7. IATEFL/TEASIG, Innsbruck 2011 Theoretical background 2: The learner Learning to read: L1/L2 differences L1 Learner  Young learner  Oral skills developed before starting reading  Learning to read and write at the same time  Large knowledge of linguistic structures and vocabulary L2 Learner  Cognitively mature  They are L1 literate, but: they have not developed L2 oral comprehension yet  different starting point in L2 reading  L2 reading processes occur in a dual- language system

8. IATEFL/TEASIG, Innsbruck 2011 Theoretical background 3: The processes Lower level processes Automatisation is essential  Automatisation of word recognition  Speed of fixation  Working memory  Rauding (Reading + auding; Carver, 1984) Higher level processes  Motivation / purpose of reading  Two levels of understanding  “a text model of comprehension“  “a situation model of reader interpretation“; (Grabe, 2009)  Strategies  Background knowledge (Schema theory)

9. IATEFL/TEASIG, Innsbruck 2011 The Studys Research Questions 1. Can L1 reading results predict L2 reading results? 2. Can reading results from PISA English predict results of the “Matura” English L2?

10. IATEFL/TEASIG, Innsbruck 2011 The Instruments TestsPISA GermanPISA English"Matura" No of tasks334 Time45 min 50 min Test methodsMCQ; Text answer + justif. MCQ; Text answer + justif. MCQ; MM; T/F/NG; NF No of items111825 Student feedback questionnaire

11. IATEFL/TEASIG, Innsbruck 2011 Operationalisation Participating schools  Allgemeinbildende Höhere Schulen (AHS; grammar schools)  Urban and rural areas (Vienna, Linz, Admont, Hallein, Reutte, Villach…)  Public and private schools  Different types of AHS BORG, BG, BRG, WIKU Different branches (foreign languages, science, arts, technology…)

12. IATEFL/TEASIG, Innsbruck 2011 Operationalisation Test takers (TT)  18-year-old students  50 females, 51 males  101 TT completed all three tests  96 questionnaires returned Teachers as test administrators  trained test administrators  precise instructions given

13. IATEFL/TEASIG, Innsbruck 2011 (Provisional) Results Tests compared PISA German PISA English “Matura“ Mean8.547.8517.43 Mean %77.36%71.36%69.72% Std. Deviation 1.451.653.91 Mode9820

14. IATEFL/TEASIG, Innsbruck 2011 (Provisional) Results All three tests show that  The test takers did well on the tests (FV, skew...)  The sample is relatively homogeneous (SD)  Normal distribution on the PISA English test and the “Matura”

15. IATEFL/TEASIG, Innsbruck 2011 (Provisional) Results - Facility values FV %1234567891011 PISA G79.263.490.1 35.697.093.183.258.474.390.1 PISA E80.258.487.169.316.894.186.173.358.478.383.2 Findings FV equal or slightly better in PISA German than in PISA English Exceptions: items 4 and 5 The PISA tasks

16. IATEFL/TEASIG, Innsbruck 2011 (Provisional) Results – Item analysis Item 4 FVCorrected Item- total Correlation Cronbach‘s Alpha if Item Deleted PISA German 90.1%.058.265 PISA English 69.3%.279.281  Possible reasons: technical language  Frequent reasoning for test difficulty given in questionnaires by TT: (lack of) vocabulary

17. IATEFL/TEASIG, Innsbruck 2011 (Provisional) Results - Item analysis Item 5 Possible reasons:  Test method  Aspect targeted (“access and retrieve”) – answer to item 5 is after answer to item 6 FVCorrected Item-total Correlation Cronbach‘s Alpha if Item Deleted PISA German35.3%.050.278 PISA English16.7%.108.358

18. IATEFL/TEASIG, Innsbruck 2011 (Provisional) Results – Item analysis Item 10 Corrected Item-total Correlation Cronbach‘s Alpha if Item Deleted PISA German.307.126 PISA English.054.379 Possible reasons:  Partial credit in English only  Possible answers for partial credit are included in the “incorrect”-answer key in German Item 10 is the same in both languages re translation, but: it is the best item in PISA German and the worst in PISA English

19. IATEFL/TEASIG, Innsbruck 2011 (Provisional) Results - Reliability and discrimination PISA tasks  Low coefficients on both reliability and discrimination  German: eight items discriminate positively  English: all items discriminate positively; but only item 4 is above.25 (at.279)  These plus the number of items might be the reasons for the low Cronbach’s alpha coefficients on the PISA tasks PISA GermanPISA English“Matura” Cronbach‘s Alpha.270.371.723

20. IATEFL/TEASIG, Innsbruck 2011 (Provisional) Findings - Item analysis The “Matura”  Problematic items mainly in one test format: T/F/NG Possible reasons:  Difficult for TT to distinguish between F and NG  This test format has now been replaced by T/F justification.

21. IATEFL/TEASIG, Innsbruck 2011 (Provisional) Results - Correlations PISA GermanPISA English.454** “Matura“.268**.383** Findings Empirical data show that research question 1 on a low possible overlap between reading in L1 and L2 can be stated. Surprisingly, TT did nearly as well on the PISA English tasks than on the PISA German tasks. **. Correlation is significant at the 0.01 level (2-tailed).

22. IATEFL/TEASIG, Innsbruck 2011 Possible impact on the results  Motivation  Unknown test methods in the PISA tasks  Order of items  Partial credit in PISA English (item 10)  Tests target different purposes or aspects  Instructions  Examples

23. IATEFL/TEASIG, Innsbruck 2011 Further steps to be conducted  Further item analysis, incl. factor analysis PISA English – “Matura”  Correlations of test results and questionnaire items, e.g. Do avid readers have higher scores?

24. IATEFL/TEASIG, Innsbruck 2011 Limitations  Only 11 items in the PISA tasks  Items low reliability values  Possible memory effect PISA English – PISA German  Due to time pressure no counter-balanced design possible

25. IATEFL/TEASIG, Innsbruck 2011 Ideas for further research Longitudinal study as a diagnostic instrument: Can reading results from the PISA reading literacy test taken by 16-year-olds in their L2 predict the ”Matura“ reading results?  Diagnostic purpose  Advice for students and parents

26. IATEFL/TEASIG, Innsbruck 2011 List of references Alderson, J.C. (1984). Reading in a foreign language: a reading problem or a language problem? In J. C. Alderson and A. H. Urquhart (eds.). Reading in a foreign language. London: Longman. Alderson, J.C. (2000). Assessing reading. Cambridge, UK: Cambridge University Press. Bernhardt, E. (2005). Progress and procrastination in second language reading. Annual Review of Applied Linguistics 25, 133-150. Carver, R. (1984). Rauding theory predictions of amount comprehended under different purposes and speed reading conditions. Reading Research Quarterly 19 (2), 205-218. Grabe, W. (2009). Reading in a second language. Cambridge, UK: Cambridge University Press. Koda, K. (2005). Insights into second language reading. Cambridge, UK: Cambridge University Press. Stanovich, K. (1980). Toward an interactive-compensatory model of individual differences in the development of reading fluency. Reading Research Quarterly 16 (1), 32-71. Walter, C. (2008). Phonology in second language reading: not an optional extra. TESOL QUARTERLY 42 (3), 455-474.

27. IATEFL/TEASIG, Innsbruck 2011 THANK YOU FOR YOUR ATTENTION! CONTACT ADDRESS i.thelen-schaefer@bifie.at