Presentation is loading. Please wait.

Presentation is loading. Please wait.

PISA for Development Technical Strand 1: Contextual questionnaires Overview of PISA instruments (constructs, indices and variables) EDU/DCD.

Similar presentations


Presentation on theme: "PISA for Development Technical Strand 1: Contextual questionnaires Overview of PISA instruments (constructs, indices and variables) EDU/DCD."— Presentation transcript:

1 PISA for Development Technical Strand 1: Contextual questionnaires Overview of PISA instruments (constructs, indices and variables) EDU/DCD

2 PISA for Development Contextual questionnaires 2 Purpose of PISA contextual questionnaires How are they used? How they are developed What will PISA for Development seek to do? Questions for discussion

3 Student and school questionnaires are part of the core assessment design of PISA Countries may also opt for additional questionnaires: -Parents, Teachers, Health, Time Use, ICT familiarity, Career… Student questionnaires: ~ 30 minute* stand- alone document (students respond after test questions) School questionnaires: ~ 30 minute stand-alone document (school authority responds before or on day of testing) * Rotated questionnaires in 2012 to cover additional material PISA for Development Purpose of PISA contextual questionnaires

4 Contextualise the performance results in reading, mathematics and science* Allow analyses of equity issues across sub-groups and populations Provide indications of possible policy interventions Some information collected is used to scale cognitive assessment data Country-specific questions can be included for additional analysis * Additional domains such as financial literacy and problem-solving PISA for Development Purpose of PISA contextual questionnaires

5 Have been used in 74 countries/economies In more than 40 languages Publicly available PISA 2009 Student Questionnaire PISA 2012 Student Questionnaire PISA 2012 School Questionnaire PISA 2009 School Questionnaire PISA for Development Purpose of PISA contextual questionnaires

6 National educ., social and economic context Structures, resource allocation and policies Social & economic outcomes of education Community and school characteristics Student learning, teacher working conditions Socio-economic background of learners Antecedents contextualise or constrained policy The learning environment at school Teaching, learning practices and classroom climate Individual attitudes, engagement and behaviour Output and performance of institutions Quality of instructional delivery Quality and distribution of knowledge & skills Policy Levers shape educational outcomes Outputs and Outcomes impact of learning Individual learner Instructional settings Schools (other institutions) Country or system This is the PISA Analytic Framework that guides data collection and analysis Levels of an education system PISA Contextual Questionnaires How they are used

7 OutcomesProcessesAntecedents Students General recurring variables (all cycles) Domain recurring variables (main domain every 3 years) Extension variables (specific cycles) System-level data Classrooms Schools Country Main data collection Student Questionnaire, School Questionnaires and System Level* This is how the PISA Analytic Framework is operationalised for data collection and analysis Variables included in instruments: * Options include questionnaires geared for Teachers, Parents, Health, Time Use, ICT familiarity, Career PISA Contextual Questionnaires How they are used

8 PISA Contextual Questionnaires System-level data collection Ratio: students / teaching staff Spending /student Instruction time (by age) Teachers salaries (by education, experience) Evaluation & accountability Educational stratification Decision-making at different levels School choice Changes in education policy Others (…) Standards and procedures for data collection in PISA Questionnaires for non-OECD economies Panel of experts to complete some data collection 8 Within-country disparities Quality, supply and distribution of educational materials and resources (Extra-)tuition Parental support Contractual arrangements of school teaching staff Sample of relevant areas to address in developing contexts These are examples of the types of information collected at the system level:

9 Grades served Public-Private Funding sources Community type Enrollment (boys, girls) Average class size Non-native speakers Grouping/tracking Teachers, by qualifications Teacher shortages Physical building limitations Instructional materials Teacher attitudes, expectations, relations Computer/network/web access Absenteeism, disruptions Teacher morale Student morale Teaching environment Time for special programmes Non-native speaker offerings Admissions requirements Assessment types Uses of assessment Assessment of teachers Decision making authority External decision makers 9 School resources, policies and practices This is the type of information collected through the school questionnaires, that can then be cross-referenced with other sources These are the constructs that will be reviewed as part of the technical work of PISA for Development PISA Contextual Questionnaires School questionnaires

10 PISA Contextual Questionnaires Student questionnaires General and domain-related processes Sense of belonging at school Student-teacher relations at school Disciplinary climate in classes Teacher support in subject-specific classes Out-of-school activities (homework, etc...) Self and domain-related cognitions Anxiety towards study subject Instrumental motivation to learn study subject Interest in and enjoyment of study subject Self-efficacy in study subject Self-concept in study subject Learning strategies Preference for competitive learning situations Preference for co-operative learning situations Control strategies Perseverance Others Health status, Time use (…) 10 These are examples of the types of non-cognitive outcomes on which PISA collects information These are the constructs that will be reviewed as part of the technical work of PISA for Development Student attitudes, behaviours and approaches to learning

11 … How are questionnaire data used for relevant analysis and comparative policy insights? (a few examples from PISA results) 11 PISA Contextual Questionnaires Examples of analysis

12 12 Examples of analysis Range of performance among non-OECD countries PISA Results for different years

13 Student performance Advantage PISA Index of socio-economic background Disadvantage School performance and students socio-economic background within schools Student performance and schools socio-economic background Private school Public school in rural area Public school in urban area Mexico Schools with a similar socio-economic profile PISA 2009 Results Examples of analysis School performance and socio-economic background

14 Student performance Advantage PISA Index of socio-economic background Disadvantage Private school Public school in rural area Public school in urban area OECD average Examples of analysis School performance and socio-economic background School performance and students socio-economic background within schools Student performance and schools socio-economic background Kyrgyzstan PISA 2009 Results

15 Examples of analysis Policy findings from contextual information Policies and practices Learning climate Discipline Teacher behaviour Parental pressure Teacher-student relationships Dealing with heterogeneity Grade repetition Prevalence of tracking Expulsions Ability grouping (all subjects) Standards /accountability Nat. examination Standardised tests Posting results Governing schools School autonomy (content) Choice and competition Private schools Managing resources Prioritising pay Student-staff ratios Length of pre-school Policy System R School R Equity E Policy briefs on these issues can be found in the PISA in Focus series

16 Examples of analysis Policy insights using contextual information Lessons from PISA from successful education systems ( equitable and improvers) Based on the analysis of cognitive outcomes and contextual information

17 17 PISA Contextual Questionnaires How they are developed Crowd sourcing and collaboration –PISA draws together leading expertise and institutions from participating countries to develop instruments and methodologies… …guided by governments on the basis of shared policy interests Cross-national relevance and transferability of policy experiences –Emphasis on validity across cultures, languages and systems –Frameworks built on well-structured conceptual understanding of assessment areas and contextual factors Continuous review and updating of assessment frameworks (cycles) –Guided by participating countries and various expert groups formed by leading international experts in different areas related PISA –Technical Advisory Group, Reading Expert Group, Mathematics EG, Science EG, Questionnaire EG, and international contractors

18 Student questionnaire: Rotated design (total of 65 questions – 25 core and 3 blocks of 27 questions) School questionnaire: 33 questions (5 additional questions on financial education at school) Parental questionnaire: 25 questions ICT questionnaire: 12 questions Education careers: 14 questions PISA Contextual Questionnaires How they are developed PISA 2012 Questionnaires Linkages between cognitive and non-cognitive outcomes

19 Selection of cognitive test items to be included in the PISA Main Study instruments (e.g. countryXcountry DIF, gender DIF) Validation of psychometric equivalence of translated instruments (and adaptations from source versions) issues addressed Identify constructs from contextual questionnaires (indices and variables) that are associated with performance for inclusion in the Main Study Determine cross-national validity of the questions in the questionnaire Technical standards ensure that data from participating countries are internationally comparable PISA Contextual Questionnaires How they are developed Purpose of the PISA Field Trial (for Main Study)

20 Adapt and enhance contextual instruments to better fit diverse contexts found in developing countries while maintaining the comparability with the main PISA scales and international results Adapt existing constructs (and indices and variables) Identify and introduce new constructs (indices and variables) Confirm and validate enhancements through field trials and main data collection in participating countries PISA Contextual Questionnaires What will PISA for Development seek to do?

21 Student Questionnaire Section 1: ABOUT YOU Section 2: YOUR FAMILY AND YOUR HOME Section 3: YOUR READING ACTIVITIES Section 4: LEARNING TIME Section 5: YOUR SCHOOL Section 6: YOUR SCHOOL LESSONS AND SUBJECTS Section 7: YOUR STRATEGIES IN READING AND UNDERSTANDING TEXTS Section 8: YOUR VIEWS ON Section 9: CAREERS AND Section 10: YOUR MATHEMATICS EXPERIENCES Examples of variables and indices PISA Contextual Questionnaires What will PISA for Development seek to do?

22 Student Questionnaire SECTION 2: YOUR FAMILY AND YOUR HOME 2009-ST08: Family structure 2009-ST09: Mothers main job (ESCS, HISEI, BMMJ) 2009-ST10: Mothers education (ESCS, HISCED, PARED, MISCED) 2009-ST11: Mothers qualifications (ESCS, HISCED, PARED, MISCED) 2009-ST12: Mothers employment status (ESCS, HISEI, BMMJ) 2009-ST13: Fathers main job (ESCS, HISEI, BMFJ) 2009-ST14: Fathers education (ESCS, HISCED, PARED, FISCED) 2009-ST15: Fathers qualifications (ESCS, HISCED, PARED, FISCED) 2009-ST16: Fathers employment status (ESCS, HISEI, BFMJ) 2009-ST17: Country of birth for student and parents (IMMIG) 2009-ST18: Age at arrival in country of test 2009-ST19: Language spoken at home 2009-ST20: Home resources (ESCS, HOMEPOS, WEALTH, HEDRES, CULTPOS) 2009-ST21: Family wealth (ESCS, HOMEPOS, WEALTH) 2009-ST22: Books in home (ESCS, HOMEPOS) PISA Contextual Questionnaires Student questionnaires Examples of variables and indices

23 Comparability across countries (not a country- specific income indicator) PISA index of Economic, Social and Cultural Status (ESCS) HISEI Highest occupational status of parents PARED Highest educational level of parents HOMEPOSHome possessions WEALTH CULTPOSS HEDRES # of books in home PISA Contextual Questionnaires Student questionnaires – examples of indices

24 WEALTH CULTPOSS HEDRES # of books in home Index of family wealth a room of their own, a link to the Internet, a dishwasher (treated as a country-specific item), a DVD player, and three other country-specific items (some items in ST20), and their responses on the number of cellular phones, televisions, computers, cars and the rooms with a bath or shower (ST21). Index of cultural possessions classic literature, books of poetry and works of art (ST20) Index of home educational resources For example, availability of a desk, a quiet place to study, a computer that students can use for schoolwork, educational software, books to help students, technical reference books and dictionary (some items in ST20) PISA index of Economic, Social and Cultural Status (ESCS) PISA Contextual Questionnaires Student questionnaires – examples of indices

25 2006-ST17: Self-efficacy in science (SCIEEFF) 2003-ST31: Self-efficacy in mathematics (MATHEFF) Self-efficacy in science and mathematics PISA Contextual Questionnaires Student questionnaires – examples of indices Relationship between constructs, indices and variables (questions) Stays the same in PISA 2012

26 06-ST17 Q42How easy do you think it would be for you to perform the following tasks on your own? (Please darken only one circle in each row.) I could do this easily I could do this with a bit of effort I would struggle to do this on my own I couldnt do this a) Recognise the science question that underlies a newspaper report on a health issue b) Explain why earthquakes occur more frequently in some areas than in others c) Describe the role of antibiotics in the treatment of disease d) Identify the science question associated with the disposal of garbage e) Predict how changes to an environment will affect the survival of certain species f) Interpret the scientific information provided on the labels of food items g) Discuss how new evidence can lead you to change your understanding about the possibility of life on Mars h) Identify the better of two explanations for the formation of acid rain Self-efficacy in science PISA Contextual Questionnaires Student questionnaires – examples of indices 2006-ST17: Self-efficacy in science (SCIEEFF)

27 2006-ST35: Instrumental motivation to learn science (INSTSCIE) 06-ST35 Q46How much do you agree with the statements below? (Please darken only one circle in each row.) Strongly agreeAgree Disagre e Strongly disagree a) Making an effort in my science class(es) is worth it because this will help me in the work I want to do later on b) What I learn in my science class(es) is important for me because I need this for what I want to study later on c) I study science because I know it is useful for me d) Studying science is worthwhile for me because what I learn will improve my career prospects e) I will learn many things in my science class(es) that will help me get a job Instrumental motivation to learn science PISA Contextual Questionnaires Student questionnaires – examples of indices

28 Identify constructs in need of adaptation and important constructs that are missing Obtain input and advice from experts in the field (grounded in developing-country contexts) Obtain input and guidance from participating countries Draw on existing work and empirical evidence Test the constructs (indices and variables) to ensure comparability with PISA scales PISA for Development How will we do this?

29 PISA for Development Technical Strand 1: Contextual questionnaires

30 Are the intentions and rationale of this strand of work clear and understood? Which components of the PISA questionnaires appear to demonstrate the greatest need of adaptation? Where do you see the largest hurdle to overcome? What existing studies and evidence should contribute to this strand of work? PISA for Development Questions for discussion


Download ppt "PISA for Development Technical Strand 1: Contextual questionnaires Overview of PISA instruments (constructs, indices and variables) EDU/DCD."

Similar presentations


Ads by Google