Presentation on theme: "Contextual questionnaires Overview of PISA instruments"— Presentation transcript:
1 Contextual questionnaires Overview of PISA instruments PISA for DevelopmentTechnical Strand 1:Contextual questionnairesOverview of PISA instruments(constructs, indices and variables)EDU/DCD
2 PISA for Development Contextual questionnaires Purpose of PISA contextual questionnairesHow are they used?How they are developedWhat will PISA for Development seek to do?Questions for discussionThis presentation presents an overview of the PISA for Development initiative and it is meant to support discussions and engagement among development partners and country Ministry officials.In addition to this presentation, additional materials include:Narrative description of ProjectLogical Framework of ProjectTimeline presentation (PREZI and pdf file)Two-page summary brief
3 PISA for Development Purpose of PISA contextual questionnaires Student and school questionnaires are part of the core assessment design of PISACountries may also opt for additional questionnaires:Parents, Teachers, Health, Time Use, ICT familiarity, Career…Student questionnaires: ~ 30 minute* stand- alone document (students respond after test questions)School questionnaires: ~ 30 minute stand-alone document (school authority responds before or on day of testing)* Rotated questionnaires in 2012 to cover additional materialThis series of slides provide an overview of PISA – focusing on what it is designed to assess. Despite the widespread reference to the “rankings” in PISA, deeper knowledge of the actual PISA assessment is quite limited.
4 PISA for Development Purpose of PISA contextual questionnaires Contextualise the performance results in reading, mathematics and science*Allow analyses of equity issues across sub-groups and populationsProvide indications of possible policy interventionsSome information collected is used to scale cognitive assessment dataCountry-specific questions can be included for additional analysis* Additional domains such as financial literacy and problem-solvingThis series of slides provide an overview of PISA – focusing on what it is designed to assess. Despite the widespread reference to the “rankings” in PISA, deeper knowledge of the actual PISA assessment is quite limited.
5 PISA for Development Purpose of PISA contextual questionnaires Have been used in 74 countries/economiesIn more than 40 languagesPublicly availablePISA 2009 Student QuestionnairePISA 2012 Student QuestionnairePISA 2012 School QuestionnairePISA 2009 School QuestionnaireThis series of slides provide an overview of PISA – focusing on what it is designed to assess. Despite the widespread reference to the “rankings” in PISA, deeper knowledge of the actual PISA assessment is quite limited.
6 PISA Contextual Questionnaires How they are used This is the PISA Analytic Framework that guides data collection and analysisOutputs and Outcomes impact of learningPolicy Levers shape educational outcomesAntecedents contextualise or constrained policyIndividual learnerQuality and distribution of knowledge & skillsIndividual attitudes, engagement and behaviourSocio-economic background of learnersInstructional settingsQuality of instructional deliveryTeaching, learning practices and classroom climateStudent learning, teacher working conditionsLevels of an education systemThis slide presents the PISA Analytic Framework on which the data collection analysis is based.The PISA analytic framework includes distinct levels and domains through which to assess an educational system:4 levels: individual learner, instructional setting, schools and country or system level.3 domains on which PISA collects and reports on data are Outputs and Outcomes, Policy Levers and Antecedents.This basic framework will be reviewed as part of the PISA for Development pilot in order to address if and how it may need to be extended to include aspects that may be relevant (currently not in the framework) considering contexts in developing countries.Schools (other institutions)Output and performance of institutionsThe learning environment at schoolCommunity and school characteristicsCountry or systemSocial & economic outcomes of educationStructures, resource allocation and policiesNational educ., social and economic context
7 PISA Contextual Questionnaires How they are used This is how the PISA Analytic Framework is operationalised for data collection and analysisMain data collectionStudent Questionnaire, School Questionnaires and System Level*OutcomesProcessesAntecedentsStudentsGeneral recurring variables (all cycles)Domain recurring variables (main domain every 3 years)Extension variables (specific cycles)System-level dataClassroomsSchoolsCountryVariables included in instruments:This slide shows how the PISA Analytic Framework is operationalised through the data collection instruments.Extended data collection procedures will inform the pilot trial with regards to how to maintain common constructs for cross-national comparability (and reporting) with additional (enhanced) data collection.* Options include questionnaires geared for Teachers, Parents, Health, Time Use, ICT familiarity, Career
8 PISA Contextual Questionnaires System-level data collection Standards and procedures for data collection in PISAQuestionnaires for non-OECD economiesPanel of experts to complete some data collectionThese are examples of the types of information collected at the system level:Ratio: students / teaching staffSpending /studentInstruction time (by age)Teachers’ salaries (by education, experience)Evaluation & accountabilityEducational stratificationDecision-making at different levelsSchool choiceChanges in education policyOthers (…)Sample of relevant areas to address in developing contextsWithin-country disparitiesQuality, supply and distribution of educational materials and resources(Extra-)tuitionParental supportContractual arrangements of school teaching staff
9 PISA Contextual Questionnaires School questionnaires School resources, policies and practicesThis is the type of information collected through the school questionnaires, that can then be cross-referenced with other sourcesThese are the constructs that will be reviewed as part of the technical work of PISA for DevelopmentGrades servedPublic-PrivateFunding sourcesCommunity typeEnrollment (boys, girls)Average class sizeNon-native speakersGrouping/trackingTeachers, by qualificationsTeacher shortagesPhysical building limitationsInstructional materialsTeacher attitudes, expectations, relationsComputer/network/web accessAbsenteeism, disruptionsTeacher moraleStudent moraleTeaching environmentTime for special programmesNon-native speaker offeringsAdmissions requirementsAssessment typesUses of assessmentAssessment of teachersDecision making authorityExternal decision makers
10 PISA Contextual Questionnaires Student questionnaires Student attitudes, behaviours and approaches to learningGeneral and domain-related processesSense of belonging at schoolStudent-teacher relations at schoolDisciplinary climate in classesTeacher support in subject-specific classesOut-of-school activities (homework, etc...)Self and domain-related cognitionsAnxiety towards “study subject”Instrumental motivation to learn “study subject”Interest in and enjoyment of “study subject”Self-efficacy in “study subject”Self-concept in “study subject”Learning strategiesPreference for competitive learning situationsPreference for co-operative learning situationsControl strategiesPerseveranceOthersHealth status, Time use(…)These are examples of the types ofnon-cognitive outcomeson which PISA collects informationThese are the constructs that will be reviewed as part of the technical work of PISA for DevelopmentThis slide makes the link between current PISA instruments and the enhancements that will be addressed through the PISA for Development.
11 PISA Contextual Questionnaires Examples of analysis … How are questionnaire data used for relevant analysis and comparative policy insights?(a few examples from PISA results)
12 Examples of analysis Range of performance among non-OECD countries PISA Results for different yearsSource: OECD (2010), PISA 2009 Results. Volume 5, Table V.2.1ACER (2012), PISA 2009 Plus Results, Table A.2.GNI per capita from 2011, World Bank Indicators. See:Shanghai:
13 Schools with a similar socio-economic profile MexicoExamples of analysis School performance and socio-economic backgroundPISA 2009 ResultsSchool performance and students’ socio-economic background within schoolsPrivate schoolPublic school in rural areaPublic school in urban areaStudent performance and schools’ socio-economic backgroundSchools with a similar socio-economic profileStudent performanceThis slide shows PISA 2009 results for the lowest-performing OECD country – Mexico.These PISA 2009 results for Mexico show that even within “poor performers” that appear at the bottom of the “league tables”, PISA provides differentiation and relevant analysis.PISA provides results in terms of performance in reading, mathematics and science on the PISA scales. On this chart, the vertical axis (y-axis) shows increasing levels of performance as one goes from bottom to top. On the bottom, the x-axis, a measure of socio-economic advantage and disadvantage is shown – as one moves from left to right, the relative advantage of students increases. The bubbles shown are the schools that participated in PISA 2009 in Mexico and their results relative to both cognitive outcomes (in this case reading, vertical scale) and socio-economic status (horizontally along the bottom).The size of the bubbles represent the relative size of the students enrolled in the school. The dark blue bubbles, therefore, represent the relative position of the schools that participated in Mexico in PISA As you can see, the general tendency in terms of the relationship between socio-economic status of students in the schools and the average performance is correlated – the more advantaged students tends to result in higher average performance of schools. However, let’s take a closer look. (Animation).Among schools that have a relatively similar socio-economic profile of students, shown by the vertical band, there is a considerable amount of variation in performance, and as we move right towards more advantaged students in schools, we see the variation continues. Why is it that some schools have much higher performance averages with students from similar neighbourhoods?Looking at a cross-section of schools that have similar performance (horizontal band), even below the country average, we see that among these schools that can be considered to be “under-performing” there is also a considerable amount of variation in terms of the relative advantage and disadvantage of their students. Some of these schools, towards the left side of the chart, show results that are similar to schools with students from more privileged backgrounds (towards the right).You see that some of them do very well even by OECD standards while others do quite poorly. If Mexican schools would achieve what Mexican schools show can be achieved, Mexico’s overall performance would be significantly higher. Similarly, you can look at the schools where rich parents sent their children, all marked in dark bubbles which tells you that these are private schools. But not all of them are of good quality.AdvantagePISA Index of socio-economic backgroundDisadvantage
14 PISA Index of socio-economic background KyrgyzstanExamples of analysis School performance and socio-economic backgroundPISA 2009 ResultsSchool performance and students’ socio-economic background within schoolsPrivate schoolPublic school in rural areaPublic school in urban areaStudent performance and schools’ socio-economic backgroundStudent performanceOECD averageThis slide shows results for the partner country Kyrgyzstanthat participated in PISA 2006 and These results are from PISA 2009.This slide is the same type of figure as the previous one for Mexico. However, even when looking at these two very different countries – both low performing in terms of overall PISA results – there are several interesting points that the PISA data show:Even between countries that can be considered “low performing”, PISA results show striking differences in terms of the relationship between socio-economic background of students and how this is associated with student performance.PISA results also clearly underscore differences in variance within and between schools, as well as the relative performance of public, private and urban/rural schools. There is also clearly a correlation in terms of size of rural area schools compared to urban schools which also tend to be higher performing.Again, as in the case of Mexico, PISA data for Kyrgyzstan shows that even between schools that serve students from similar socio-economic backgrounds, there is a large variation in performance. And among schools that serve very different types of students, there can be similar results.AdvantagePISA Index of socio-economic backgroundDisadvantage
15 Examples of analysis Policy findings from contextual information Policies and practicesLearning climateDisciplineTeacher behaviourParental pressureTeacher-student relationshipsDealing with heterogeneityGrade repetitionPrevalence of trackingExpulsionsAbility grouping (all subjects)Standards /accountabilityNat. examinationStandardised testsPosting resultsGoverning schoolsSchool autonomy (content)Choice and competitionPrivate schoolsManaging resourcesPrioritising payStudent-staff ratiosLength of pre-schoolPolicySystemRSchoolEquityEExamples of analysis Policy findings from contextual informationPolicy briefs on these issues can be found in the PISA in Focus seriesLet me briefly summarise the influences that we have measured in PISA.
16 Commitment to universal achievement Goals, gateways, instructional systemsCapacity at point of deliveryIncentives and accountabilityResources where they yield mostA learning systemCoherenceBased on the analysis of cognitive outcomes and contextual informationExamples of analysis Policy insights using contextual informationLessons from PISA from successful education systems (equitable and improvers)Key Messages:- Findings from different cycles of PISA and trends over timeThis is the type of analysis that can be tailored for a particular country – Strong Performers/Successful reformers series of publications (show examples)The analysis for countries participating in PISA for Development will be tailored based on the relevance and policy priorities established by partners.Part of the value added will come from looking closely at how some of these and other policy issues actually present themselves in countries – and the pointers to possible policies and interventions.Original notes on slide:First, there is no question that most nations declare that education is important. But the test comes when these commitments are weighed against others. How do countries pay teachers, compared to other highly-skilled workers? How are education credentials weighed against other qualifications when people are being considered for jobs? Would you want your child to be a teacher? How much attention do the media pay to schools and schooling? What we have learned from PISA is that in high performing systems political and social leaders have persuaded citizens to make choices that show they value education more than other things.But placing a high value on education is only part of the equation. Another part is belief in the possibilities for all children to achieve success. In some countries, students are separated into different tracks at an early age, reflecting a notion shared by teachers, parents and citizens that only a subset of the nation’s children can or need to achieve world class standards. Our analysis shows that systems that track students in this way, based differing expectations for different destinations, tend to be fraught with large social disparities.By contrast, the best performing systems deliver strong and equitable learning outcomes across very different cultural and economic contexts. In Finland, Japan, Singapore, Shanghai-China and Hong Kong-China, parents, teachers and the public at large share the belief that all students are capable of achieving high standards and need to do so, and they provide great examples for how public policy can support the achievement of universal high standards.
17 PISA Contextual Questionnaires How they are developed ‘Crowd sourcing’ and collaborationPISA draws together leading expertise and institutions from participating countries to develop instruments and methodologies…… guided by governments on the basis of shared policy interestsCross-national relevance and transferability of policy experiencesEmphasis on validity across cultures, languages and systemsFrameworks built on well-structured conceptual understanding of assessment areas and contextual factorsContinuous review and updating of assessment frameworks (cycles)Guided by participating countries and various expert groups formed by leading international experts in different areas related PISATechnical Advisory Group, Reading Expert Group, Mathematics EG, Science EG, Questionnaire EG, and international contractorsSMEG (subject matter expert group)REG (reading)MEG (math)CPEG (collaborative problem solving)SEG (science)QEG (questionnaire)EL (environmental)FL (financial)How many people are involved?Major domains - 6 Core members and 8 Extended membersMinor domains - 2 Core members and 2 Extended members
18 PISA Contextual Questionnaires How they are developed PISA 2012 QuestionnairesStudent questionnaire: Rotated design (total of 65 questions – 25 core and 3 blocks of 27 questions)School questionnaire: 33 questions (5 additional questions on financial education at school)Parental questionnaire: 25 questionsICT questionnaire: 12 questionsEducation careers: 14 questionsPISA 2012 Mathematics FrameworkWhat mathematical content knowledge can we expect of 15-year-old students?What processes do students engage in when solving contextual mathematical problems, and what capabilities do we expect students to be able to demonstrate as their mathematical literacy grows?In what contexts is mathematical literacy able to be observed and assessed?Linkages between cognitive and non-cognitive outcomes
19 PISA Contextual Questionnaires How they are developed Purpose of the PISA Field Trial (for Main Study)Selection of cognitive test items to be included in the PISA Main Study instruments (e.g. countryXcountry DIF, gender DIF)Validation of psychometric equivalence of translated instruments (and adaptations from source versions) issues addressedIdentify constructs from contextual questionnaires (indices and variables) that are associated with performance for inclusion in the Main StudyDetermine cross-national validity of the questions in the questionnaireTechnical standards ensure that data from participating countries are internationally comparable
20 PISA Contextual Questionnaires What will PISA for Development seek to do? Adapt and enhance contextual instruments to “better fit” diverse contexts found in developing countries while maintaining the comparability with the main PISA scales and international resultsAdapt existing constructs (and indices and variables)Identify and introduce new constructs (indices and variables)Confirm and validate enhancements through field trials and main data collection in participating countries
21 Examples of variables and indices PISA Contextual Questionnaires What will PISA for Development seek to do?Examples of variables and indicesStudent QuestionnaireSection 1: ABOUT YOUSection 2: YOUR FAMILY AND YOUR HOMESection 3: YOUR READING ACTIVITIESSection 4: LEARNING TIMESection 5: YOUR SCHOOLSection 6: YOUR SCHOOL LESSONS AND SUBJECTSSection 7: YOUR STRATEGIES IN READING AND UNDERSTANDING TEXTSSection 8: YOUR VIEWS ON <BROAD SCIENCE>Section 9: CAREERS AND <BROAD SCIENCE>Section 10: YOUR MATHEMATICS EXPERIENCES
22 PISA Contextual Questionnaires Student questionnaires Examples of variables and indicesStudent QuestionnaireSECTION 2: YOUR FAMILY AND YOUR HOME2009-ST08: Family structure2009-ST09: Mother’s main job (ESCS, HISEI, BMMJ)2009-ST10: Mother’s education (ESCS, HISCED, PARED, MISCED)2009-ST11: Mother’s qualifications (ESCS, HISCED, PARED, MISCED)2009-ST12: Mother’s employment status (ESCS, HISEI, BMMJ)2009-ST13: Father’s main job (ESCS, HISEI, BMFJ)2009-ST14: Father’s education (ESCS, HISCED, PARED, FISCED)2009-ST15: Father’s qualifications (ESCS, HISCED, PARED, FISCED)2009-ST16: Father’s employment status (ESCS, HISEI, BFMJ)2009-ST17: Country of birth for student and parents (IMMIG)2009-ST18: Age at arrival in country of test2009-ST19: Language spoken at home2009-ST20: Home resources (ESCS, HOMEPOS, WEALTH, HEDRES, CULTPOS)2009-ST21: Family wealth (ESCS, HOMEPOS, WEALTH)2009-ST22: Books in home (ESCS, HOMEPOS)
23 PISA Contextual Questionnaires Student questionnaires – examples of indices PISA index of Economic, Social and Cultural Status(ESCS)Comparability across countries (not a country-specific income indicator)WEALTHHISEIHighest occupational status of parentsCULTPOSSPrincipal component AnalysisThe reliability of the index ranged from 0.41 to 0.81Scale indices are the variables constructed through the scaling of multiple items. Scaled using a weighted maximum likelihood estimate (WLE).Simple indices are the variables that are constructed through the arithmetic transformation or recoding of one or more items, inexactly the same way across assessments.The index of home possessions (HOMEPOS) comprises all items on the indices of WEALTH, CULTPOSS and HEDRES, as well as books in the home recoded into a four-level categorical variable (0-10 books, or books, or books, more than 500 books).The index of home possessions, obtained by asking students whether they had a desk at which they studied at home, a room of their own, a quiet place to study, educational software, a link to the Internet, their own calculator, classic literature, books of poetry, works of art (e.g. paintings), books to help them with their school work, a dictionary, a dishwasher, a DVD player or VCR, three other country-specific items and the number of cellular phones, televisions, computers, cars and books at home.HOMEPOSHome possessionsHEDRES# of books in homePAREDHighest educational level of parents
24 PISA Contextual Questionnaires Student questionnaires – examples of indices PISA index of Economic, Social and Cultural Status(ESCS)Index of family wealtha room of their own, a link to the Internet, a dishwasher (treated as a country-specific item), a DVD player, and three other country-specific items (some items in ST20), and their responses on the number of cellular phones, televisions, computers, cars and the rooms witha bath or shower (ST21).Index of cultural possessionsclassic literature, books of poetry and works of art (ST20)WEALTHCULTPOSSHEDRES# of books in homeIndex of home educational resourcesFor example, availability of a desk, a quiet place to study, a computer that students can use for schoolwork, educational software, books to help students, technical reference books and dictionary (some items in ST20)Principal component AnalysisThe reliability of the index ranged from 0.41 to 0.81Scale indices are the variables constructed through the scaling of multiple items. Scaled using a weighted maximum likelihood estimate (WLE).Simple indices are the variables that are constructed through the arithmetic transformation or recoding of one or more items, inexactly the same way across assessments.The index of home possessions (HOMEPOS) comprises all items on the indices of WEALTH, CULTPOSS and HEDRES, as well as books in the home recoded into a four-level categorical variable (0-10 books, or books, or books, more than 500 books).The index of home possessions, obtained by asking students whether they had a desk at which they studied at home, a room of their own, a quiet place to study, educational software, a link to the Internet, their own calculator, classic literature, books of poetry, works of art (e.g. paintings), books to help them with their school work, a dictionary, a dishwasher, a DVD player or VCR, three other country-specific items and the number of cellular phones, televisions, computers, cars and books at home.
25 2006-ST17: Self-efficacy in science (SCIEEFF) PISA Contextual Questionnaires Student questionnaires – examples of indicesSelf-efficacy in science and mathematics2006-ST17: Self-efficacy in science (SCIEEFF)2003-ST31: Self-efficacy in mathematics (MATHEFF) Stays the same in PISA 2012Relationship between constructs, indices and variables (questions)Students’ self-belief is another context variable that shows a strong correlation with student’s performance in the subjects. Similar to the level of engagement with the subjects, self-belief can be an important part of improving the performance. Self-belief is measured for mathematics and science in PISA 2003 and 2006, but not for reading in PISA 2009.Self-belief is measured in terms of self-efficacy (how much students believe in their own ability to handle tasks effectively and overcome difficulties) and self-concept (how much students believe in their own academic abilities).
26 Self-efficacy in science PISA Contextual Questionnaires Student questionnaires – examples of indicesSelf-efficacy in science2006-ST17: Self-efficacy in science (SCIEEFF)06-ST17Q42How easy do you think it would be for you to perform the following tasks on your own?(Please darken only one circle in each row.)I could do this easilyI could do this with a bit of effortI would struggle to do this on my ownI couldn’t do thisa)Recognise the science question that underlies a newspaper report on a health issue●1●2●3●4b)Explain why earthquakes occur more frequently in some areas than in othersc)Describe the role of antibiotics in the treatment of diseased)Identify the science question associated with the disposal of garbagee)Predict how changes to an environment will affect the survival of certain speciesf)Interpret the scientific information provided on the labels of food itemsg)Discuss how new evidence can lead you to change your understanding about the possibility of life on Marsh)Identify the better of two explanations for the formation of acid rainStudents’ self-belief is another context variable that shows a strong correlation with student’s performance in the subjects. Similar to the level of engagement with the subjects, self-belief can be an important part of improving the performance. Self-belief is measured for mathematics and science in PISA 2003 and 2006, but not for reading in PISA 2009.Self-belief is measured in terms of self-efficacy (how much students believe in their own ability to handle tasks effectively and overcome difficulties) and self-concept (how much students believe in their own academic abilities).
27 Instrumental motivation to learn science PISA Contextual Questionnaires Student questionnaires – examples of indicesInstrumental motivation to learn science2006-ST35: Instrumental motivation to learn science (INSTSCIE)06-ST35Q46How much do you agree with the statements below?(Please darken only one circle in each row.)Strongly agreeAgreeDisagre eStrongly disagreea)Making an effort in my science class(es) is worth it because this will help me in the work I want to do later on●1●2●3●4b)What I learn in my science class(es) is important for me because I need this for what I want to study later onc)I study science because I know it is useful for med)Studying science is worthwhile for me because what I learn will improve my career prospectse)I will learn many things in my science class(es) that will help me get a job
28 PISA for Development How will we do this? Identify constructs in need of adaptation and important constructs that are missingObtain input and advice from experts in the field (grounded in developing-country contexts)Obtain input and guidance from participating countriesDraw on existing work and empirical evidenceTest the constructs (indices and variables) to ensure comparability with PISA scales
29 Contextual questionnaires PISA for DevelopmentTechnical Strand 1:Contextual questionnaires
30 PISA for Development Questions for discussion Are the intentions and rationale of this strand of work clear and understood?Which components of the PISA questionnaires appear to demonstrate the greatest need of adaptation?Where do you see the largest hurdle to overcome?What existing studies and evidence should contribute to this strand of work?For example:- Capturing a wider range of socio-economic and cultural contextsInternational and national development of the PISA context questionnaires- Measuring the knowledge and skills of children not in school- Ensuring the relevance and validity of the skills that are measured- Balance between country-specific contexts and international comparability