Study Overview National Research Coordinators Meeting Amsterdam, October 16-19

NRC Meeting Amsterdam Oct 2006 IEA studies in the field 1971: First IEA Civic Education Study in 10 countries (part of six-subject study) 1999/2000: IEA CIVED –Assessment of 14-year-olds in 28 countries –Assessment of upper secondary students in 16 countries 2008/2009: IEA ICCS (more than 40 expressions of interest so far)

NRC Meeting Amsterdam Oct 2006 International Study Centre and partner institutions - 1 Australian Council for Educational Research (ACER), Melbourne/Australia –International Study Centre –Overall Project management –Test Development –Student Background Questionnaire –Scaling and Data analysis John Ainley (Project Coordinator) Wolfram Schulz (Research Director)

NRC Meeting Amsterdam Oct 2006 International Study Centre and partner institutions - 2 National Foundation for Educational Research (NFER), Slough/UK –National Contexts Survey –Student Perceptions Questionnaire –European Regional Module David Kerr (Associate Research Director)

NRC Meeting Amsterdam Oct 2006 International Study Centre and partner institutions - 3 Laboratorio di Pedagogia Sperimentale (LPS), Rome/Italy –School Questionnaire –Teacher Questionnaire Bruno Losito (Associate Research Director)

NRC Meeting Amsterdam Oct 2006 International Study Centre and partner institutions - 4 IEA Data Processing Center (DPC), Hamburg/Germany –Sampling and Weighting –Data Processing –Scaling and Analysis Falk Brese (Coordinator) Heiko Sibberns (TEG and PAC representative)

NRC Meeting Amsterdam Oct 2006 International Study Centre and partner institutions - 5 IEA Secretariat, Amsterdam/Netherlands –Translation Verification –Quality Monitoring Barbara Malak (Manager Membership Relations) Hans Wagemaker (Executive Director)

NRC Meeting Amsterdam Oct 2006 Project Management Joint Management Committee (JMC) Meets frequently (≥ monthly) with consortium staff Meets occasionally as a JMC Project Advisory Committee (PAC) 1 st meeting July nd meeting October 2006 IEA Secretariat IEA Technical Expert Group (TEG)

NRC Meeting Amsterdam Oct 2006 Purposes of study International comparison of outcomes of civic and citizenship education at start of 21 st century Provide trend data for comparison with CIVED 1999 Response to challenge in civic and citizenship education in changed contexts of democracy and participation

NRC Meeting Amsterdam Oct 2006 ICCS and CIVED Opportunity to review changes Extended framework and broadening of scope Different approach to teacher survey Regional modules

NRC Meeting Amsterdam Oct 2006 Research questions directed at exploration of Variations in student achievement Changes since 1999 Extent of interest and dispositions towards citizenship Student perceptions of recent threats and future developments Relationship of learning outcomes with curriculum, teaching and school organisation Relationship of learning outcomes with student background

NRC Meeting Amsterdam Oct 2006 Assessment Framework Outlines what will be assessed in ICCS Consists of two parts: –Civics and Citizenship framework –Contextual framework Current version to be discussed at this meeting

NRC Meeting Amsterdam Oct 2006 Instrumentation Student instruments comprising: –International (core) cognitive test (60 minutes) –International (core) student background questions (15 minutes) –International (core) student perceptions on civic- related issues (30 minutes) –Regional cognitive test module(s) (20-30 minutes) –Regional student questionnaire on civic-related issues (10-20 minutes) Teacher questionnaire (30 minutes) School questionnaire (30 minutes) National Contexts Survey (on-line)

NRC Meeting Amsterdam Oct 2006 Instrument development Establishing assessment framework Piloting of items in subset of participating countries (Feb-Mar 07) Review with experts and NRCs Field Trial of preliminary set of instruments (Oct-Nov 07) Review with experts and NRCs Final selection (Feb-Apr 08)

NRC Meeting Amsterdam Oct 2006 General survey design Population: Grade 8 (minimum average age of 13.5) Two-stage cluster samples in participating countries –PPS sampling of (at least 150) schools –Random selection of one classroom within each sampled school Random selection of teachers within sampled schools (no student-teacher link) teaching grade 8

NRC Meeting Amsterdam Oct 2006 Instrument administration Test administrators responsible for test and questionnaire sessions Student instrument administration: –International cognitive test –International background questionnaire –International perceptions questionnaire –Regional Cognitive test (optional) –Regional perceptions questionnaire (optional)

NRC Meeting Amsterdam Oct 2006 Translation and verification Instruments translated from (English) source version into national languages National adaptations reviewed in cooperation with international study centre Translations verified by language experts Final layout review

NRC Meeting Amsterdam Oct 2006 Field Operations Manuals will be developed to guide operations undertaken by national centres (coordination at school, test administration, coding, data entry) School and teacher questionnaires administered through school coordinators International control monitors visiting sub-samples of schools

NRC Meeting Amsterdam Oct 2006 Data Management and Scaling DPC creates international database (cleaning procedures) DPC computes sampling weights DPC provides item statistics DPC and ACER undertake scaling of data –IRT scaling model still to be decided (1-PL vs. 3- PL) –Plausible values for cognitive measures –IRT weighted likelihood scores for some questionnaire-based indices

NRC Meeting Amsterdam Oct 2006 Analysis and Reporting (Probably) Jackknife Repeated Replication (JRR) to provide correct standard errors Described scales (cognitive and some affective-behavioural) Descriptive tables with uni- and bivariate analyses Single and multi-level analyses

NRC Meeting Amsterdam Oct 2006 Reports International Report (June 2010) Regional Report(s) (June 2010) Technical Report (Nov 2010) –Instrument development –Field Operations –Sampling –Scaling –Reporting

NRC Meeting Amsterdam Oct 2006 International Database Student, teacher and school databases including: –ID variables –Item responses –Cognitive Scale(s) – Plausible Values –Affective-Behavioural Scales –Weights –JRR Indicators

NRC Meeting Amsterdam Oct 2006 Assistance to NRCs Item development workshop (this meeting) Data Management and Marker Training seminars (FT and MS) International Database Seminar (Nov 2010)

NRC Meeting Amsterdam Oct 2006 Timeline

NRC Meeting Amsterdam Oct 2006 Questions?