National adaptations to main survey instruments and layout verification National Research Coordinators Meeting Windsor, June 2008

NRC Meeting Windsor June 2008 Contents of presentation Principles of instrument adaptation Outline of adaptation review undertaken by the ISC National Adaptation Form (example) Problems identified in field trial Procedures for layout verification

NRC Meeting Windsor June 2008 Adapting main survey instruments All instruments have to be adapted –including those for English-speaking countries Adaptations should be kept to a minimum Adaptations have to be documented in National Adaptation Forms (NAF) There needs to be a consultation with the International Study Centre (ISC)

NRC Meeting Windsor June 2008 Types of adaptations There are parts of instruments that have to be adapted –Provision in translation notes and NAF –Usually in carets ( ) There may be the need for adapting other items –This should only be done when absolutely necessary! In some cases, NRCs may want to add items or categories –Should be discussed and agreed with ISC

NRC Meeting Windsor June 2008 Purpose of National adaptation forms Necessary information for translation verification Reference for layout verification Facilitates data processing –Documenting any deviation from international format Informs later data analysis

NRC Meeting Windsor June 2008 Process of adapting instruments Completing NAF in cooperation with translator and reviewer (> Version I) Consultation about NAF with ISC (> Version II) Documenting any changes from translation verification (> Version III) Documenting any changes from layout verification (> Version IV)

NRC Meeting Windsor June 2008 National adaptation forms (NAF) EXCEL format One general information spreadsheet Five (or six) adaptation forms –Cognitive test –Student background questionnaire –Student perceptions questionnaire –Teacher questionnaire –School questionnaire –Regional instrument (if applicable)

NRC Meeting Windsor June 2008 Columns in NAF - 1 Columns 1 to 3 should not be modified: –1: International question numbers –2: International question/category wording –3: International variable or code Column 4 to 5 may have to be completed –Only when adding items or categories!

NRC Meeting Windsor June 2008 Columns in NAF - 2 Columns 6 to 7 have to be completed when marked in yellow: –6: National wording of adaptation –7: English translation of this adaptation Column 8 is reserved for any comments NRC would like to make –Useful for consultation when implementing adaptations that are additional Column 9 is reserved for comments by the ISC during the consultation

NRC Meeting Windsor June 2008 Adding items or categories Not acceptable for cognitive test Requires discussion and agreement with ISC Only acceptable in exceptional circumstances MUST be documented in NAF! For countries that participated in CIVED no changes in adaptation/translation are permitted!

NRC Meeting Windsor June 2008 NAF example

NRC Meeting Windsor June 2008 International and national options International options will be included in NAF –NRCs should indicate whether they want to include the optional question(s) National options should be documented in the NAF –Please list questions (with English translation) that should be added –A special spreadsheet will be included for this purpose –National questionnaire parts should not be excessively long (otherwise follow-up sessions would be required)

NRC Meeting Windsor June 2008 Assistance with NAF More detailed information at Data Management Seminar in Hamburg NRCs should contact the ISC in case of any problems with the completion of NAF

NRC Meeting Windsor June 2008 Experience from the field trial - 1 Generally, procedures for adaptation review worked very well Adds important element to quality assurance procedures Provides important information for –Translation verification –Data processing –Data analysis

NRC Meeting Windsor June 2008 Experience from the field trial - 2 Some improvements that were identified during this process –Adaptation notes needs to be complemented based on experiences during adaptation review –There should be a space for recording adaptations to general directions –There should be a space for recording national items Some incomplete NAF were submitted which required re-submission –Reminder to national centres to document ALL adaptations

NRC Meeting Windsor June 2008 Layout verification Final step for instrument preparation before going into the field The layout verification is carried out at the International Study Centre (ACER) The layout verification is done AFTER translation verification –Assembly of test booklets once translation verification results have been received ALL countries are required to go through layout verification BEFORE printing their instruments

NRC Meeting Windsor June 2008 How to prepare for layout verification Finalise instruments based on advice from translation verification Assembling and layouting material according to sample instruments –available in PDF and WORD Prepare instruments in WORD and converting them into PDF

NRC Meeting Windsor June 2008 What to submit? Countries should submit PDF files of: –Six test booklets –Student questionnaire –Teacher questionnaire –School questionnaire Version III of the NAF –including any updates after translation verification

NRC Meeting Windsor June 2008 What we will do Compare the layout of national instruments with the sample instruments Generally, no further checks of translations or adaptations –if spotted we will advise NRCs Results of layout verification sent back to national centres (EXCEL spreadsheet)

NRC Meeting Windsor June 2008 After layout verification National centres should implement corrections as advised Sometimes the ISC may ask for re- submission in case of severe layout problems Final instruments should be submitted with data files to IEA DPC together with final NAF (version IV)

NRC Meeting Windsor June 2008 Experience from the field trial Generally, procedures worked well There were a few cases where instruments had to be re-submitted Due to the narrow timelines some instruments had not been submitted for layout verification –National centres should ensure that their national instruments go through ALL steps in the verification process (adaptation review, translation verification and layout verification)

NRC Meeting Windsor June 2008 Questions or comments?