Assessments for Monitoring and Improving the Quality of Education Kyongah Sang(KICE)

Assessments and Quality of Education Assessment is a systematic process of determining the extent to which instructional objectives are achieved by students It serves as a tool for monitoring and improving student learning and quality of education by… making curriculum priorities more explicit and focusing in instructional effort providing information about current status and progress of students’ achievement and quality of education holding educators accountable for results

Assessment and Quality of Education More and more Countries now conduct assessments to monitor and improve educational quality assessments for monitoring educational quality are often conducted at the national level the percentage of countries conducting at least one national assessments grow rapidly since 2000 the number of countries conducting national assessments was about 100 in 2013(EFA Global Monitoring Report, 2015)

Characteristics of the Assessments for Educational Quality Develop assessment based on curriculum/ standards Interpret assessment results based on performance standards Secure representativeness of examinees Conduct assessment periodically and equate tests to monitor achievement trends Collect educational context or background information

Methods to develop and implement Assessment for Educational Quality Develop test specification which guides the selection of items/tasks Set performance standards/ establishing cut-off scores Define target population & sampling Equate tests administered in different years Questionnaires Survey

Develop test specification which guides the selection of items/tasks Assessment for quality monitoring aims to measure the extent students achieve the intended learning outcomes Curriculum/ standards specify what should be taught and what students should learn Therefore, Assessments for monitoring educational quality are usually curriculum-based tests Test specifications help to ensure that assessments measure a representative sample of instructionally relevant tasks (content validity)

Develop test specification which guides the selection of items/tasks Role of test specification -Test specification guides the selection of test items and assessment tasks -It ensures assessments measure a representative sample of instructionally relevant tasks (content validity) -Select or develop test items according to the test specification ※ In next presentation, methods to develop items from test specification and standards will be discussed in details

Test specification used in NAEA (9th grade, Science, 2013, ) Item Content Behavior Domain Unit Standard memory understand apply … 1 Motion and energy Heat Explain thermal equilibrium, specific heat, thermal capacity ○ 2 Force and Motion Understand how to measure force and Calculate resultant force 3 Matter State of matter Understand solubility is the characteristics of matter 4 Structure of matter Understand matter is composed of elements

Set performance standards Assessment results are reported and interpreted based on fixed performance standards in Assessment for quality monitoring ex) basic, proficient, advance etc. Performance standards specify levels of performance that students are expected to achieve for the performance to be considered partially proficient, proficient, advance, etc. They provide basis for defining the level of performance(set cut-off scores) that is judged to be adequate or exemplary

Set performance standards Generic steps in setting performance standards 1 Select a representative panel 2 Choose a standard-setting method 3 Prepare Performance Level Descriptions 4 Train Participants to use the standard-setting method 5 Compile item ratings or other judgments from participants 6 Provide feedback and facilitate discussion 7 Compile Panelist ratings and obtain final performance standards

Standard setting in NAEA Performance levels are presented in 4 levels (establish 3 cut-off scores) Standard-setting panels - Subject experts who are mostly teachers(20 teachers for each subject) - Teachers from diverse regions - Teachers with experience of teaching target grade students(minimum 2 years) Standard-setting method : Modified Angoff

Standard setting in NAEA Performance Level Descriptions ADVANCED Level PROFICIENT BASIC Superior academic performance of required knowledge and skills Solid academic performance of required knowledge and skills Partial mastery of required knowledge and skills BELOW- BASIC Base Line of basic academic achievement

Test equating Assessments are developed based on same specification every year but there can be little difference in difficulty level It is necessary to equate tests implemented in each year to monitor achievement trends regardless of small changes in test difficulty Test equating refers to the statistical process of determining comparable scores on different forms of an exam(Same construct and the same levels of difficulty and reliability). It allows the scores from each test to be used interchangeably

Test equating in NAEA Equating design Non-equivalent group design using anchor items(external anchor items) Equating Method -Calibration Model for Multiple –Choice items : 3PL Model for Construct Response items : GPC Model -Scale transformation Extended Stocking-Lord Method

Test equating in NAEA Matrix sampling Test design (sample size : 2000 for each test form) Item clusters Test Forms

Define Target Population & Sampling Specify target population - Who should take part in assessment - Who will be excluded - for whom to approve test modification Census test vs. sample-based test - Census test : high-stake, emphasize accountability - Sample-based test : representative sample from pop. needed

Target Population & Sampling in NAEA Target population -All 9th,11th grade students learning National Curriculum (including private schools) Census test except assessments of Social Studies and Science for 9th graders Grades Subjects Contents 9th grade Korean language, Mathematics, English, Science, Social Studies** and Questionnaires All the content covered from the 7th to the 9th grade, 1st semester 11th grade Korean language, Mathematics, English, and Questionnaires All the content covered in the 10th grade

Questionnaires Survey in NAEA Collect context information in which learning takes place Analyze the factors influencing achievement obtain implications to improve learning School Features, School Climate, School Facilities - School-Level Curriculum Management SCHOOLS - Personal Background, Job Satisfaction - Teaching Activities Related to Each Subject TEACHERS* - Personal and Family Background - Extracurricular Activities - Learning Method and Attitude - Learning Experience related to Each Subject STUDENTS