1. Assessments for Monitoring and Improving the Quality of Education
Kyongah Sang(KICE)

2. Assessments and Quality of Education
Assessment is a systematic process of determining the extent to which instructional objectives are achieved by students
It serves as a tool for monitoring and improving student learning and quality of education by…
making curriculum priorities more explicit and focusing in instructional effort
providing information about current status and progress of students’ achievement and quality of education
holding educators accountable for results

3. Assessment and Quality of Education
More and more Countries now conduct assessments to monitor and improve educational quality
assessments for monitoring educational quality are often conducted at the national level
the percentage of countries conducting at least one national assessments grow rapidly since 2000
the number of countries conducting national assessments was about 100 in 2013(EFA Global Monitoring Report, 2015)

4. Characteristics of the Assessments for Educational Quality
Develop assessment based on curriculum/ standards
Interpret assessment results based on performance standards
Secure representativeness of examinees
Conduct assessment periodically and equate tests to monitor achievement trends
Collect educational context or background information

5. Methods to develop and implement Assessment for Educational Quality
Develop test specification which guides the selection of items/tasks
Set performance standards/ establishing cut-off scores
Define target population & sampling
Equate tests administered in different years
Questionnaires Survey

6. Develop test specification which guides the selection of items/tasks
Assessment for quality monitoring aims to measure the extent students achieve the intended learning outcomes
Curriculum/ standards specify what should be taught and what students should learn
Therefore, Assessments for monitoring educational quality are usually curriculum-based tests
Test specifications help to ensure that assessments measure a representative sample of instructionally relevant tasks (content validity)

7. Develop test specification which guides the selection of items/tasks
Role of test specification -Test specification guides the selection of test items and assessment tasks -It ensures assessments measure a representative sample of instructionally relevant tasks (content validity) -Select or develop test items according to the test specification
※ In next presentation, methods to develop items from test specification and standards will be discussed in details

8. Test specification used in NAEA (9th grade, Science, 2013, )
Item
Content
Behavior
Domain
Unit
Standard
memory
understand
apply … 1
Motion and energy
Heat
Explain thermal equilibrium, specific heat, thermal capacity ○ 2
Force and Motion
Understand how to measure force and Calculate resultant force 3
Matter
State of matter
Understand solubility is the characteristics of matter 4
Structure of matter
Understand matter is composed of elements

9. Set performance standards
Assessment results are reported and interpreted based on fixed performance standards in Assessment for quality monitoring ex) basic, proficient, advance etc.
Performance standards specify levels of performance that students are expected to achieve for the performance to be considered partially proficient, proficient, advance, etc.
They provide basis for defining the level of performance(set cut-off scores) that is judged to be adequate or exemplary

10. Set performance standards
Generic steps in setting performance standards 1
Select a representative panel 2
Choose a standard-setting method 3
Prepare Performance Level Descriptions 4
Train Participants to use the standard-setting method 5
Compile item ratings or other judgments from participants 6
Provide feedback and facilitate discussion 7
Compile Panelist ratings and obtain final performance standards

11. Standard setting in NAEA
Performance levels are presented in 4 levels (establish 3 cut-off scores)
Standard-setting panels - Subject experts who are mostly teachers(20 teachers for each subject) - Teachers from diverse regions - Teachers with experience of teaching target grade students(minimum 2 years)
Standard-setting method : Modified Angoff

12. Standard setting in NAEA
Performance Level Descriptions
ADVANCED
Level
PROFICIENT
BASIC
Superior academic performance of required knowledge and skills
Solid academic performance of required knowledge and skills
Partial mastery of required knowledge and skills
BELOW- BASIC
Base Line of basic academic achievement

13. Test equating
Assessments are developed based on same specification every year but there can be little difference in difficulty level
It is necessary to equate tests implemented in each year to monitor achievement trends regardless of small changes in test difficulty
Test equating refers to the statistical process of determining comparable scores on different forms of an exam(Same construct and the same levels of difficulty and reliability). It allows the scores from each test to be used interchangeably

14. Test equating in NAEA
Equating design Non-equivalent group design using anchor items(external anchor items)
Equating Method -Calibration Model for Multiple –Choice items : 3PL Model for Construct Response items : GPC Model -Scale transformation Extended Stocking-Lord Method

15. Test equating in NAEA Matrix sampling Test design
(sample size : 2000 for each test form)
Item clusters
Test Forms

16. Define Target Population & Sampling
Specify target population - Who should take part in assessment - Who will be excluded - for whom to approve test modification
Census test vs. sample-based test - Census test : high-stake, emphasize accountability - Sample-based test : representative sample from pop. needed

17. Target Population & Sampling in NAEA
Target population -All 9th,11th grade students learning National Curriculum (including private schools)
Census test except assessments of Social Studies and Science for 9th graders
Grades
Subjects
Contents
9th grade
Korean language,
Mathematics, English, Science, Social Studies** and Questionnaires
All the content covered from the 7th to the 9th grade, 1st semester
11th grade
Korean language, Mathematics, English, and Questionnaires
All the content covered
in the 10th grade

18. Questionnaires Survey in NAEA
Collect context information in which learning takes place
Analyze the factors influencing achievement
obtain implications to improve learning
School Features, School Climate, School Facilities
- School-Level Curriculum Management
SCHOOLS
- Personal Background, Job Satisfaction
- Teaching Activities Related to Each Subject
TEACHERS*
- Personal and Family Background
- Extracurricular Activities
- Learning Method and Attitude
- Learning Experience related to Each Subject
STUDENTS