1. Assessment Framework and Test Blueprint
Margaret Wu

2. Development of a Framework and Test Blueprint Draft items
Item Development
Development of a Framework and Test Blueprint
Draft items
Item panelling (shredding!)
Iterative process:
Draft items to illustrate, and clarify and sharpen up framework. Framework to guide item development.

3. Framework and Test Blueprint-1
Clearly identify
‘Why’ you are assessing (Purpose)
‘Whom’ to assess (Population)
‘What’ to assess (Construct domain)
What need to be reported
Overall score
Subscores?
For individual students or for cohorts (by district, gender, school)?
Define parameters for the test,e.g.:
Duration of the test and test administration procedures
Scoring/marking constraints; item formats
Other issues: security, feedback.

4. Specifying the Purpose
How will the results be used?
Determine pass/fail, satisfactory/unsatisfactory
Award prizes
Provide diagnostic information
Compare students
Set standards
Provide information to policy makers
Who will use the information?
Teachers, parents, students, managers, politicians

5. Specifying the Population
Grade, age level. In an industry. Profession. Ethnicity/Culture/Language issues. Gender.
Notion of population and sample.
Sampling method: random, convenience
Size of sample
Validity of test results could depend on the population/sample you are assessing

6. Specifying the Construct Domain - Examples
Familiarity with sport:
What is meant by ‘sport’? Include “Gym”? “Taichi?” “Gymnastics”? In Australian contexts?
Define Problem Solving:
As viewed in the workforce
Workforce competencies
As viewed in the cognitive sciences
Cognitive processes: decoding, reasoning, domain specific knowledge
Interpersonal skills
Negotiation/conflict resolution skills
Leadership skills
Work with people from diverse backgrounds

7. Specifying the Construct Domain – Examples
Achievement domains:
Content oriented:
Number, measurement, data, algebra
Competency oriented:
Conceptual understanding, procedural knowledge, problem solving
Taxonomy of educational objectives (Bloom’s taxonomy of learning outcomes): cognitive and affective.

8. Blooms’ Taxonomy - cognitive
Knowledge
Comprehension
Application
Analysis
Synthesis
Evaluation

10. Considerations in defining the Construct of a test
Validity Consideration
Does the construct cover what the test is claimed to be assessing?
E.g., language proficiency: speaking, listening, reading, writing
Measurement Consideration
How well the specifications for a construct “hang together” to provide meaningful scores?
The idea of “unidimensionality”
On-balance judgement
Boundaries are never clear

11. Test Blueprint
Sufficiently detailed so that test developers can work from these specifications.
Range of difficulty
Target reliability
Item format.
Weights of sub-domains
Test administration procedures
Timing, equipment, resources
Marking requirements

12. Test Blueprint – example (PISA Reading)
Aspect
% of test %
constructed
% MC
Retrieving information 20 7 13
Broad understanding
Interpretation 30 11 19
Reflecting on content 15 10 5
Reflecting on form
Total
100

13. Report by individual student
What will be reported
Any subscore
By content domains. If so, then there needs to be sufficient number of items for each content domain
Report by individual student
Need to work out measurement error: the accuracy with which each student is measured.
Typically, we need a very long test to measure individuals accurately.
This has an impact on the test length.
Report by subgroups
Ensure that sample size is sufficient to report by subgroups
Report possible factors related to achievement
Need to collect student background information

14. Define parameters for the test
Duration of the test
Has an impact on measurement error
If the purpose of the test is for cohort reporting, the tests can be shorter.
Total number of items needed
How many items are needed to cover the content domain?
As many as possible.
The use of rotated test booklet design
Allow sufficient number of items to cover the content domain
Prevent cheating
Moderate fatigue effect
How many open-ended/constructed items
The need to hire markers
The cost of marking

15. Uses of Frameworks & Blueprints
To guide item development
Don’t ignore specifications. Cross-check with specs constantly.
To ensure that there is a clear and well-defined construct that can be stable from one testing occasion to another.
Different item writing team
Parallel tests
To establish test validity
Design test -> Administer test -> Process test results -> Interpret test results
The last stage of interpreting test results relies on the test specifications (test blueprints) to establish what is being tested.

16. Adapting/borrowing test blueprints from other assessment projects
Assessment framework may not suit – different purposes of assessment.
e.g., differences between PISA and TIMSS (curriculum based or competency based)
Identify learning goals, and not just what is taught
Look beyond the textbook
May avoid rote learning
Avoid the possibility of students memorising test items
Talk to stakeholders
Policy makers
Employers
Educators

17. Using rotated booklet design to cover content areas and counteract fatigue effect
Block 1
Block 2
Block 3 1 C1 C2 C4 2 C3 C5 3 C6 4 C7 5 6 7

18. An example of rotated test booklet design
Block 1
Block 2
Block 3 1 C1 C2 C4 2 C3 C5 3 C6 4 C7 5 6 7