1. The Item-Writing Process for ACCESS for ELLs™ Online Involvement by ESL Teachers
Jim Bauman
Center for Applied Linguistics
Washington, DC
January 2005
Illinois 28th Annual Statewide Conference for Teachers Serving Linguistically and Culturally Diverse Students 1

2. Presentation Outline The WIDA Assessment Framework
Structure of ACCESS for ELLs™
The Item Writing Course 2

3. WIDA’s Foundation Professional Development Improved teaching
Standards
Assessment Accountability
Higher levels
of learning
Clear expectations for students and schools Motivation to work hard
Adapted from:
National Research Council, 1999
Testing, Teaching, and Learning:
A Guide for States and School Districts 3

4. WIDA Enhanced Assessment System
ELP Standards Classroom Assessment
ELP Standards Large-scale Assessment
State Content
Standards
Lesson Plan
Specifications
Specifications
Assessment Framework
Test Blueprint
Standards Task Blueprint
Classroom Assessments
English Language Proficiency Test
(ACCESS for ELLS)
Alternate Assessment
of Academic Achievement
(Alternate ACCESS) 4

5. Organization of the WIDA Standards
Frameworks for Classroom & Large-Scale Assessment (2)
English Language Proficiency Standards (5) SI LA MA SC SS
Language Domains (4)
Grade Level Clusters (4) L R W S
Language Proficiency Levels (5)
K-2
3-5
6-8
9-12
Model Performance Indicators (>800)
Model PIs are the lowest level of expression of the standards 1 2 3 4 5 5

6. WIDA Standards Examples
Social Studies, Listening, 6-8
English language learners process, understand, interpret, and evaluate spoken language in a variety of situations.
Level 1: Identify icons on maps or graphs from oral statements.
Level 3: Categorize resources or products of regions from oral descriptions.
Level 5: Draw conclusions about resources or products in various regions based on oral descriptions.
Social & Instructional, Writing, 6-8
English language learners engage in written communication in a variety of forms for a variety of purposes and audiences.
Level 4: Suggest ideas for making changes in school, such as rearranging a schedule or adding subjects.
Language Arts, Reading, 6-8
English language learners process, interpret, and evaluate written language, symbols, and text with understanding and fluency.
Level 5: Draw conclusions from explicit and implicit text. 6

7. Aligning State Content Standards with WIDA ELP Standards
MIDDLE/JUNIOR HIGH SCHOOL
Illinois State Social Science Standards
STATE GOAL 17: Understand world geography and the effects of geography on society, with an emphasis on the United States.
17.A.3b Explain how to make and use geographic representations to provide and enhance spatial information.
17.B.3a Explain how physical processes including climate, plate tectonics, erosion, soil formation, water cycle, and circulation patterns in the ocean shape patterns in the environment and influence availability and quality of natural resources.
17.C.3b Explain how patterns of resources are used throughout the world.
WIDA Social Studies Standards — Listening
English language learners process, understand, interpret, and evaluate spoken language in a variety of situations.
Level 1: Identify icons on maps or graphs from oral statements.
Level 3: Categorize resources or products of regions from oral descriptions.
Level 5: Draw conclusions about resources or products in various regions based on oral descriptions. 7

8. Presentation Outline The WIDA Assessment Framework and Standards
Structure of ACCESS for ELLs™
The Item Writing Course 8

9. Structure of ACCESS for ELLs
Grade Level and Tier K
1-2
3-5
6-8
9-12
A (adaptive)
A B C
Domains
Listening — group admin, machine scored
Reading — group admin, machine scored
Speaking — individual admin, adaptive, TA scored
Writing — group admin, rater scored
Forms
(roll-out Spring 2005)
(used to produce screener)
(roll-out Spring 2006) 9

10. Theme and Item Structure
Item Model
Item Stimulus
Question and Response Options 10

11. Expression of the Thematic Folder
Orientation
Contextualizes the items
Engages background knowledge
Stimulus — Text and/or Graphic
Presents new information
Organizes information visually
Reduces memory load
3 or more items at same or scaffolded proficiency levels
Item levels determined by specs
Address single strand of PIs 11

12. Sample Listening Theme
Script: Look at the map. It shows some of the important economic activities in different regions of modern Russia. Each symbol in the legend shows a different economic activity. A fish indicates fishing, a factory heavy industry, logs forestry, wheat agriculture, a pick and shovel mining, and an oil rig petroleum.
The Russian Economy 12

13. Sample Listening Items1
Script: Find the symbol for oil.
Script: Russia has many kinds of natural resources. These include coal and iron ores from mining, oil from drilling, and wood from logging. Which region in Russia has the greatest variety of natural resources? 3
Urals
Western Siberia
Eastern Siberia
Far East
Script: The part of Russia from Siberia east is extremely cold for much of the year and is sparsely populated. The western part of the country, particularly in the south, has a milder climate and a longer growing season. Almost 80 percent of the population lives in the western part.
Since factories require a large labor force to run them but they also need raw materials and energy, where would you probably find the greatest concentration of Russia’s heavy industry? 5
In the southern part of the country where food is plentiful
In the northern part of the country where the population is large
In the eastern part of the country where raw materials are abundant
In the middle part of the country where raw materials and people are abundant 13

14. Changes in Cafeteria Menu
Sample Writing Theme
The principal at your school has decided to change the prices of some foods in the school cafeteria. Some students like the changes and some do not.
Changes in Cafeteria Menu
Item
Old Price
New Price
Hamburger
1.25
1.50
Soda
.45
.60
Milk
.50
.25
Apple
.40
Banana
.30
Pie
.90
1.00
Ice Cream
.80
1.10
Yogurt 14

15. Sample Writing Item
Look at the new cafeteria price list. Suggest changes you would make to the prices for these or other foods.
Write a short paragraph of at least 8 sentences. Write about changes you would make to cafeteria prices. 15

16. Goals of an Item Specification
Social Studies, Listening, 6-8, Level 1
State the assessment objective
By grade
By PI
Describe the item stimulus
Use of graphics/text
Mode of presentation to test taker
Describe the test taker’s task
Method of selecting a correct response
Method of constructing a scorable response
Properties of response options
Assessment Objective
Assessment tasks will allow ELLs in the 6th to 8th grade to demonstrate the ability to identify icons on maps or graphs from oral statements by selecting the corresponding picture.
Stimulus Description
Stimulus is a simple oral statement or command (no more than one sentence) that requires the students to identify an icon (e.g., stars, trees, compass rose, ) on a map.
The teacher will read the stimulus aloud from the test administration book. The stimulus will not be printed in the student's test booklet.
Task Description
Description:
Given four pictures on a map, the student selects the picture that matches the oral statement. The map can be of a fictitious country.
Response Options:
There are four options.
Options consist of pictures of icons on a map. There is a single, correct answer. One picture is the key; the others are distractors representing other icons commonly confused with the key. 16

17. Life Cycle of a Test Item
Receive theme & items from item writer
Submit items to external review
Commission illustrations
Check thematic folder for internal consistency
Submit folder and items to bias review
Layout folder in test form
Field test to establish item properties
Operationalize for one year
Retire the folder and items 17

18. What the Item Writer Does
R e v i e w &
R e v I s e
Read the scope of the item in the specs statement
Read the stimulus description & identify an appropriate content area
Write or describe the theme stimulus and/or model
Write the item stimuli or prompts
Read the task description & write response options for the items 18

19. Bias & Content Reviewers Approved Items (ready for field testing)
Item Creation & Review
Specifications
Raw Items
Item Writers
Internal Reviewers
Cleaned Items
Bias & Content Reviewers
Approved Items (ready for field testing) 19

20. Raw Item Selected-Response Item
Language Arts, Reading, Grade 6-8, Level 5
Orientation
In your test booklet are some sentences that tell why something happened in the story I will read.
Theme Stimulus
Leo and his father went to the grocery store. They bought food for a party they are having. At the grocery store, they picked out a cake and some ice cream. When they got home, Leo’s father put the cake on the table in the kitchen and walked away. Then, Leo’s dog, who had not eaten all day, came into the kitchen and ate the whole cake!
Specification (in part)
Performance Indicator
Draw conclusions from explicit and implicit text
Assessment Objective
Assessment tasks will allow ELLs in the 6th to 8th grade to draw a text supported conclusion from a reading passage by selecting the corresponding text option.
Stimulus Description
Stimulus is a fictional story of 4 to 6 sentences, some of which may show complex sentence structures. Vocabulary is familiar.
Stimulus
None
Question
Why did Leo’s dog eat the cake?
Options
A. Because he was hungry
B. Because he was a bad dog
C. Because Leo told him to eat it
D. Because he was thirsty
Key
A. Because he was hungry 20

21. Cleaned Item
Orientation
The Brown family is going to have a party for their friends and neighbors to celebrate the Fourth of July.
Item is reconceived as part of a thematic folder with items at levels 3, 4, and 5 (only level 5 is shown here)
Theme Stimulus
Generic picture of a family making preparations for a party.
Leo and his father went to the grocery store to buy food for the party, but they forgot to feed their dog Rambo before they left. At the grocery store, they picked out a cake, some ice cream, and soda. When they got home, Leo put the food on table in the kitchen and went to help his father. When he got back, he was horrified to see that Rambo had his nose in the cake, happily eating away.
Item Stimulus (Level 5)
Question
Why did Rambo eat the cake?
Options
A. Because he was hungry
B. Because he was happy
C. Because Leo took him to the grocery store
D. Because Leo did not buy him dog food
Key
A. Because he was hungry 21

22. Approved Item Item after review by content and bias committee.
Orientation
The Rodriquez family is going to have a party for their friends and neighbors to celebrate Cinco de Mayo, an important holiday in Mexico.
Item after review by content and bias committee.
Theme Stimulus
Generic picture of a family making preparations for a party. Family members should look Hispanic.
Leo and his father went to the grocery store to buy food for the party, but they forgot to feed their dog Cabo before they left. At the grocery store, they picked out a cake, some ice cream, and soda. When they got home, Leo put the food on table in the kitchen and went to help his father. When he got back, he was horrified to see that Cabo had his nose in the cake, happily eating away.
Item Stimulus (Level 5)
Question
Why did Cabo eat the cake?
Options
A. Because he was hungry
B. Because he was happy
C. Because Leo took him to the grocery store
D. Because Leo did not buy him dog food
Key
A. Because he was hungry 22

23. The Review Process Check for fit to PIs
Does length of stimulus match spec?
Does linguistic complexity of stimulus match spec?
Is vocabulary appropriate to spec?
Check response options properties
Is there only one correct response option?
Are distractors motivated and equally attractive?
Are response options balanced in length & complexity?
Check for inter-item balance & integrity
Are items consistent with theme?
Are items independent?
Check for content accuracy
Check for language, culture, and gender bias 23

24. Presentation Outline The WIDA Assessment Framework and Standards
Structure of ACCESS for ELLs™
The Item Writing Course 24

25. Course Goals
Understand and apply established principles of language test construction, including notions of test validity and reliability
Develop test items which accord with guidelines established for determining item difficulty, bias and authenticity
Develop test items which accurately address a set of predetermined item specifications (the WIDA specifications), including writing test prompts and scoring rubrics
Develop test items across the range of item formats designated by the WIDA specifications
Develop test items that target specific levels of English language proficiency
Develop test items that are aligned with the functional language requirements of social and classroom language and of the content areas: English language arts, mathematics, science and social studies. 25

26. Course Work Requirements
Study and apply the specifications for the participant’s designated grade level
Write test items in each of the four language domains for the designated grade level
Communicate with the class on substantive issues via discussion boards
Communicate with the class on procedural issues via chat room
Interact with other members of designated grade level
Take and pass quizzes and final exam 26

27. Course Participants Lead Instructor: Jim Bauman
Associate Instructors: Jessica Motz & Ellen Parkhurst
Teaching Assistants: George Motz (1-2), Lee Gough (3-5), Jennifer Himmel (6-8), Ellen Daniels (9-12)
Active Participants (28)
From IL:
Marion Flaman—SD54 Michael Safina—CCSD59 Diane Wilkin—SD99 Delia Rodriguez—CUSD300 Irene Pelc—CCSD15 Deb Grant—HSC220
Auditors (5) 27

28. Course Grading Structure
Item submission 60%
Participation in chat sessions and discussions 20%
Quizzes 20%
Successful completion of course requirements at an 80% level earns 2 university credits from the University of Wisconsin 28

29. Course Home Page
Course Menu 29

30. Structure of the Course30

31. Structure of a Module Each module contains: Readings
Self-Paced Exercises
Interactive Exercises
Discussions
Chats
Assignments
Quizzes 31

32. Documents 32

33. Discussion Boards 33

34. Discussion Board Results
Discussion Forum
Responses
Pages
Words
Exploring the item specifications 41 15
3,000
Sources of difficulty in listening items
106 36
10,700
How to evaluate constructed responses
115 43
13,700
Assessing language vs assessing content 85 34
11,725
Sources of reading errors 97 35
9,225
Total
444
163
48,350 34

35. Completing an Assignment
Specification
Download appropriate set of item specs from Documents area
Download data entry form from Documents
Conceive & research folder and items
Complete item data entry form
Submit completed form to Dropbox
Revise per instructor’s feedback
Resubmit to Dropbox
Blank Data Entry Form
Content Standards
Completed Data Entry Form
Dropbox 35

36. The Dropbox 36

37. Test Item Results Number of Items Produced Standard Language Domain
Standard   
Language Domain
GradeLevel LA MA SC SI SS
Grand Total
Listening
1-2 39 32 38 53 21
183
3-5 49 56 51
108 33
297
6-8 60 44 83 42
268
9-12 43 34 29 65 15
186
Listening Total
191
161
162
309
111
934
Reading 28 25 20 11
135 48 45 89
247 71 19
188 26 13 12
Reading Total
141
116
109
249 63
678
332
277
271
558
174
1612 37

38. What Teachers Got as Item Writers
University credit or CEUs
Direct experience with on-line learning
Professional development on the WIDA standards and writing items for large-scale assessments
Informed discussion on how to apply WIDA standards to the curriculum
Professional contacts with other teachers across the consortium
The deep gratitude of the ACCESS for ELLs™ development staff!! 38

39. What WIDA Got from Teacher Item Writers
Approximately 1600 raw items
A nucleus in each state of WIDA informed staff
A pool of talent to serve as test coordinators, administrators, and future item reviewers
Valuable insight on teachers’ perspectives on standardized testing
Important clues on how to build professional development to apply the WIDA framework in the classroom 39

40. Questions, Discussion, Feedback?40