Presentation is loading. Please wait.

Presentation is loading. Please wait.

April 2008Getting Involved in STAR1 STAR Item Writing, Assessment Review Panels, Standard Setting, and Field Testing Webcast April 23, 2008 START TIME.

Similar presentations


Presentation on theme: "April 2008Getting Involved in STAR1 STAR Item Writing, Assessment Review Panels, Standard Setting, and Field Testing Webcast April 23, 2008 START TIME."— Presentation transcript:

1 April 2008Getting Involved in STAR1 STAR Item Writing, Assessment Review Panels, Standard Setting, and Field Testing Webcast April 23, 2008 START TIME 9:00 a.m. Presenters Jonathan Halling - ETS Stephanie Sartain Herrera - ETS Mary Anne Arcilla - ETS Mark McLean - CDE

2 April 2008Getting Involved in STAR2 Agenda 1.Life of a Test Item 2.Item Writer Training 3.Assessment Review Panels (ARPs) 4.Standard Setting 5.Field Testing 6.Open Forum

3 April 2008Getting Involved in STAR3 Life of a Test Item

4 April 2008Getting Involved in STAR4 Study state guiding documents ARP & CDE develop blueprints SBE action Develop item writer materials CDE review of item writer materials Train item writers Receive items Prepare, review, and revise items ARP & CDE review items Revise items SPAR review Create field test sets with CDE review Conduct field tests Score and process data from field test ARP & CDE review data from field test Create operational forms ARP & CDE review forms Administer operational test

5 April 2008Getting Involved in STAR5 Study state guiding documents Develop item writer materials CDE review of item writer materials Train item writers Receive items Prepare, review, and revise items ARP & CDE review items Revise items SPAR review Create field test sets with CDE review Conduct field tests Score and process data from field test Create operational forms ARP & CDE review forms Administer operational test ARP & CDE review data from field test ARP & CDE develop blueprints SBE action

6 April 2008Getting Involved in STAR6 Study state guiding documents ARP & CDE develop blueprints SBE action Develop item writer materials Create field test sets with CDE review Conduct field tests Score and process data from field test Create operational forms ARP & CDE review forms Administer operational test ARP & CDE review data from field test CDE review of item writer materials Train item writers Receive items Prepare, review, and revise items ARP & CDE review items Revise items SPAR review

7 April 2008Getting Involved in STAR7 Study state guiding documents ARP & CDE develop blueprints SBE action Develop item writer materials CDE review of item writer materials Train item writers Receive items Prepare, review, and revise items ARP & CDE review items Revise items SPAR review Create field test sets with CDE review Create operational forms ARP & CDE review forms Administer operational test Conduct field tests Score and process data from field test ARP & CDE review data from field test

8 April 2008Getting Involved in STAR8 Study state guiding documents ARP & CDE develop blueprints SBE action Develop item writer materials CDE review of item writer materials Train item writers Receive items Prepare, review, and revise items ARP & CDE review items Revise items SPAR review Create field test sets with CDE review Conduct field tests Score and process data from field test Create operational forms ARP & CDE review data from field test ARP & CDE review forms Administer operational test

9 April 2008Getting Involved in STAR9 Item Writer Training

10 April 2008Getting Involved in STAR10 Study state guiding documents ARP & CDE develop blueprints SBE action Develop item writer materials CDE review of item writer materials Train item writers Receive items Prepare, review, and revise items ARP & CDE review items Revise items SPAR review Create field test sets with CDE review Conduct field tests Score and process data from field test CDE & ARP review data from field test Create operational forms ARP & CDE review forms Administer operational test

11 April 2008Getting Involved in STAR11 Item Writer Training What will be covered by ETS in the training session? –What makes a good item –The item-writing process –How to write items specifically to meet California standards Who should attend? Persons who: –Possess knowledge of California content standards –Are current or recent teachers –Have at least three (3) years teaching experience

12 April 2008Getting Involved in STAR12 Item Writer Training How many are trained? –20–30 attendees per content area in CAPA, CST, CMA, and STS Where does training take place? –2008: Long Beach, CA When does the 2008 training occur? –July 15–16: CAPA items –July 17–18: CST, CMA, and STS items

13 April 2008Getting Involved in STAR13 Item Writer Training Benefits –Professional development –Increased familiarity with California content standards –Potential for becoming professional item writer Expenses –$150 per day or substitute teacher reimbursement –Travel and lodging expenses paid by ETS

14 April 2008Getting Involved in STAR14 Assessment Review Panels (ARPs)

15 April 2008Getting Involved in STAR15 Assessment Review Panels Panel members meet in an advisory capacity to ETS and CDE to review: Item specifications and blueprints Proposed new test items Data collected on field-tested items Proposed new operational test forms Proposed released test questions (RTQs) –CST only Differential Item Functioning (DIF)

16 April 2008Getting Involved in STAR16 Study state guiding documents ARP & CDE develop blueprints SBE action Develop item writer materials CDE review of item writer materials Train item writers Receive items Prepare, review, and revise items ARP & CDE review items Revise items SPAR review Create field test sets with CDE review Conduct field tests Score and process data from field test CDE & ARP review data from field test Create operational forms ARP & CDE review forms Administer operational test

17 April 2008Getting Involved in STAR17 Assessment Review Panels Item Specifications and Blueprints Help define recommended blueprints for new tests –Develop foundation for future item and test form development –Examples: Revised General Mathematics –Conducted April 2, 2008 California Modified Assessment (Grades 9–11)

18 April 2008Getting Involved in STAR18 Study state guiding documents ARP & CDE develop blueprints SBE action Develop item writer materials CDE review of item writer materials Train item writers Receive items Prepare, review, and revise items ARP & CDE review items Revise items SPAR review Create field test sets with CDE review Conduct field tests Score and process data from field test CDE & ARP review data from field test Create operational forms ARP & CDE review forms Administer operational test

19 April 2008Getting Involved in STAR19 Assessment Review Panels Item Review Advise if the items developed are: –Measuring the standard –Grade-level appropriate –Free from bias Ethnic Socioeconomic Gender Geographical –Interesting and appropriate to students Suggest ways in which an item may be adjusted to meet the above criteria, if necessary Do NOT rewrite items

20 April 2008Getting Involved in STAR20 Study state guiding documents ARP & CDE develop blueprints SBE action Develop item writer materials CDE review of item writer materials Train item writers Receive items Prepare, review, and revise items ARP & CDE review items Revise items SPAR review Create field test sets with CDE review Conduct field tests Score and process data from field test CDE & ARP review data from field test Create operational forms ARP & CDE review forms Administer operational test

21 April 2008Getting Involved in STAR21 Assessment Review Panels Data Review Recommend whether field-tested items should be included in the “bank” and hence be available to be selected for inclusion on future operational tests –Review statistics on the item’s performance as a field-test item –No editing of items is performed at this stage

22 April 2008Getting Involved in STAR22 Assessment Review Panels Differential Item Functioning (DIF) Review Occurs when item performance is significantly different for members of reference groups and comparison groups Participants representing various demographic groups review test questions to determine whether or not items are appropriate One-day meeting, conducted after field testing Note: DIF review panel consists of different participants from other ARPs

23 April 2008Getting Involved in STAR23 Study state guiding documents ARP & CDE develop blueprints SBE action Develop item writer materials CDE review of item writer materials Train item writers Receive items Prepare, review, and revise items ARP & CDE review items Revise items SPAR review Create field test sets with CDE review Conduct field tests Score and process data from field test CDE & ARP review data from field test Create operational forms ARP & CDE review forms Administer operational test

24 April 2008Getting Involved in STAR24 Assessment Review Panels Forms Review Last stage in the process of building the test Assembled operational forms: –Contain a range of difficulty of items –Match the blueprint and statistical specifications –Avoid clueing No editing of items

25 April 2008Getting Involved in STAR25 Assessment Review Panels Released Test Question (RTQ) Review After use on operational test forms, some questions are released for review and use by the public –Contain a range of item difficulty –Are proportional to the blueprint –Released annually No editing of items

26 April 2008Getting Involved in STAR26 ARP Meeting Schedules

27 April 2008Getting Involved in STAR27 2008 ARP Meeting Schedule CST Item Review –April 2–5 Data Review –July 29–30 Forms Review –September 23 Released Test Questions –October 29 Note: ARP members are expected to attend meetings at ALL stages of development

28 April 2008Getting Involved in STAR28 2008 ARP Meeting Schedule CMA Item Review –Grades 3–8 September 9–12 Data and Forms Review –Grades 6–8 November 19–20 Data Review –Grades 3–5 To be scheduled early 2009 Note: ARP members are expected to attend meetings at ALL stages of development

29 April 2008Getting Involved in STAR29 2008 ARP Meeting Schedule CAPA Item Review –May 13–14 Data Review –September 25 Forms Review –To be scheduled early 2009 Note: ARP members are expected to attend meetings at ALL stages of development

30 April 2008Getting Involved in STAR30 2008 ARP Meeting Schedule STS Item Review –RLA grades 8–11, Algebra I, and Geometry April 8–11 –RLA and Math grades 2–7 July 8–11 Data Review –RLA and Math for grades 2–7 October 8–9 Forms Review –Grades 2–4 October 10 Note: ARP members are expected to attend meetings at ALL stages of development

31 April 2008Getting Involved in STAR31 Becoming an ARP Member

32 April 2008Getting Involved in STAR32 The Process 1.Review the ARP member requirements 2.Obtain application form: E-mail star@ets.org Call Kristin Elizalde 925-808-2009 3.Submit a completed application form 4.Application reviewed by: ETS content experts CDE Standards and Assessment staff State Board of Education staff 5.Application approved or denied Note: Process can take several weeks

33 April 2008Getting Involved in STAR33 ARP Members Needed! California Standards Tests –Science (Grades 5 and 8) –History–Social Science (Elementary and Middle School) –ELA (All grades) California Modified Assessment –English–Language Arts (ELA) –Mathematics –Science Note: CMA – Middle and High School especially needed!

34 April 2008Getting Involved in STAR34 ARP Members Needed! California Alternate Performance Assessment –English–Language Arts (ELA) –Mathematics –Science Standards-based Tests in Spanish –Reading/Language Arts (RLA) –Mathematics, Algebra I, and Geometry Note: STS – Middle and High School especially needed!

35 April 2008Getting Involved in STAR35 Benefits of Being an ARP Member Become more familiar with the testing program Professional development –Learn more about what attributes a high quality test item should display –Learn more about assessing how an item performs on a test Contribute to the development of high- quality tests All meeting attendance expenses covered

36 April 2008Getting Involved in STAR36 ARP Member Requirements

37 April 2008Getting Involved in STAR37 ARP Member Requirements All Panels Educators from –Urban areas –Rural areas –Suburban areas Experienced in teaching K–12 students from diverse populations

38 April 2008Getting Involved in STAR38 ARP Member Requirements CST, CMA, CAPA, and STS Bachelor’s or higher degree in content area or related subject Knowledge of California content standards Familiarity with student capabilities K–12 teachers must also have: –Three or more years teaching courses in grades K–12 ELA, math, science, or history–social science County, district, or university content specialists must also have: –Three or more years as a district or county content specialist or university instructor ELA, math, science, or history–social science

39 April 2008Getting Involved in STAR39 Additional ARP Member Requirements for CMA English–language arts, mathematics, and science only K–12 teachers –Three or more years of experience teaching ELA, math, or science in grades K–12 and experience working with special education students –Experience with more than one type of disability School administrators, district/county content/program specialists, or university educators –Three or more years of experience as a school administrator, district/county content/program specialist, or university instructor in the area of special education or areas related to English–language arts, mathematics, or science

40 April 2008Getting Involved in STAR40 Additional ARP Member Requirements for CAPA English–language arts, mathematics, and science only K–12 teachers –Experience with students with severe cognitive disabilities –Experience with more than one type of disability –Three or more years of K–12 experience teaching special education or teaching ELA, math, or science in a general education classroom School administrators, district/county content/program specialists, or university educators –Three or more years of experience as a school administrator, district/county content/program specialist, or university instructor in the area of special education or areas related to English–language arts, mathematics, or science

41 April 2008Getting Involved in STAR41 Additional ARP Member Requirements for STS Reading/language arts (RLA) and mathematics only Bilingual and biliterate in Spanish Three or more years experience teaching students in Spanish Bachelor’s or higher degree in Spanish or a subject area related to bilingual education, reading/language arts, or mathematics

42 April 2008Getting Involved in STAR42 Break 10 Minutes

43 April 2008Getting Involved in STAR43 Standard Setting

44 April 2008Getting Involved in STAR44 What is Standard Setting? Process used to recommend the minimum test score needed for a student to be considered performing at a certain performance level –For example, the minimum test score needed to be at the proficient level Bookmark standard setting methodology –Participants review test items that have been sequenced from the easiest to most difficult item –Participants identify the items that likely exceed the ability of a student at each of the California performance levels –The difficulty level of these identified items form the basis for the minimum test scores needed to reach each performance level

45 April 2008Getting Involved in STAR45 Standard Setting CAPA Levels I–V (ELA, Math, and Science) Three-day meeting –September 16–18, 2008 Participant requirements same as ARP members

46 April 2008Getting Involved in STAR46 Standard Setting CMA Grades 3–5 (ELA, Math, and Science) Two three-day meetings –ELA September 24–26, 2008 –Math and Science October 1–3, 2008 Participant requirements same as ARP members

47 April 2008Getting Involved in STAR47 Standard Setting STS Grades 2–4 RLA and Math Three-day meeting –RLA January/February 2009 –Math January/February 2009 Participant requirements same as ARP members

48 April 2008Getting Involved in STAR48 Standard Setting Member Requirements Educators from: –Urban, rural, and suburban areas –Large and small districts –Geographically dispersed throughout California Experienced in teaching students in appropriate grades/content areas and from diverse populations Community members –Parent Teacher Association (PTA) –School board –Business

49 April 2008Getting Involved in STAR49 Standard Setting For CAPA, CMA and STS Application Form –E-mail star@ets.org or –Call Kristin Elizalde 925-808-2009 Reimbursement for: –Travel and accommodation expenses –Substitute teacher for school/district Apply now!

50 April 2008Getting Involved in STAR50 Field Testing

51 April 2008Getting Involved in STAR51 Field Test California Modified Assessment (CMA) September 15–29, 2008 Eligible students –IEP –Not taking CAPA –Administered to grades 5, 7, 8, and 9 ELA (grades 7, 8, and 9) ELA writing (grades 5 and 8) Mathematics (grades 7 and 8) Science (grade 9) –Student may test in one or more content areas –2008 CST results Far below basic Below basic

52 April 2008Getting Involved in STAR52 Questions E-mail: star@ets.org Subject: 2008 CMA Field Test Telephone: STAR TAC 800-955-2954 CMA Field Test

53 April 2008Getting Involved in STAR53 Questions E-mail: star@ets.org Subject: April Webcast Open Forum

54 April 2008Getting Involved in STAR54 Next Webcast STAR 2008: Making Sense of the STAR Data CD May 21, 2008 Webcast starts at 9:00 a.m.


Download ppt "April 2008Getting Involved in STAR1 STAR Item Writing, Assessment Review Panels, Standard Setting, and Field Testing Webcast April 23, 2008 START TIME."

Similar presentations


Ads by Google