1. Development of Study Guides and Exams – A Colorado Perspective Betsy Buffington Colorado State University

2. Enter Test Items In ParTEST Enter Test Items In ParTEST Validate Existing Questions Validate Existing Questions Develop Study Guide Develop Study Guide Develop Guide Outline/Blueprint Develop Guide Outline/Blueprint Finalize Test Items Finalize Test Items Conduct Job Analysis Conduct Job Analysis Recruit Item Writing Committee Recruit Item Writing Committee Recruit Study Guide Workgroup Recruit Study Guide Workgroup Validate Draft Test Items Validate Draft Test Items Orientation Workshop Orientation Workshop Develop Draft Test Items Develop Draft Test Items Select Guide Study Guide Development Exam Development

3. Select Guide Pass/fail rates of category Number of applicators testing/year Rotate between Agricultural, Ornamental, and Structural Select Guide

4. Recruit Study Guide Workgroup CDA requests professional organizations to submit volunteer(s) Individuals may also volunteer themselves All levels of experience –Certified Operator Mixes, loads, or applies any pesticide, including RUPs, under the supervision of a QS –Qualified Supervisor Without supervision, evaluates pest problems, recommends pest controls using pesticides or devices, mixes, or loads, or applies any pesticide, sells application services or operates devices Supervises in any of these functions Trains Technicians Recruit Study Guide Workgroup Recruit Study Guide Workgroup

5. Develop Guide Outline/Blueprint One day question and answer session Discuss what information to include –What is important for entry-level pesticide application? Equipment types Pesticides used –Refer general information to Core Develop outline and order of chapters –Layout mirrors exam categories Job analysis Develop Guide Outline/Blueprint Develop Guide Outline/Blueprint

6. Guide Blueprint

7. Develop Study Guide - Writing Glean information from other study guides –Tailor to Colorado Workgroup participants provide additional resources –Emails –Other references Develop Study Guide Develop Study Guide

8. Writing Study Guides

9. Develop Study Guide – Review Send copies to CSU staff, CDA staff, and Industry –Workgroup participants plus other key identified individuals Highlight particular chapters to review –Ensures all chapters will be reviewed in-depth –Reduces amount of time spent reviewing guide –CEPEP and CDA staff review entire guide Incorporate comments when guides are received Second review if time allows

10. Develop Study Guide – Publish

11. Enter Test Items In ParTEST Enter Test Items In ParTEST Validate Existing Questions Validate Existing Questions Develop Study Guide Develop Study Guide Develop Guide Outline/Blueprint Develop Guide Outline/Blueprint Finalize Test Items Finalize Test Items Conduct Job Analysis Conduct Job Analysis Recruit Item Writing Committee Recruit Item Writing Committee Recruit Study Guide Workgroup Recruit Study Guide Workgroup Validate Draft Test Items Validate Draft Test Items Orientation Workshop Orientation Workshop Develop Draft Test Items Develop Draft Test Items Select Guide Study Guide Development Exam Development

12. Recruit Item Writing Advisory Committee CDA requests each professional organization to submit a list of volunteers Individuals can also volunteer Committee participants: –Must be licensed as CO or QS in category Various levels of experience Security Agreement –All test items become sole property of CDA Recruit Item Writing Committee Recruit Item Writing Committee

13. Security Agreement

14. Orientation Workshop Teach participants basic skills necessary to write high quality exam items –Discuss pro’s and con’s of each question type: True/False Multiple Choice –Discuss do’s and don’ts of writing good items Orientation Workshop Orientation Workshop

15. Conduct Job Analysis Determine topics to be included –i.e,. Applicator safety, laws and regulations, calibration, pest ID, etc. Determine the percentage of questions that should come from each topic –CO and QS exams may be weighted differently Results of job analysis serve as blueprint for exam Conduct Job Analysis Conduct Job Analysis

16. Example Job Analysis

17. Validate Existing Questions Committee edits each item for –Content –Format –Style –Grammar –Absence of potentially inflammatory or biased language CO questions must be covered in study guide QS questions may be in study guide or from other sources Item(s) may be –Accepted –Reworked and accepted –Rejected –Referred to another item workgroup Validate Existing Questions Validate Existing Questions

18. Validating Existing Questions

19. Develop Draft Test Items Participants asked to submit questions –Usually 5 or more questions/participant –May assign questions to fulfill test blueprint Develop Draft Test Items Develop Draft Test Items

20. Item Submission Form

21. Validate Draft Test Items Items are reviewed and critiqued by the full Item Writing Workgroup Item(s) may be –Accepted –Reworked and accepted –Rejected If accepted, the item becomes property of CDA Validate Draft Test Items Validate Draft Test Items

22. Finalize Test Items Make sure minimum requirements are met Make sure ~75% of QS questions are answered in study guide Check to see if true/false ratio is ~50% Ensure we have produced a minimum of 150% of total items needed Finalize Test Items Finalize Test Items

23. Enter Finalized Test Items into ParTEST Enter items into computerized item banking system Enter Test Items In ParTEST Enter Test Items In ParTEST

24. What We Have Learned Mix of experience levels are best Individuals contribute differently Industry buy-in results in –Industry ownership of system –Support for future development of guides/exams –Cheap labor Relationship between CEPEP and CDA key to success