Presentation is loading. Please wait.

Presentation is loading. Please wait.

2007 Annual Conference Basic Testing Issues Ron Rodgers CTS/Employment Research Institute.

Similar presentations


Presentation on theme: "2007 Annual Conference Basic Testing Issues Ron Rodgers CTS/Employment Research Institute."— Presentation transcript:

1 2007 Annual Conference Basic Testing Issues Ron Rodgers CTS/Employment Research Institute

2 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Session Agenda A.Clarify test development Standards B.Summarize test development process C.Write & validate multiple choice items D.Respond to your questions

3 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation APA Standards* ( ) APA Standards used by US Supreme Court to validate fair, valid examinations Test must reflect job qualifications Credentialing identifies qualified practitioners for public, employers Panels of experts should help specify certification performance standards *Standards for Educational and Psychological Testing, American Psychological Association, American Educational Research Association, National Council on Measurement in Education

4 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation APA Standards* ( ) Candidates who fail receive report w/information about weak areas Credentialing must depend on the knowledge and skills needed for acceptable professional performance Criteria should not be adjusted to regulate the number or proportion of persons who pass the test

5 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Define Exam Objectives Create SME panel to represent experienced professionals in field Identify references & initial specs Conduct job analysis Link test specifications to job tasks

6 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Write & Validate Questions Select item writing model Draft item bank to meet test specs Document each Q to references Validate/edit Qs with SME panel

7 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Score & Report Results Assemble test forms Set passing criteria Review item performance statistics Equate test forms Adjust scoring if needed

8 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Provide Candidate Info & Tech Support Prepare candidate materials Monitor test & item performance Revalidate non-performing Qs Amend items/test forms as needed

9 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Maintain Test Security Limit access to secure materials Use non-disclosure agreements Emphasize security to candidates Define consequences of cheating

10 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Non-Disclosure Agreement Each Subject Matter Expert (SME) acknowledges that all work performed under this Agreement for and on behalf of (program sponsor) is confidential and that no disclosure of any aspects of test data, information, materials and questions is permitted. The SME also agrees to adhere to all test security procedures and requirements specified by __________.

11 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Responding to Your Questions Writing good test questions Bruce Anderson, PES Identifying your concerns Bruce Anderson, PES Ron Rodgers, CTS Finding sources of assistance

12 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Speaker Contact Information Ron Rodgers Continental Testing Service Employment Research Institute 809 Ridge Road, Suite 201 Wilmette, IL Phone:

13 2007 Annual Conference “Writing Good Multiple-Choice Items” Bruce Anderson Professional Examination Service

14 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Agenda Overview of the examination development process Review of item-writing principles Filling out the item-writing form Review and development of items

15 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Developing the Certification Examination Conduct a job analysis/role delineation. Generate test specifications. Write items. Validate/Improve items. Enter items into item bank. Assemble draft examinations. Review and revise examination forms.

16 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Developing the Certification Examination (contd.) Conduct a passing point study. Review item statistics. Equate examination forms. Score and report examination results.

17 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation What are the chances of an uninformed candidate selecting the correct answer to your item? 1 in 4, and that’s if you write a good item!!

18 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Guessing the Correct Answer For a multiple-choice item with four possible responses: 3 good distracters = a 25% chance 1 BAD distracter = a 33% chance 2 BAD distracters = a 50% chance 3 BAD distracters = a 100% chance of an uninformed candidate guessing the correct answer.

19 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation The Effects of Guessing on the Value of the Certification The greater the ability of the uninformed candidate to guess correctly, the less valid and reliable the examination is. The less valid and reliable the examination is, the less value is placed on the certification.

20 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation The Critical Objective when Writing Good Multiple-Choice Items Reducing the ability of uninformed candidates to guess correctly by providing NO CLUES to the correct answer.

21 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Where are the clues which help candidates to guess correctly? The type of format chosen for the question The format of the question asked (STEM) The way the correct answer (KEY) is written The way the incorrect answers (DISTRACTORS) are written

22 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation The Format of the Stem DIRECT QUESTION: Which of the following scientists was responsible for the development of the first polio vaccine? or INCOMPLETE STATEMENT: The first polio vaccine was developed by: 1.Jonas Salk 2.Louis Pasteur 3.Edward Jenner 4.Robert Koch

23 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Correct vs. Best Answer Stems What is the capital of Kentucky? 1. Frankfort* 2. Lexington 3. Louisville 4. Ashland The major goal of raising the legal drinking age is to: 1.Reduce driving fatalities and injuries.* 2.Reduce liquor consumption. 3.Discourage the habit of drinking by adolescents. 4.Reduce roadside litter.

24 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Guidelines for Constructing the Stem Clarity and Directness: Is the requirement of the item clear without reference to the choices? Sufficiency: Is enough information provided to answer the question? Focus: Is the item focused on a single problem which assesses a meaningful fact or concept?

25 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation EXAMPLE: Clarity? Sufficiency? Focus? The 2000 U.S. Presidential Election was decided by: 1. fewer than 1000 votes 2. the Electoral College* 3. the Supreme Court 4. December 21, 2000

26 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Guidelines for Constructing the Stem (cont’d) Avoid negative stems. Provide just enough information. Do not use a word in the stem which only appears in the choice which is the key. Do not provide gratuitous instruction. Provide most information in the stem so that the options do not have repetition of words.

27 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation EXAMPLE: Negatives are confusing. Which of the following authors is not a major American novelist? 1. George Orwell 2. William Faulkner 3. John Steinbeck 4. Ernest Hemmingway

28 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation EXAMPLE: Improved Which of the following authors is a major British novelist? 1. George Orwell 2. William Faulkner 3. John Steinbeck 4. Ernest Hemmingway

29 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation EXAMPLE: Gratuitous instruction Poor In 1850, Adolphe Chatin, the professor of pharmacy in Paris, believed that goiter resulted from an inadequate amount of iodine in the diet. The thyroid is part of which body system? 1.Endocrine* 2.Nervous 3.Musculoskeletal 4.Cardiovascular Better The thyroid is part of which body system? 1. Endocrine* 2. Nervous 3. Musculoskeletal 4. Cardiovascular

30 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Guidelines for the Correct Answer (KEY) Is the intended key unambiguously correct and recognized as such by the experts? Is the key parallel to the other choices (e.g., not shorter or longer, or more technical, or more detailed)? Is there ONLY ONE clearly correct or best answer?

31 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Guidelines for Distracters SOURCES OF DISTRACTORS -Common misconceptions -Common job performance errors -Statements which are true but which do not satisfy the requirements of the question -Statements which are either too broad or too narrow for the requirements of the question -Carefully worded incorrect statements which may sound plausible to the uninformed

32 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Guidelines for The Choices Does each choice follow logically and grammatically from the stem? Are the choices parallel in: -length? -grammatical structure? -terminology? -content? -level of detail?

33 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation EXAMPLES: Parallelism An American city noted for its production of automobiles is: 1. Detroit* 2. Ohio 3. Indiana 4. Philadelphia An American city noted for its production of automobiles is: 1.Detroit* 2.Cincinnati 3.Indianapolis 4.Philadelphia

34 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation EXAMPLES: Parallelism, length. Serious communicable diseases that can cause developmental disability to the offspring of a pregnant mother are: 1. rubella and chicken pox* 2. cancer 3. rheumatism and arthritis 4. sinus infection A serious communicable disease that can cause developmental disability to the offspring of a pregnant mother is: 1.rubella* 2.cancer 3.rheumatism 4.sinus infection

35 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Guidelines for Choices (contd.) Do not repeat the same word in each answer choice; include the word once in the stem. As much time as possible is to be spent figuring out the answer rather than reading. Make the choices mutually exclusive. Do not use “all of the above” or “none of the above” as choices. The latter, for example, does not test whether a candidate knows the correct answer.

36 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation EXAMPLES: Overlap The right to vote in the United States is granted to individuals of what age? * The right to vote in the United States is granted to individuals beginning at what age? * 4. 19

37 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation General Considerations Is it RELEVANT to professional practice? Is current BEST practice in the field tested? Does it test an important concept of piece of knowledge? Is it appropriate for entry-level? Is the answer clear to a prepared candidate (not a ‘trick question’)?

38 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation General Considerations Does it avoid controversial issues, e.g., religious, political, philosophical? Are the context, setting, and content equally appropriate and familiar to all segments of the candidate population? Is the terminology equally appropriate and familiar to all segments of the candidate population (e.g., international candidates)? Is it free of bias re: persons or groups?

39 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation EXAMPLE: Sensitive Issue Of the following profile analysis groups, the one most likely to have alcohol-related problems is: 1. residents of rural areas in the South 2. widowed males over 60 years of age 3. women who are married and are wine drinkers 4. men with no religious affiliation*

40 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Other Sensitivity Issues Avoid humor in test questions. Avoid any subject that appears to be advancing a political or business agenda. Avoid topics that could cause undue stress to the candidates, e.g., death, disease, war, brutality. Avoid any stereotypes.

41 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Rules of English Don’t use no double negatives. Make each pronoun agree with their antecedent. Join clauses good, like a conjunction should. Watch them participles when dangling. About them sentence fragments. Verbs has to agree with their subject. Just between you and I, case is important too. Don’t write run-on sentences they are hard to read. Don’t use commas, that are not necessary. Do’nt misuse apostrophes; its’ wrong.

42 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation THE I GNU TEST

43 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation 1. The purpose of the cluss in furmpaling is to remove: 1. cluss-prags 2. tremalis 3. cloughs 4. plumots

44 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation 2. Trassig is true when: 1. lusp trasses the vom 2. the viskal flans, if the viskal is donwill or zortil 3. the belgo frulls 4. dissles lisk easily

45 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation 3. The sigla frequently overfesks the treslum because: 1. all siglas are mellious 2. siglas are always votial 3. the treslum is usually tarious 4. no trelsa are feskable

46 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation 4. The fribbled breg will minter best with an: 1. derst 2. morst 3. sortar 4. ignu

47 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation 5. The mintering function of the ignu is most effectively carried out in connection with: a razma tol the groshing stantol the fribbled breg a frally sush

48 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Problem Items

49 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Problem Items Which of the following persons was President of the United States when the Civil Rights Act of 1964 was enacted? A. Lester Pearson B. Anthony Eden C. Jefferson Davis D. Lyndon Johnson

50 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Problem Items Who orchestrated the December 7, 1941 attack on Pearl Harbor? A. Isoroku Yamamoto B. Chester Nimitz C. Bernard Montgomery D. Seiji Ozawa

51 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Problem Items Which Swedish opera singer was known for her role in Tristan und Isolde? A. Liv Ullman B. Birgit Nilsson C. Jussi Bjorling D. Ingmar Bergman

52 Atlanta, Georgia 2007 Annual Conference Council on Licensure, Enforcement and Regulation Contact Information Bruce R. Anderson Senior Program Director Professional Examination Service 475 Riverside Drive New York, NY Phone: Fax:


Download ppt "2007 Annual Conference Basic Testing Issues Ron Rodgers CTS/Employment Research Institute."

Similar presentations


Ads by Google