In April 2011, The CPE PhD, LLC became Course Outcomes, LLC. The training materials are referred to in this packet are owned and copyrighted by Course.

In April 2011, The CPE PhD, LLC became Course Outcomes, LLC. The training materials are referred to in this packet are owned and copyrighted by Course Outcomes, LLC and can only be used for free trainings. They may not be rebranded nor sold without express written permission from Course Outcomes. © 2011 Course Outcomes, www.courseoutcomes.com

Thank you for using The CPE PhD’s training services. The content of this training is copyrighted by The CPE PhD and may not be resold, repackaged, or printed for profit. Organization Members may download and repackage materials for their organization but not for profit without express written permission from The CPE PhD. © 2011 Course Outcomes, www.courseoutcomes.com

Classify the goal of each item (i.e., know what you are seeking to measure with each item). Tether each item to a specific learner outcome (SLO) which defines the goal of each item. As you recall from the last training, It helps to identify the verb that describes what a person needs to know, feel, or do to correctly answer the item? (see next slide for a list of verbs). Example: SLO = IDENTIFY the purpose of each article of the U.S. Constituion. This item will measure if a person can IDENTIFY that section II protects against a president receiving compensation from individual states. What was a portion of article II meant to prevent? a. federal seizure of individual property b. an abuse of judicial power c. succession from the union d. presidential conflicts of interest © 2011 Course Outcomes, www.courseoutcomes.com

Knowledge: count, define, describe, draw, enumerate, find, identify, label, list, match. name, quote, read, recall, recite, record, reproduce, select, sequence, state, tell, view. Comprehend: classify, cite, conclude, convert, describe, discuss, estimate, explain, generalize, give examples, illustrate, interpret, locate, make sense of, paraphrase, predict, report, restate, review, summarize, trace, understand. Apply: act, administer, articulate, assess, change, chart, choose, collect, compute, construct, contribute, control, demonstrate, determine, develop, discover, dramatize, draw, establish, extend, imitate, implement, interview, include, inform, instruct, paint, participate, predict, prepare, produce, provide, relate, report, select, show, solve, transfer, use, utilize. Analyze: breakdown, characterize, classify, compare, contrast, correlate, debate, deduce, diagram, differentiate, discriminate, distinguish, examine, focus, illustrate, infer, limit, outline, point out, prioritize, recognize, research, relate, separate, subdivide. Synthesize: adapt, anticipate, categorize, collaborate, combine, communicate, compare, compile, compose, construct, contrast, create, design, develop, devise, express, facilitate, formulate, generate, incorporate, individualize, initiate, integrate, intervene, invent, make up, model, modify, negotiate, organize, perform, plan, pretend, produce, progress, propose, rearrange, reconstruct, reinforce, reorganize, revise, rewrite, structure, substitute, validate. Evaluate: appraise, argue, assess, choose, compare and contrast, conclude, criticize, critique, decide, defend, evaluate, interpret, judge, justify, predict, prioritize, prove, rank, rate, reframe, select, support. © 2011 Course Outcomes, www.courseoutcomes.com

This is referred to as the “stem” of an item. The stem should always present a clear question, problem, or issue to be solved. A person who has the skills or ability that the item is seeking to measure should be able to read the stem and answer the question without looking at the options. A stem should allow you to create independent options rather than several dependent options. 1. Why was the SOX legislation passed? To protect against accounting corruption in… a) privately run businesses b) public companies c) government entities d) non-profit corporations © 2011 Course Outcomes, www.courseoutcomes.com

1. Could a knowledgeable person answer the item if they only read the stem? 2. Does the stem pose one, and only one, clear question? 3. Does the stem allow you to write options that have one independent thought in each option? 4. Can the stem be simplified? 5. Are you asking a question that would better be measured in an essay or other format? © 2011 Course Outcomes, www.courseoutcomes.com

#2. The stem is asking about two groups rather than one. By asking how a business AND a consumer are benefited, the item is setting itself up for two possible answers and very lengthy options. The only exception to violating this guideline would be if the answer was the same for both businesses and consumers. Guidelines for writing item stems 1.Could a knowledgeable person answer the item if they only read the stem? 2.Does the stem pose one, and only one, clear question? 3.Does the stem allow you to write options that have one independent thought in each option? 4.Can the stem be simplified? 5.Are you asking a question that would better be measured in an essay or other format? What is wrong with this item stem? How do the voter registration laws help businesses and consumers? © 2011 Course Outcomes, www.courseoutcomes.com

#3. With two possible answers, you may have to write options that are dependent on both parts of the option being correct. For example, if I asked the question above, my options might be: a)Business are helped by knowing the probable tax laws while consumers know if taxes are likely to be passed on to them. (Is the first half of the option correct or the second half? The items are dependent on both halves of the option being correct) Note: Sometimes a correct answer will require a list of correct responses. When you need to have two or more correct options, write the correct options and add “(select all that are correct)” at the end of the stem. Each correct option contains one part of the correct answer and each correct option is independent of selecting the other options (i.e. I don’t have to select option “A” to know I need to select option “C”). However, remember that when you are asking questions that require a list of correct answers, sometimes it is safe to assume that if they know part of the list of correction options, they know the rest of the list. Guidelines for writing item stems 1.Could a knowledgeable person answer the item if they only read the stem? 2.Does the stem pose one, and only one, clear question? 3.Does the stem allow you to write options that have one independent thought in each option? 4.Can the stem be simplified? 5.Are you asking a question that would better be measured in an essay or other format? What is wrong with this item stem? How do the voter registration laws help businesses and consumers? © 2011 Course Outcomes, www.courseoutcomes.com

#1. I can’t answer this item without looking at the options. #2. Conditional items are good items. However, in a multiple choice format, you should not require a person to select more than one part of the condition. A better conditional item would use one blank. #3. The stem is setting up some complex options which will require two dependent answers. What is wrong with this item stem? Executive power will NOT _____ unless _____. Guidelines for writing item stems 1.Could a knowledgeable person answer the item if they only read the stem? 2.Does the stem pose one, and only one, clear question? 3.Does the stem allow you to write options that have one independent thought in each option? 4.Can the stem be simplified? 5.Are you asking a question that would better be measured in an essay or other format? © 2011 Course Outcomes, www.courseoutcomes.com

#2. This item is asking to compare several aspects rather than one. #5. This item would be better answered in an essay or short answer format. However, you could test a person’s understanding of this same information in several multiple choice items. It all depends on what you want to know about your students’ abilities. What is wrong with this item stem? Contrast how US constitutional privacy laws compare with Great Britian’s constitutional privacy laws in terms of the costs and benefits of individual, state, and federal rights. Guidelines for writing item stems 1.Could a knowledgeable person answer the item if they only read the stem? 2.Does the stem pose one, and only one, clear question? 3.Does the stem allow you to write options that have one independent thought in each option? 4.Can the stem be simplified? 5.Are you asking a question that would better be measured in an essay or other format? © 2011 Course Outcomes, www.courseoutcomes.com

#4. This stem could be simplified. “Why do some accountants see recent accounting regulations as a threat to international competition?” What is wrong with this item stem? Why do some accounting lawyers claim or feel that recent accounting legislation is a possible detrimental threat to US based businesses who must, or chose to, compete internationally in business? Guidelines for writing item stems 1.Could a knowledgeable person answer the item if they only read the stem? 2.Does the stem pose one, and only one, clear question? 3.Does the stem allow you to write options that have one independent thought in each option? 4.Can the stem be simplified? 5.Are you asking a question that would better be measured in an essay or other format? © 2011 Course Outcomes, www.courseoutcomes.com

#1 and #2. I could complete this sentence several ways. I have to look at the options to answer this item. What is wrong with this item stem? Article Six of the constitution was… Guidelines for writing item stems 1.Could a knowledgeable person answer the item if they only read the stem? 2.Does the stem pose one, and only one, clear question? 3.Does the stem allow you to write options that have one independent thought in each option? 4.Can the stem be simplified? 5.Are you asking a question that would better be measured in an essay or other format? © 2011 Course Outcomes, www.courseoutcomes.com

#5. This would be better asked as an essay question where a person presents both sides of an argument, takes a stand, and then logically defends that stand. What is wrong with this item stem? Did greenhouse emission legislation have positive or negative effects on the US economy? Guidelines for writing item stems 1.Could a knowledgeable person answer the item if they only read the stem? 2.Does the stem pose one, and only one, clear question? 3.Does the stem allow you to write options that have one independent thought in each option? 4.Can the stem be simplified? 5.Are you asking a question that would better be measured in an essay or other format? © 2011 Course Outcomes, www.courseoutcomes.com

These are referred to as the “options”. The incorrect options are called “distracters”. Notice that all the options belong to a similar category, are at the same level of specificity, have similar grammar forms, similar length, and similar complexity. The distracters should represent common misconceptions, incomplete knowledge, or careless reasoning. They should be enticing to the less informed. 1. An Optometrist would recommend wearing adequate eye protection to prevent … a) Photokeratisis b) Retinoschisis c) Glaucoma d) Esotropia © 2011 Course Outcomes, www.courseoutcomes.com

These are referred to as the “options”. The incorrect options are called “distracters”. Notice that all the options belong to a similar category, are at the same level of specificity, have similar grammar forms, similar length, and similar complexity. The distracters should represent common misconceptions, incomplete knowledge, or careless reasoning. They should be enticing to the less informed. 1. An Optometrist would recommend wearing adequate eye protection to prevent … a) Photokeratisis b) Retinoschisis c) Conjunctivitis d) Dacryoadenitis © 2011 Course Outcomes, www.courseoutcomes.com

These are referred to as the “options”. The incorrect options are called “distracters”. Notice that all the options belong to a similar category, are at the same level of specificity, have similar grammar forms, similar length, and similar complexity. The distracters should represent common misconceptions, incomplete knowledge, or careless reasoning. They should only be enticing to the less informed. 1. An Optometrist would recommend wearing adequate eye protection to prevent … a) Photokeratisis b) Retinoschisis c) Conjunctivitis d) Dacryoadenitis © 2011 Course Outcomes, www.courseoutcomes.com

1. Why was the SOX legislation passed? To protect against accounting corruption in… a) privately run businesses b) public companies c) government entities d) non-profit corporations © 2011 Course Outcomes, www.courseoutcomes.com

Notice that when all options begin with the same phrase, I place the phrase above the options. This reduces cognitive load so that the only thing you are testing is whether they have the skills to answer the item. (not endurance or reading comprehension). 1. Why was the SOX legislation passed? To protect against accounting corruption in… a) privately run businesses b) public companies c) government entities d) non-profit corporations © 2011 Course Outcomes, www.courseoutcomes.com

Test-wise student: A test-wise student has learned a few things about how test items are written and can answer test questions correctly without knowing anything about the subject matter. The following guidelines will help you avoid giving away your correct answers to test-wise students. You are probably “test-wise.” In your experience, what are some clues that give away or narrow down the correct answer option? © 2011 Course Outcomes, www.courseoutcomes.com

All options should: 1. belong to a similar category. Less Effective Example: Who discovered America? A.Amerigo Vaspugi B.Christopher Columbus C.Bill Clinton D.Eric the Red Less Effective Example: Who discovered America? A.Amerigo Vaspugi B.Christopher Columbus C.Bill Clinton D.Eric the Red Not from the same category Not from the same category Effective Example: Who discovered America? A.Amerigo Vaspugi B.Christopher Columbus C.Francisco Pizarro D.Eric the Red Effective Example: Who discovered America? A.Amerigo Vaspugi B.Christopher Columbus C.Francisco Pizarro D.Eric the Red All from the same category All from the same category © 2011 Course Outcomes, www.courseoutcomes.com

All options should: 1. belong to a similar category. 2. be written at the same level of specificity. Less Effective Example: Select the equation for twice the sum of a number and seven. A.2 (N + 7) B.2N +7 C.(1.5 3 *N+56) / 8 D.2*7+N Less Effective Example: Select the equation for twice the sum of a number and seven. A.2 (N + 7) B.2N +7 C.(1.5 3 *N+56) / 8 D.2*7+N Not at the same level of specificity or generality Not at the same level of specificity or generality Effective Example: Select the equation for twice the sum of a number and seven. A.2 (N + 7) B.2N +7 C.2 + 7N D.2*7+N Effective Example: Select the equation for twice the sum of a number and seven. A.2 (N + 7) B.2N +7 C.2 + 7N D.2*7+N All at the same level of specificity or generality All at the same level of specificity or generality © 2011 Course Outcomes, www.courseoutcomes.com

All options should: 1. belong to a similar category. 2. be written at the same level of specificity. 3. have similar grammar forms with the stem and each option. Less Effective Example: The statistical category that includes correlation and regression is called an A.Association B.Central tendency C.Dispersion D.Robust estimator Less Effective Example: The statistical category that includes correlation and regression is called an A.Association B.Central tendency C.Dispersion D.Robust estimator Grammatically, a test-wise student will notice that “association” is the only option that can be preceded by “an” Grammatically, a test-wise student will notice that “association” is the only option that can be preceded by “an” Grammatically aligned Grammatically aligned And the last option begins with an adjective And the last option begins with an adjective Notice that the first three options are nouns Effective Example: The statistical category that includes correlation and regression is called a(n) A.Association B.Central tendency C.Dispersion D.Frequency Effective Example: The statistical category that includes correlation and regression is called a(n) A.Association B.Central tendency C.Dispersion D.Frequency © 2011 Course Outcomes, www.courseoutcomes.com

All options should: 1. belong to a similar category. 2. be written at the same level of specificity. 3. have similar grammar forms with the stem and each option. 4. be similar in complexity-length, vocabulary, thought, etc. Less Effective Example: What factor played the largest role in the demise of the Soviet financial system? A.Their communist system had disincentivized their workforce B.Miltiary pressure C.Inflation D.Corruption Less Effective Example: What factor played the largest role in the demise of the Soviet financial system? A.Their communist system had disincentivized their workforce B.Miltiary pressure C.Inflation D.Corruption Effective Example: What factor played the largest role in the demise of the Soviet financial system? A.Disincentivized workforce B.Military pressure C.Inflation D.Political Corruption Effective Example: What factor played the largest role in the demise of the Soviet financial system? A.Disincentivized workforce B.Military pressure C.Inflation D.Political Corruption © 2011 Course Outcomes, www.courseoutcomes.com

All options should: 1. belong to a similar category. 2. be written at the same level of specificity. 3. have similar grammar forms with the stem and each option. 4. be similar in complexity-length, vocabulary, thought, etc. 5. be associated with the stem. Less Effective Example: What heightened Israel consciousness among all sectors of the American Jewish community? A.World War II B.The Six Day War C.The Persian Gulf War D.The Vietnam War Less Effective Example: What heightened Israel consciousness among all sectors of the American Jewish community? A.World War II B.The Six Day War C.The Persian Gulf War D.The Vietnam War Effective Example: Which war most raised American Jews consciousness of the nation of Israel? A.World War II B.The Six Day War C.The Persian Gulf War D.The Vietnam War Effective Example: Which war most raised American Jews consciousness of the nation of Israel? A.World War II B.The Six Day War C.The Persian Gulf War D.The Vietnam War Notice you are asking about what heightened consciousness but all the options are about wars. Notice you are asking about what heightened consciousness but all the options are about wars. © 2011 Course Outcomes, www.courseoutcomes.com

All options should: 1. belong to a similar category. 2. be written at the same level of specificity. 3. have similar grammar forms with the stem and each option. 4. be similar in complexity-length, vocabulary, thought, etc. 5. be associated with the stem. Less Effective Example: How do the SOX accounting laws protect investors? They protect investors from… A.loss B.gain C.inflation D.risks Less Effective Example: How do the SOX accounting laws protect investors? They protect investors from… A.loss B.gain C.inflation D.risks Effective Example: How do the SOX accounting laws protect investors? They protect investors by double- checking the________ in reporting. A.accuracy B.consistency C.competency D.legality Effective Example: How do the SOX accounting laws protect investors? They protect investors by double- checking the________ in reporting. A.accuracy B.consistency C.competency D.legality Notice you are asking HOW, not WHAT the laws protect from. Notice you are asking HOW, not WHAT the laws protect from. © 2011 Course Outcomes, www.courseoutcomes.com

All options should: 1. belong to a similar category. 2. be written at the same level of specificity. 3. have similar grammar forms with the stem and each option. 4. be similar in complexity-length, vocabulary, thought, etc. 5. be associated with the stem. 6. contain only one idea. Less Effective Example: What caused the explosion at the Danville Processing Plant? A.Electrical problems and safety oversight B.Sabotage from within C.Too relaxed managerial style Less Effective Example: What caused the explosion at the Danville Processing Plant? A.Electrical problems and safety oversight B.Sabotage from within C.Too relaxed managerial style Effective Example: What caused the explosion at the Danville Processing Plant? (select all that are correct). A.Electrical problems B.Safety oversight C.Sabotage from within D.Too relaxed managerial style Effective Example: What caused the explosion at the Danville Processing Plant? (select all that are correct). A.Electrical problems B.Safety oversight C.Sabotage from within D.Too relaxed managerial style Two Ideas © 2011 Course Outcomes, www.courseoutcomes.com

All options should: 1. belong to a similar category. 2. be written at the same level of specificity. 3. have similar grammar forms with each other and the stem. 4. be similar in complexity-length, vocabulary, thought, etc. 5. be associated with the stem. 6. contain only one idea. Distracters should be enticing to those who: 1. have common misconceptions. 2. have incomplete knowledge. 3. use careless reasoning. 4. are less informed. In translation, the language you translate into is known as the _________ language. A.Source B.Intended C.Target D.End In translation, the language you translate into is known as the _________ language. A.Source B.Intended C.Target D.End Wrong answers or Distracters © 2011 Course Outcomes, www.courseoutcomes.com

All options should: 1. belong to a similar category. 2. be written at the same level of specificity. 3. have similar grammar forms with each other and the stem. 4. be similar in complexity-length, vocabulary, thought, etc. 5. be associated with the stem. 6. contain only one idea. Distracters should be enticing to those who: 1. have common misconceptions. 2. have incomplete knowledge. 3. use careless reasoning. 4. are less informed. © 2011 Course Outcomes, www.courseoutcomes.com

1. Simplify—make the item as short as possible. 2. Do not favor one letter option above another (a, b, c, d). 3. Brainstorm with a colleague to write meaningful distracters or have another few persons review them. 1. Don’t get too attached to your distracters because others may not see what you see in them. Be prepared to revise your items upon review. 4. Review how test-takers responded to your items and revise them based on this (this may involve consulting with a psychometrician). © 2011 Course Outcomes, www.courseoutcomes.com

1. Write or revise existing items that align to your General Instructional Objectives (GIOs) and Specific Learner Outcomes. 2. I find the following format works well: GIO = Understand the SOX laws. SLO = IDENTIFY the purpose of each SOX section. This item will measure if a person can IDENTIFY that section II protects against conflicts of interest. What was section II of the SOX laws meant to prevent? a. estimation procedures of past performance b. forecasted sale of future assets c. negligent practices for reporting losses d. conflicts of interest of external auditors* Feedback: Section II of the SOX laws protects against conflicts of interest of external auditors. © 2011 Course Outcomes, www.courseoutcomes.com

If you have any questions or need assistance check the discussion board or contact Dr. Daniel Winder at the courseoutcomes@gmail.com courseoutcomes@gmail.com © 2011 Course Outcomes, www.courseoutcomes.com

In April 2011, The CPE PhD, LLC became Course Outcomes, LLC. The training materials are referred to in this packet are owned and copyrighted by Course.

Similar presentations

Presentation on theme: "In April 2011, The CPE PhD, LLC became Course Outcomes, LLC. The training materials are referred to in this packet are owned and copyrighted by Course."— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

In April 2011, The CPE PhD, LLC became Course Outcomes, LLC. The training materials are referred to in this packet are owned and copyrighted by Course.

Similar presentations

Presentation on theme: "In April 2011, The CPE PhD, LLC became Course Outcomes, LLC. The training materials are referred to in this packet are owned and copyrighted by Course."— Presentation transcript:

Similar presentations

About project

Feedback