William H. Angoff Distinguished research scientist at ETS for more than 40 years. Angoff Method: One of his many contributions to psychometrics. Author of “Scales, Norms, and Equivalent Scores”.
Angoff Method: Determines the passing % or cutscore. Most widely used SME are subject matter experts who examine each question. Predict how many minimally qualified candidates would answer correctly. The average of the SME’s predictions becomes its predicted difficulty. The sum of the predicted difficulty values averaged is the recommended cut score Relies on SME Rate each test item To determine whether a minimally qualified candidate would answer the item correctly or incorrectly. Goal
Mechanics of Psychometrics Statistics Validity Reliability
A Reliable Measure: Measures test items consistently across time, individuals, and situations. Reliability is necessary, but not sufficient for validity. Alpha Coefficient Retest Test
Validity Valid Measure: Measures what it is intended to measure. Sample question: What is the capital of Texas? This question is a reliable measure but would not be a valid measure for the CCP exam. A valid answer might be Houston. A valid measure for a CCP exam must measure only CCP components such as Cost Estimating, Scheduling, Project Management, etc.
Negative Correlation This Negative number indicates that those who did well on the overall exam actually performed very poorly on this particular item.. Sometimes it indicates that the answer might be keyed incorrectly. Sample question: What is the capital of Texas? In this case, some candidates who passed the overall exam did not get this question correct – as marked. They selected the distractor “Austin”. A valid answer was thought to be “Houston” and was erroneously marked as correct answer. This statistic shows that a dis-proportionate number of successful candidates missed this question. So, those questions with Negative Correlation are reviewed very closely. Sometimes, candidates “over-think” the question.
Test Bias A form of systematic (non-random) error which leads to examinees from one demographic group having an advantage over examinees from another demographic group. Test bias may cause differences in average scores across demographic groups. Example: USA uses different system of measurement than Europe. They use metric system so they may have problems with our questions re measurement. But differences in group scores are not sufficient evidence that Test Bias is present. AACE Certification Board has committee working on problem – Goal is to reduce Test Bias.
Statistics Both Reliability & Validity can be assessed Statistically. P-Value (Pass Rate or Item Difficulty)Point Biserial Correlation (Rbpis)Negative CorrelationAlpha Correlation Statistical Assessments:
Easy Test Question 90% tested answer correctly. Sample Question: What is the Capital of Florida? a.Miami b.Tallahassee c.Jacksonville d.Orlando
Easy Test Question Sample Question: What is the Capital of Florida? a.Miami b.Tallahassee c.Jacksonville d.Orlando
Hard Test Question 50% tested answer incorrectly. Sample Question: What is the Capital of Michigan? a.Detroit b.Ann Arbor c.Lansing d.Kalamazoo
Hard Test Question Alternative is to change distractor(s) Sample Question: What is the Capital of Michigan? a.Detroit – change to say, Boston b.Ann Arbor – change to say, Cloquet c.Lansing d.Kalamazoo
Hard Test Question 50% tested answer incorrectly but what if we changed distractor answers? Sample Question: What is the Capital of Michigan? a.Boston b.Cloquet c.Lansing d.Kalamazoo
Hard Test Question Sample Question: What is the Capital of Michigan? a.Detroit b.Ann Arbor c.Lansing d.Kalamazoo
Sample of Math Question Less than 60% tested answer incorrectly. Sample Math Question: What is $5,100 invested at 12% compounded yearly, at end of 6 years? a.$5,712 b.$8,988 c.$10,066 d.$98,950
Solution to Math Problem Solution : There are a couple of different ways to work this. Start BeginRateEnd Year 1$5,1001.12$5,712 Year 2$5,7121.12$6,397 Year 3$6,3971.12$7,165 Year 4$7,1651.12$8,025 Year 5$8,0251.12$8,988 Year 6$8,9881.12 $10,066
Sample Math Problem - Tips for Solution There are a couple of different ways to work this example. Solution : Work up a Table shown on previous slide. Solution : Remember the formulas. There should be some basic formulas you want to remember. Solution : Remember the ‘Rule of 72”. To determine the amount of time to double an investment, divide 72 by the interest rate. In this case, 72/12 = 6 years. The $5k should double and maybe answer is approx $10k. Get rid of obviously wrong answers. These would be answer a and d. Answer a is too small and d is too high. That narrows it down to 2 plausible answers. If you had to guess, odds are 50% instead of 25%.
Certification Board Review: Review the question to confirm that the marked answer is correct. Change the wording of the question. Modify the distractor to add or decrease the difficulty. Make no changes. Retire the question. Steps in Making the Question Easier or Harder
Top Tips to Get a Great Score Purpose Understand the purpose of the exam. Expect Know what to expect in the timed testing environment. Attitude A positive attitude is a most.
Top Tips to Get a Great Score Practice Especially necessary for anyone who has never taken a Psychometric Exam. Find websites & books that have practice test. Focus on tests that specialize in your area(s) of weakness. Time Yourself Test yourself working against the clock. Broaden Sample Test Don’t rely only on the study guides. Use AACE Recommended Practices (RP) on their web page. Many of the new questions on exams come from the new RPs.
Top Tips to Get a Great Score Refresh Math Skills Fact 1: AACE has Math Questions that other exams do not. Fact 2: Cost Engineers are expected to know the Rate of Return and other Math problems. Get a Good Night’s Sleep Make sure you get a good night’s sleep before the exam so you are fully alert. Follow Instructions Carefully read or listen to the instructions on taking the exam.
Top Tips to Get a Great Score Answering Questions Trust yourself…Usually your first answer is correct. Answer easy questions first, skip those you get stuck on. At the end, return to the questions you skipped. Do not leave any questions unanswered. Eliminate all obviously wrong choices. Last resort…Guess if you need to as you have a 25% chance of getting it right. Complete the Feedback Section on Exam Complete the Feedback Section at the end of the exam. Your comments or concerns help the Reviewers modify and improve future exams. Don’t Give Up Keep trying until you Pass! Identify what areas that kept you from passing. Study these problematic areas to improve your skills.
Conclusion: Achieve Certification Use Angoff Method Learn the Language Implement Processes Study & Prepare for Exam
Technical Paper Know the Basic Guidelines used to determine whether your Technical Paper is acceptable CCP’s Need Passing Grades on the Certification Exam and the Technical Paper for Certification Graded Independently of the Certification Exam Must Submit Your Technical Paper Before You Can Take the Exam Important information you should know before writing your paper
Tips for a Winning Paper Table of Contents Introduction: Purpose & Background Basis & Analysis of Your Data References 1 2 3 4 5 Abstract Make sure your paper Includes:
Tips for a Winning Paper Graphs or Tables to Support Premise Logical Conclusion Do Word Count: Minimum 2,500 Req. Extra Points for Appendix 6 7 8 9 10 Organization & Clarity Make sure your paper Includes: