Presentation on theme: "Creating A Multiple Measures Placement System"— Presentation transcript:
1 Creating A Multiple Measures Placement System An ExerciseWithRon Gordon & Armand Brunhoeber
2 The ConceptAlthough test scores may predict failure, they do not necessarily predict success.Research shows that students’ backgrounds, environments, and personal habits may have more influence on their potential academic success than their residual academic skills.Find a way to factor that information into the placement decision at testing time.
3 Decision Process Which Courses Which Tests Back Ground Questions How manyVarietySelect Weight ValuesUse negative weights?How much possible total weight
4 Planning Flow Who Discipline faculty Testing staff Counseling IT Others – Research?
5 Planning Flow What Which courses in which disciplines What information can students provide at the time of testing?Which questions will be used in each discipline?How much total value should additional measures have?
6 Planning Flow Why State mandate? Literature Improve placement accuracy Give students better opportunities to learn
7 How Does it Work?Students answer locally developed background questions during the test sessionACCUPLACER placement rules compute a weight value for the student’s answers based on locally developed weighting rulesThe test score, plus a percentage based on the background information is used for placement
8 The Process Educate faculty and others on how the system functions Find examples of background questions and edit to fitAssign weight values to each answer choice.Write placement rules using the multiple measures editing function
9 Select QuestionsQuestions must, in some way, relate to student success.Must be information the student will have at the time of testingMust be multiple choiceShould solicit behavioral, historical/experiential, and environmental information
10 LimitsLimit total weight so that background information does not allow students to skip a course levelLimit number of questions to a manageable numberMore questions adds to testing timePlacement rules can become unmanageableAnswer choices must be mutually exclusive and all inclusive
11 Example (If Arithmetic, plus all weighted choices is >= 75 OR Algebra, plus all weighted choices is >= 48)AND(Algebra, plus all weighted choices is < 65 OR Algebra Not Taken) AND(CLM, plus all weighted choices is <62 ORCLM Not Taken)Then Placement is Elementary Algebra
12 Example, ContinuedIf this rule had 5 questions with 4 weighted answer choices each, there would be 80 lines just for weights.With too many questions, or too many choices per question, rules can become unmanageable
13 Conditional WeightsHigh School Accomplishments Have Limited Shelf LifeHow Much Does it Matter That a 25-Year-old Student Had 2 years of High School Algebra?Does it Matter That The Same 25-Year-old Student Works for a Surveyor and Uses Algebra Daily?
14 ExampleHow long has it been since you were enrolled in high school or other formal educational process?Less than 2 years or still enrolled2 to 5 yearsMore than 5 but less than 7 years7 years or moreUse high school data for up to 5 years, experience for more than 5 years.
15 Assigning WeightsTotal possible weight should not move student more than one level in either directionSet maximum possible weight so a student who scores at or above the midpoint of a placement range could move up, but one who scores below the midpoint could not.Use faculty to select BGQ and assign weightGuide them
17 Sample Question With Weights Which choice below best describes you when you read textbooks or other complex information?I usually need to read material several times before I understand it wellSometimes I can understand what I read the first time, but often I must reread it .00I usually understand what I read if I take notes or highlight passagesI always understand what I read the first time through
18 Think About Uploading to SIS Multiple measures may change placements, but not scores.SIS has no place to store multiple measuresTo upload multiple measures, the placement (course name) must be uploadedIn ACCUPLACER, course name can be numeric
19 Preparing to Build the System Assign numeric codes to course namesDetermine which tests will be used for each course in each disciplineCreate cut score TableCreate a BGQ weight Matrix
20 Building the System Create Background questions Assign BGQ to groups Create branching profilesCreate course groupsCreate courses and assign to groups ListCreate majors if usedCreate placement rules Edit1 Edit2placement rule sample.pdf
21 Verify Write most complex rule first Run verify function in branching profileUse several BGQ and score combinations to test the placement ruleCompute weighted score for each runTry to hit cut scores to test for bad weight or answer choice selections
22 Computing the Weight 1 Question Response # Weight High English A -.02 Grade EnglishCGoal Import.B.02Understand ReadD.03Study Time.01Total Weight+.04
23 Computing the Weight 2 Question Response # Weight High English A -.02 Grade EnglishFGoal Import.-.01Understand ReadD.03Study TimeCTotal Weight-.03
24 Computation Score is multiplied by 1 plus the accumulated weight. 85 * (1+.04) = 88.4Placement will be based on a score of 88.Example 285 * (1+ [-.03]) = 82.45Placement will be based on a score of 82
25 Common Errors Unequal weights between rules Misplaced Parentheses E.G. A response has .01 weight in one rule and -.01 weight in the next ruleMisplaced ParenthesesThe multiple measures weights make the rule larger and more difficult to visualizeMisuse of AND/ORMisuse of arithmetic operatorsWrong answer choice in rule line
26 TroubleshootingFrom the score report, determine what the student’s weight should be from the BGQ responsesUsing the weight, compute the weighted scoreDetermine what the placement should beExamine the appropriate rule for errors
27 Creating a Multiple Measures Placement System An exercise withRon Gordon & Armand BrunhoeberThank you for not throwing things at the presenters