1. Creating A Multiple Measures Placement System
An Exercise
With
Ron Gordon & Armand Brunhoeber

2. The Concept
Although test scores may predict failure, they do not necessarily predict success.
Research shows that students’ backgrounds, environments, and personal habits may have more influence on their potential academic success than their residual academic skills.
Find a way to factor that information into the placement decision at testing time.

3. Decision Process Which Courses Which Tests Back Ground Questions
How many
Variety
Select Weight Values
Use negative weights?
How much possible total weight

4. Planning Flow Who Discipline faculty Testing staff Counseling IT
Others – Research?

5. Planning Flow What Which courses in which disciplines
What information can students provide at the time of testing?
Which questions will be used in each discipline?
How much total value should additional measures have?

6. Planning Flow Why State mandate? Literature Improve placement accuracy
Give students better opportunities to learn

7. How Does it Work?
Students answer locally developed background questions during the test session
ACCUPLACER placement rules compute a weight value for the student’s answers based on locally developed weighting rules
The test score, plus a percentage based on the background information is used for placement

8. The Process Educate faculty and others on how the system functions
Find examples of background questions and edit to fit
Assign weight values to each answer choice.
Write placement rules using the multiple measures editing function

9. Select Questions
Questions must, in some way, relate to student success.
Must be information the student will have at the time of testing
Must be multiple choice
Should solicit behavioral, historical/experiential, and environmental information

10. Limits
Limit total weight so that background information does not allow students to skip a course level
Limit number of questions to a manageable number
More questions adds to testing time
Placement rules can become unmanageable
Answer choices must be mutually exclusive and all inclusive

11. Example (If Arithmetic, plus all weighted choices is >= 75 OR
Algebra, plus all weighted choices is >= 48)AND
(Algebra, plus all weighted choices is < 65 OR Algebra Not Taken) AND
(CLM, plus all weighted choices is <62 OR
CLM Not Taken)
Then Placement is Elementary Algebra

12. Example, Continued
If this rule had 5 questions with 4 weighted answer choices each, there would be 80 lines just for weights.
With too many questions, or too many choices per question, rules can become unmanageable

13. Conditional Weights
High School Accomplishments Have Limited Shelf Life
How Much Does it Matter That a 25-Year-old Student Had 2 years of High School Algebra?
Does it Matter That The Same 25-Year-old Student Works for a Surveyor and Uses Algebra Daily?

14. Example
How long has it been since you were enrolled in high school or other formal educational process?
Less than 2 years or still enrolled
2 to 5 years
More than 5 but less than 7 years
7 years or more
Use high school data for up to 5 years, experience for more than 5 years.

15. Assigning Weights
Total possible weight should not move student more than one level in either direction
Set maximum possible weight so a student who scores at or above the midpoint of a placement range could move up, but one who scores below the midpoint could not.
Use faculty to select BGQ and assign weight
Guide them

16. Multiple Measures Movement Model

17. Sample Question With Weights
Which choice below best describes you when you read textbooks or other complex information?
I usually need to read material several times before I understand it well
Sometimes I can understand what I read the first time, but often I must reread it .00
I usually understand what I read if I take notes or highlight passages
I always understand what I read the first time through

18. Think About Uploading to SIS
Multiple measures may change placements, but not scores.
SIS has no place to store multiple measures
To upload multiple measures, the placement (course name) must be uploaded
In ACCUPLACER, course name can be numeric

19. Preparing to Build the System
Assign numeric codes to course names
Determine which tests will be used for each course in each discipline
Create cut score Table
Create a BGQ weight Matrix

20. Building the System Create Background questions Assign BGQ to groups
Create branching profiles
Create course groups
Create courses and assign to groups List
Create majors if used
Create placement rules Edit1 Edit2
placement rule sample.pdf

21. Verify Write most complex rule first
Run verify function in branching profile
Use several BGQ and score combinations to test the placement rule
Compute weighted score for each run
Try to hit cut scores to test for bad weight or answer choice selections

22. Computing the Weight 1 Question Response # Weight High English A -.02
Grade English C
Goal Import. B
.02
Understand Read D
.03
Study Time
.01
Total Weight
+.04

23. Computing the Weight 2 Question Response # Weight High English A -.02
Grade English F
Goal Import.
-.01
Understand Read D
.03
Study Time C
Total Weight
-.03

24. Computation Score is multiplied by 1 plus the accumulated weight.
85 * (1+.04) = 88.4
Placement will be based on a score of 88.
Example 2
85 * (1+ [-.03]) = 82.45
Placement will be based on a score of 82

25. Common Errors Unequal weights between rules Misplaced Parentheses
E.G. A response has .01 weight in one rule and -.01 weight in the next rule
Misplaced Parentheses
The multiple measures weights make the rule larger and more difficult to visualize
Misuse of AND/OR
Misuse of arithmetic operators
Wrong answer choice in rule line

26. Troubleshooting
From the score report, determine what the student’s weight should be from the BGQ responses
Using the weight, compute the weighted score
Determine what the placement should be
Examine the appropriate rule for errors

27. Creating a Multiple Measures Placement System
An exercise with
Ron Gordon & Armand Brunhoeber
Thank you for not throwing things at the presenters