1. Bias & Fairness in Tests (Rust & Golombok)
Important to note: serious consequences follow from test results!
Think about it from the client’s perspective
Selection test: You don’t get the job
Academic test: You lose a year’s work
Clinical tests: get/don’t get help
Important to make the right decisions based on the results

2. Bias and fairness
How “correct” our decisions are can be thought of in terms of two properties:
Fairness - The social justice issues surrounding the employment of the test
Bias - A statistical artefact in the test which makes it respond differently to different groups

3. Fairness in tests
This is not so much a flaw the test as of the use of the test
Can only consider fairness in terms of the societal norms which apply
We can examine the pattern of decisions which have been made based on that test

4. Fairness and Justice
Depending on the focus on authority in a society, tests might be applied more or less strictly
An “unfair test” is one whose consequences do not match the value system of the society

5. Fairness: an example Imagine we set a super hard exam for 206F
Two thirds of the class fails the test
I can do two things:
Accept the marks as they are and see you next year
Adjust the test marks to increase the pass rate
Which is the fair thing to do?

6. Fairness: an example
If the society’s norms are such the institution is emphasised over theindividual, accepting the results is the fair thing to do
If the society’s norms are such that the individual is emphasised over the institution, adjusting the results is the fair thing to do

7. Bias in tests
A bias exists in a test if gives different results for different populations
Example: Army Alpha
Soldiers almost always
scored lower than
officers

8. Bias: good or bad?
Good: Bias can be used to identify which population a client belongs to
Should he be officer or soldier?
Bad: Creates a false impression of difference between groups
Foreign language ability can “reduce” intelligence

9. Bias: Good or Bad?
Irrelevant: If the test only really gets used by one population, who cares?
Decide on the importance of bias based on the situation
Bias is never an issue of “right” or “wrong” - it is a purely statistical concept

10. Forms of bias Bias can appear in three forms: Item bias
Intrinsic test bias
Extrinsic test bias
We can examine each of these sources of bias separately and address each individually

11. Item bias The bias exists in individual questions
eg. a questions about dollars, quarters and dimes would be biased
Linguistic bias (idioms, slang, etc)
common interracial bias
This type of item is common in IQ tests (!)

12. Identifying item bias Do item analysis
I.e. check out each item of the test separately
Identify possible relevant subgroups
Work out the “facility value” of each question for each group
the proportion of people who get it right

13. Item bias: example Imagine we have a test with 3 questions
We think it might be language biased
Look at the groups: native english speakers, others
Work out facility value
Native: A: 0.68 B: 0.96 C: 0.57
Other: A: 0.59 B: 0.24 C: 0.59
Big difference in item B, so it is biased

14. Item offensiveness
Eg: Shown an engineer and a psychologist “Which is smarter?”
Related to item bias
offensive items not necessarily biased
Offensive items should be removed
May interfere with subsequent items

15. Intrinsic test bias
The test has different mean scores for different groups
Does not exist in specific questions (Item bias), but rather a general phenomenon
Common in language groups
It is a matter of degree

16. Causes of intrinsic bias
Tests created with a specific group in mind are biased in this way
Other groups perform worse
The more different the group, the bigger the difference

17. Extrinsic test bias
When the test is unbiased, but decisions made using the test are biased
Eg. a test finds a true difference, and this leads to one group getting selected more than another
Extrinsic bias is the overlap between fairness and bias

18. Extrinsic bias: example
On the SAT, poorer children tend to perform worse than richer children
This is a real difference - poorer children have less access to the requirements to academic success
The SAT was used to select university applicants
poorer children were selected far less frequently

19. Extrinsic bias and ideology
Do you believe in “true differences”
Or do you rather believe in “unexplored potrential”?
It is a fact that poorer kids did worse at university (in the USA)
Do we use this as a basis for not selecting them?
No way to take a “scientific” ideology-free standpoint on extrinsic bias