Evidence & Preference: Bias in Scoring TEDS-M Scoring Training Seminar Miami Beach, Florida

Bias Bias is part of being a thinking person. Bias is part of the way we see the world. Bias is our personal way of interacting with the world. Bias is built into the issues important to us. Bias is part of our disagreements with others – sometimes we dont agree with anyone. Bias informs how we evaluate our own work.

Judgments We make judgments throughout the day, about how we conduct ourselves and how we interact with others – how we do our work and evaluate the work of others. In part, our judgments originate from assumptions because of experience. As we grow, we receive messages about competence, accomplishment, ways of talking and behaving, appearance, gender, race, class.

Influence We use certain assumptions or biases, often unconsciously, to make judgments about things so we can understand our surroundings in our terms. Each of us prefer specific approaches and strategies for doing quality work and we know when we see quality work and effort. We often critique others on characteristics we do not like to see in ourselves.

Evidence As we score responses, particularly those on the pedagogy-related tasks, we need to be able to separate our personal biases from the evaluation of evidence of understanding, knowledge, and skill. A focus on evidence provided in a response will provide scores that are meaningful and useful to inform

Bias-Free Evaluation Not possible. We all hold specific biases – and that is in part a good thing – they hold us accountable. At times, our biases interfere with our ability to understand situations and others. At times, our biases interfere with fair assessment of the performance of others.

Validity The degree to which we have evidence to suggest that the inferences, interpretations, and uses of assessment results are meaningful, appropriate, and useful. An argument we develop to support our uses of assessment results. Variation in scores is construct-relevant.

Toward Fair Assessment We can identify our own biases. We can learn how biases interfere with our assessment of performance. We can see how bias affects judgment. We can separate bias from judgment.

Scoring & Professional Development Scoring provides a great opportunity for professional development. However, scoring is not about you. Scoring is about assessing Future Teacher performance. Scoring is about identifying aspects of understanding, knowledge, skill, competence.

Scoring Tools The scoring system is developed to guide scoring toward evidence and to focus scorers on the nature of mathematics and mathematics pedagogy. Scoring training, scoring guides, and examples are provided as guides to focus judgments. Providing opportunities to uncover personal biases allows us to attend to them, to keep them under control.

Skills for Teaching Mathematics What WOULD you like to see in responses to pure or contextual mathematics problems? What WOULD you like to see in responses to problems seeking mathematical knowledge for teaching? How do you know that a teacher is prepared to teach mathematics to diverse students?

Bias Training Tool (1) Developing a List of Personal Preferences Allows us to identify and control the impact of our personal preferences. This is a list that can be developed in the initial training session and scorers can add to the list as they continue scoring. This list should be in front of us as we score.

Bias Training Tool (2) Develop a List of Teaching Practices that we prefer or try to teach in our profession. Identify those that are construct-relevant. Identify those that are construct-irrelevant. Teaching practices I like Teaching practices I dont like 1. 2.

Bias Training Tool (3) Create a List of Characteristics of people that are different than us. – How might these characteristics be evident in responses – affect responses? Create a List of Characteristics of Competence in teaching mathematics. – What characteristics are superficial and should be guarded when scoring?

Biases About Written Responses ISSUES: Grammar, spelling Organization Failure to follow directions Concise writing; response length Missing words Descriptive writing INTERPRETATIONS: Careless, lazy Poorly educated Does not want to demonstrate limited knowledge Limited evidence Little to say – covering up lack of knowledge

More Biases About Written Responses Use of non-standard form of the language Fluent writing Drawing ability Providing grade-level examples

Interpreting the Scoring Guide Some scorers will interpret scoring guides literally – word for word with no room for interpretation Others will interpret scoring guides with a great deal of interpretation

Consistency & Fairness Consistency is often seen as an important characteristic of Fairness. Consistency improves comparability. Consistency may require a level of agreement that cannot be achieved. Consistency can restrict the opportunity to fairly judge novel, unique, or culturally appropriate approaches.

Training Provide an opportunity for scorers to identify their own biases. Provide time for discussion of how biases may influence scoring. Discuss methods for reducing the degree to which construct-irrelevant influences affect scores.