What is mathematical understanding? How can we measure it? Theoretical and empirical reflections Dr. Jon STAR Harvard University Seminar Presentation, Department of Elementary Education Seminar, Middle East Technical University, Ankara, Turkey; 27 August 2007

2 Acknowledgements Dr. Cigdem HASER –Department of Elementary Education METU

3 Some background information Who is Dr. Jon STAR? What is his connection to METU? What is the format of this seminar? Who is the intended audience for this seminar? Can I interrupt the seminar with questions?

4 My goals for the seminar Push you to think more carefully about –What “understanding” means –How understanding is assessed Convince you (if you need convincing!) that the nature of and assessment of understanding are issues that you need to think seriously about in your work with teachers and students

5 Getting started Please get in groups of 2-3 people and share your answers to the following questions: 1.What does it mean to understand a mathematical topic? 2.How can you tell when someone has developed understanding of a mathematical topic?

6 “Understanding” words? understand, know, explain, concept, procedure, rote, memorize, transfer, apply, recall, generate, prove, problem solving, network, connections, web, schema,...

7 Does this assess understanding?

8

9

10 Does this assess understanding?

11 Does this assess understanding?

12 Does this assess understanding?

13 Generalizing? Can you make any general statements about what understanding is and how to assess it? Five categories of responses to the question –“What does it mean for someone to understand a mathematical topic”?” And my thoughts on each...

14 1. Understanding is hard to define. I know it when I see it, but I can’t define it Understanding is difficult to define precisely People smarter than I am have tried and failed over the past several thousand years! Is it really necessary to come up with a good definition of “understanding”? What is understanding?

15 Jon says... YES, it is necessary to come up with a definition Understanding is often the instructional outcome that we aim for. –We need a target when we are designing instructional materials Need to measure whether our goal has been achieved –Designing measures requires a careful and precise definition and operationalization of understanding 1. Understanding is hard to define.

16 2. Cite an expert. You can’t expect me to re-invent the wheel! I use understanding in the same way that (insert your favorite mathematics education scholar here) does. –For example, Hiebert, Skemp, Brownell, Sfard, Star? If another framework is widely used and accepted, isn’t it OK to just use it? What is understanding?

17 Jon says... Certainly it is important to ground your ideas about understanding with the established literature It helps to connect with a literature that is well- known and a framework for understanding that is commonly used But Cite an expert.

18 Jon says... Most who “piggy-back” on others’ frameworks are not particularly knowledgeable about who and what they are citing –Pervasive citation of Hiebert (1986) and Skemp (1978) fall into this category Before citing someone’s ideas about understanding, become very familiar with the framework 2. Cite an expert.

19 Jon says... Many assume that different frameworks mean the same thing –For example, that procedural/conceptual (Hiebert) is the same as relational/instrumental (Skemp) Different frameworks and terminological distinctions shed light on different aspects of understanding –All theoretical frameworks are not the same 2. Cite an expert.

20 Assessment frameworks Many adopt a framework because of its use in national or international assessments –Widespread use of an assessment framework is not a compelling reason to choose a framework Assessment frameworks are idiosyncratic and difficult to faithfully apply outside of the assessment –Assessment frameworks often not theoretically sound 2. Cite an expert.

21 Consider these frameworks NAEP (US national maths assessment) –Procedural knowledge, Conceptual understanding, Problem solving PISA (large international maths assessment) –Reproduction, Connections, Reflection TIMSS (large international maths assessment) –Solving routine problems, Knowing facts and procedures, Reasoning, Using concepts 2. Cite an expert.

22 TIMSS th grade M Knowing facts and procedures Solving routine problems Knowing facts and procedures Reasoning Using concepts

23 TIMSS th grade M Solving routine problems Solving routine problems Knowing facts and procedures Reasoning Using concepts

24 TIMSS th grade M Using concepts Solving routine problems Knowing facts and procedures Reasoning Using concepts

25 TIMSS th grade M Reasoning Solving routine problems Knowing facts and procedures Reasoning Using concepts

26 NAEP 8th grade M6 #27 Conceptual understanding Conceptual understanding Procedural knowledge Problem solving

27 NAEP 8th grade M7 #18 Procedural knowledge Conceptual understanding Procedural knowledge Problem solving

28 NAEP 8th grade M7 #11 Problem solving Conceptual understanding Procedural knowledge Problem solving

29 Jon says... Know what you are getting into when you cite an expert! Don’t assume that assessment frameworks reflect a coherent and theoretically sound vision of what understanding is 2. Cite an expert.

30 3. Knowledge organization in head. Understanding refers to how knowledge is organized in someone’s memory/brain/head. Understanding is when knowledge is tightly connected, is a web, is a network, is organized into schema, etc. What is understanding?

31 Jon says... It may be true that understanding is indicated by or a result of a certain organization of knowledge in the head It may be helpful to use metaphors and abstractions (such as links and connections) to talk about knowledge organization But Knowledge organization in head.

32 Jon says... Defining understanding this way is not particularly helpful in designing assessments The metaphors or abstractions are not directly transferable to learning and assessment –If links = understanding, do we teach links? –Does the ability to draw a concept map (with lots of links in a network) indicate understanding? 3. Knowledge organization in head.

33 4. Quality of verbalization. A student understands if he/she can explain it to someone else. If the student knows it well enough to provide a thorough and principled explanation to someone else, I am satisfied that he/she understands. What is understanding?

34 Jon says... Certainly the ability to verbally explain is a potential indicator that a student understands But Quality of verbalization.

35 Jon says... Impractical in terms of assessment; we can’t interview every child to see if he/she understands –We need a more efficient way to tap understanding Students’ verbalizations are often spontaneous –Typically not a well-thought-out, carefully worded, articulate response –Requires an interviewer to carefully probe and explore the student’s ZPD 4. Quality of verbalization.

36 5. Quality of written performance. Understanding is evident in what students do When given a task/problem, we judge whether or not the student understands based on his/her performance Verbalization can give an additional window into what students are thinking, and knowledge organization is potentially useful as a metaphor “Intelligent performances” (Ryle) What is understanding?

37 Jon says... Closest to what I believe Major implications for assessment –It is incumbent on me to design very good questions so that how students respond does indicate that they understand Let’s talk about creating good assessments of understanding! 5. Quality of written performance.

38 Assessments of understanding Don’t be swayed by existing frameworks for categorizing items or by the assignment of particular items to categories Questions from other assessments may be quite useful, but the category label (“Solving routine problems”) is likely not helpful

39 Avoid stereotypes Stereotyped views of conceptual knowledge –“If it is possible to complete this question using a memorized procedure, then it does not assess understanding” –“If it is a multiple choice question, it cannot assess conceptual understanding”

40 Avoid stereotypes Stereotyped views of procedural knowledge –Procedures are either known (and student can execute them, often by rote) or they are not known Also consider items that assess “deep procedural knowledge” –knowledge of multiple procedures, knowledge of which procedures are better for certain circumstances, ability to adapt procedures to changing circumstances, ability to evaluate procedures See Star, 2005, 2007, articles in J. for Research in Mathematics Education for more on this.

41 Avoid “problem solving” Many assessment frameworks include a category called “problem solving” Overlaps with other categories (e.g., procedural knowledge and conceptual knowledge) Word problems fall into multiple categories, not just problem solving Has become a political term and thus relatively useless for mathematics educators

42 Name the concept When designing assessments of conceptual knowledge, it is important to be able to name the target concept Conceptual knowledge as knowledge of a concept –Rather than as a descriptor for a quality of knowledge

43 Validity Evaluate the psychometic validity of your assessments –Cronbach’s alpha for inter-item reliability –Factor analyses to establish or confirm ‘grouping’ of items

44 Bottom line Be thoughtful and deliberate about the frameworks, terms, and citations that you use Make sure your definition of understanding is tightly linked to the ways that you operationalize and assess it

45 Examples from my research I assess three components of understanding (of algebra equation solving) Conceptual knowledge Procedural knowledge Procedural flexibility

46 Conceptual knowledge Knowledge of key concepts used in equation solving –Concept of equivalence –Concept of variable

47 For example If m is a positive number, which of these is equivalent to (the same as) m + m + m + m? (Responses are: 4m; m 4 ; 4(m + 1); m + 4) –Concept of variable For the two equations: 213x = x = Without solving either equation, what can you say about the answers to these equations? Explain your answer. –Concept of equivalence

48 Procedural knowledge Knowledge of procedures –Ability to successfully execute equation solving procedures on problems that are similar to those seen in the instructional intervention (“familiar” or “learning” problems) –Ability to successfully execute equation solving procedures that are somewhat different than those seen in the intervention (“transfer” problems)

49 Procedural flexibility Ability to generate, recognize, and evaluate multiple solution methods for the same problem One example of “deep procedural knowledge” Short-answer and multiple choice assessment designed to measure flexibility (e.g., Beishuizen, van Putten, & van Mulken, 1997; Blöte, Klein, & Beishuizen, 2000; Blöte, Van der Burg, & Klein, 2001; Star, 2005, 2007; Star & Seifert, 2006; Rittle-Johnson & Star, 2007)

50 Examples of flexibility items A student’s first step for solving the equation 3(x + 2) = 12 was x + 2 = 4. What step did the student use to get from the first step to the second step? Do you think that this way of starting this problem is (a) a very good way; (b) OK to do, but not a very good way; (c) Not OK to do? Explain your reasoning. For the equation 2(x + 1) + 4 = 12, identify all possible steps (among 4 given choices) that could be done next. Solve 4(x + 2) = 12 in two different ways.

51 Also, direct measure of flexibility Look at students’ strategies on procedural knowledge items How they solve equations, and what this indicates about their knowledge of equation solving strategies

52 Successes Reliable measure of flexibility that does assess an aspect of understanding that is of particular interest to me –Creating and refining the assessment also forced us to think deeply about what flexibility means Thinking about understanding and thinking about assessing understanding interact with each other

53 Challenges Creating a valid assessment for conceptual knowledge –Several items that seem to target key concepts –Items are successful in measuring students’ learning from our instructional intervention –But items as a whole are not as strong from a psychometric standpoint as we would like

Thanks! You can download this presentation and other related papers and talks at Jon Star