Using Student Data to Inform Design Force and Motion Tom Regan Session C, Strand 6: Assessment KSI 2006.

Presentation on theme: "Using Student Data to Inform Design Force and Motion Tom Regan Session C, Strand 6: Assessment KSI 2006."— Presentation transcript:

Using Student Data to Inform Design Force and Motion Tom Regan Session C, Strand 6: Assessment KSI 2006

Copyright © 2006 AAAS Project 2061 2 Outline Generic Items “Balanced” & “Unbalanced” Speed Table Arrows v. Words Logical Operators

Copyright © 2006 AAAS Project 2061 3 Pilot Test Sites 6 th grade; four classrooms Suburban 10% Free & Reduced Lunch Had studied “Forces’ Effect on Matter” 8 th grade; four classrooms, one G/T Small town 33% “economically disadvantaged” Had 15 days’ exposure to forces and motion

Copyright © 2006 AAAS Project 2061 4 Interview Sites 8 th grade; three students Urban school 78% FRE Had studied motion 7 th & 8 th grade; ~ 10 students. Most students 8 th grade G/T Suburban 11% FRE Had one year of physical science, including forces & motion

Copyright © 2006 AAAS Project 2061 5 Generic Items Item 36-2, Classroom 2A6B1 (6 th grade; n=34) An object is moving. A force acts on the object in the direction of the object's motion. What will happen to the object's motion while this force acts? A. (47%) The object's motion will depend on the type of force. B. (6%) The object's speed will stay the same. *C. (21%) The object will speed up. D. (15%) The object will slow down. Items that used generic “force” and “object” rather than, for example, “box” and “push”

Copyright © 2006 AAAS Project 2061 6 Selected Student Comments on Answer Choices (1 of 2) Opposing Force (8 of 34 students) –Students thought that the force is in the opposite direction to the object’s motion. –Ex. “Why would it speed up if the force is in the way of its motion?” Two Forces (4) –Students thought that there are two forces acting on the object. –Some of these were “force of motion” misconception Introduction of Extraneous Forces (2) –Students introduced the forces of gravity and friction into the problem –These students don’t realize that the force mentioned in the stem is the only force on the object

Copyright © 2006 AAAS Project 2061 7 Selected Student Comments on Answer Choices (2 of 2) Type of Force (3 of 34) –Students thought that Hard/Soft and Big/Little are types of forces –Item writer’s intent: “Type” includes everything about force besides strength, direction, and duration –Popularity of choice A suggests that this was not students’ interpretation. Force “hits” (4) –Students used the word “hit” where we would say “acts on” –Could this reflect/lead to confusion with an impulsive force? No students commented on the word “object”

Copyright © 2006 AAAS Project 2061 8 Generic Item Interviews Suburban School: ~6 students, mostly 8 th grade G/T (two students) prefer generic to particular (three students) compare situation to pulling-cart-lab (and get the right answer). (one student) Is it a “tap” force or a “constant” force? 7 th grade G/T, hadn’t taken physics: prefers particular to generic Urban School 8 th grade: either generic or particular is okay (gets right answer)

Copyright © 2006 AAAS Project 2061 9 “Type of force” Distractor Item 36-2, Classroom 2A6B1 (n=34) An object is moving. A force acts on the object in the direction of the object's motion. What will happen to the object's motion while this force acts? A. 47% The object's motion will depend on the type of force. B. (6%) The object's speed will stay the same. *C. (21%) The object will speed up. D. (15%) The object will slow down. Many students selected answer choice A It is an effective distractor and should be retained. It distracts students from the targeted learning goal (Newton’s 2 nd Law) and should be removed. ?

Copyright © 2006 AAAS Project 2061 10 “Balanced” and “Unbalanced” Forces Item 38-2, Classroom 2A6B2 (6 th grade; n=14) and Item 57-1, Classroom 2A6A2 (6 th grade; n=26) An object is [38-2] speeding up / [57-1] slowing down. Which of the following statements about the forces acting on the object is TRUE? A. (29%) / (35%) Balanced forces are acting on the object. B. (*50%) / (12%) An unbalanced force is acting on the object in its direction of motion. C. (14%) / (*35%) An unbalanced force is acting on the object in the direction opposite to its motion. D. (7%) / (8%) No forces are acting on the object.

Copyright © 2006 AAAS Project 2061 11 Whether an object speeds up or slows down depends on the direction of the unbalanced force relative to the motion But some students think that speeding up/slowing down depends on whether the force is balanced or unbalanced: Forces are balanced  object speeds up3 of 14 students Forces are unbalanced  object slows down2 of 14 students Forces are balanced  object stops3 of 14 students Forces are balanced  object slows down3 of 26 students Forces are unbalanced  object speeds up1 of 26 students Forces are balanced  object stops2 of 26 students Some Student Ideas about Balanced/Unbalanced Speeding up item Slowing down item Your experiences with balanced/unbalanced?

Copyright © 2006 AAAS Project 2061 12 Speed Table FM64-1 In the drawing below, the arrow labeled "FORCE" represents a force acting on an object. The length of the arrow represents the strength of the force, and the direction of the arrow shows the direction of the force. The arrow labeled "Motion" shows the direction of the object's motion. Every second, starting when the clock reads "0 seconds," the speed of the object is measured. Which row of the table could be a correct representation of the object's speed between 0 seconds and 6 seconds? Clock ► 0 sec1 sec2 sec3 sec4 sec5 sec6 sec *Row A (~40%)10 mi/hr11 mi/hr12 mi/hr13 mi/hr14 mi/hr15 mi/hr16 mi/hr Row B (~10%)10 mi/hr Row C (~5%)10 mi/hr 13 mi/hr Row D (~15%)10 mi/hr11 mi/hr12 mi/hr13 mi/hr12 mi/hr11 mi/hr10 mi/hr

Copyright © 2006 AAAS Project 2061 14 Improving Distractor C Choice C is incorrect because the speed should change the entire time that the force acts, not just between 2 sec and 3 sec. Students wrote that it was incorrect because of the abruptness of the change in speed. Not exactly right. Perhaps abruptness is implausible. Alternate distractor Still incorrect, but perhaps more plausible Clock ► 0 sec1 sec2 sec3 sec4 sec5 sec6 sec Row C (~5%)10 mi/hr 13 mi/hr Clock ► 0 sec1 sec2 sec3 sec4 sec5 sec6 sec 10 mi/hr 11 mi/hr12 mi/hr13 mi/hr

Copyright © 2006 AAAS Project 2061 15 Speed Table Next Steps Specify the time-dependence of the force. Example: “The force is constant & acts for all six seconds.” Change distractor C. Pilot equivalent item with all-word stem (no force & motion arrows) to isolate source of confusion Compare to items using speed-time graphs.

Copyright © 2006 AAAS Project 2061 16 Arrows vs. Words: Arrow Item FM11-2: In the drawing below, the arrows labeled "WIND" and "WATER" represent forces acting on the sailboat. The length of the arrow represents the strength of the force, and the direction of the arrow shows the direction of the force. The arrow labeled "Motion" shows the direction of the sailboat's motion. Which of the following statements correctly describes the sailboat's motion while these forces act? *A. The sailboat will speed up. B. The sailboat will speed up for a short time and then slow down. C. The sailboat will move at a constant speed. D. The sailboat will speed up for a short time and then move at a constant speed.

Copyright © 2006 AAAS Project 2061 17 Equivalent Word Item FM11-3: A sailboat travels on a lake. Two horizontal forces act on the sailboat: the force of the wind on the sailboat and the force of the water on the sailboat. The force of the wind on the sailboat is in the same direction as the sailboat's motion, and the force of the water on the sailboat is in the opposite direction to the sailboat's motion. The force of the wind on the sailboat is stronger than the force of the water on the sailboat. Which of the following statements correctly describes the sailboat's motion while these forces act ? *A. The sailboat will speed up. B. The sailboat will speed up for a short time and then slow down. C. The sailboat will move at a constant speed. D. The sailboat will speed up for a short time and then move at a constant speed. There’s also a pair of equivalent items, 55-1 & 55-2, in which both forces oppose the motion

Copyright © 2006 AAAS Project 2061 18 Sixth Grade Results ItemP-An*ABCDNo choice Confusing Arrows (FM11-2; classroom 2A6A1) 33%2619%8%58%8%7%4% Words (FM11-3; classroom 2A6A2) 52%2719%15%37%15%14%33% ItemP-An*ABCDNo choice Confusing Arrows (FM55-1; classroom 2A6B1) 54%3364%0%21%12%3%9% Words (FM55-2; classroom 2A6B2) 40%1233%8%17% 25%33% “P-A” is a measure of the classroom’s force & motion preparation and/or ability. It is the average of the classroom’s scores on two common items.

Copyright © 2006 AAAS Project 2061 19 Eighth Grade Results Conclusion: Arrow items easier & clearer than word items ItemP-An*ABCDNo choice Confusing Arrows (FM11-2; classroom16A8A1) 23%1724%0%41%18%17%24% Words (FM11-3; classroom16A8A5) 16%1513%20% 13%34%40% ItemP-An*ABCDNo choice Confusing Arrows (FM55-1; classroom16A8B5) 29%2868%4% 11%13%14% Words (FM55-2; classroom16A8B4) 30%2425%4%33%13%25%33%

Copyright © 2006 AAAS Project 2061 20 Logical Operators: Must Item FM73-4, classroom 16A8A1 (8 th grade; n=18) A person pushes a box across the floor. There are two horizontal forces on the box: the force of the push, and the force of friction. The force of friction is in the opposite direction to the box's motion. The speed of the box is increasing. Which of these statements must be true? A. (11%) The force of friction is weak. B. (6%) The force of the push is strong. C. (6%) The force of the push is the same strength as the force of friction. *D. (61%) The force of the push is stronger than the force of friction. A & B could be true; D must be true

Copyright © 2006 AAAS Project 2061 21 Logical Operators: Goal of Item Targeted Learning Goals: –Newton’s 2 nd Law –Multiple Forces Goal of Piloting & Interviews: Determine whether the concept “must” can be used in items at the middle level. If students cannot handle “must,” then the item will not yield clear information about science content knowledge

Copyright © 2006 AAAS Project 2061 22 Logical Operators: Results Most students (11 of 18) got question right 9 students’ comments indicated some realization of the logical issue (7 of 9 got question right) A. The force of friction is weak –“The force may be weak but I think there is a better answer.” –“It could be, but it doesn’t explain what was happening, correctly.” B. The force of the push is strong –“That doesn’t mean it’s stronger than friction.” –“It doesn’t give enough information.” –The best comment, in response to A: “The force of friction may not be weak, it just may be weaker than the force of the push.”

Copyright © 2006 AAAS Project 2061 23 Logical Operators: Interviews 8 th grade G/T Students answer question correctly Students preferred D to A & B because it’s the “best answer.” Students were comfortable selecting the best answer. No mention of “must.”

Copyright © 2006 AAAS Project 2061 24 Logical Operators: Discussion Questions Which (if any) student comments demonstrate facility with “must” ? How to determine student’s engagement with “must” ? –With an item? –In an interview? Is it really an issue of science content? –“Students should know that an object’s change in motion depends on all the forces acting on the object.” Research Question: What if “Not Sure” were not an answer option? –# Not Sure per Student = 1.17