1. California Educational Research Association Annual Meeting San Diego, CA November 18 – 19, 2010 Terry Vendlinski Greg Chung Girlie Delacruz Rebecca Buschang High Quality Practices for the Digital Age: Designing Games for Math Learning

2. 2 / 27 Overview Developing sound assessment and instruction Demonstration of Instructional Video Games Hands-on Experience Summary of Research Results Questions and Answers

3. 3 / 27 Developing Assessments and Instruction What is the goal of instruction? How do we know we have reached the goal? What do we think will get us to those goal? Did we achieve our goal? What do we do next?

4. 4 / 27 Knowledge Specifications What do we want students to learn? What key concepts are important to understanding? How do these concepts build on or integrate with what students should already know? How are Knowledge Specifications developed? Pre-think individual “experts” Delphi method or group discussion

5. 5 / 27 Knowledge Specifications - example Recurring problem with understanding fractions and operations with fractions (Siegler, et al., 2010; US Department of Education, 2008) Students should know how to add fractions What does “addition” mean? What does “fraction” mean? How does one apply that meaning of addition to fractions? What are some common ill- or mis-conceptions that students bring to our instruction?

6. 6 / 27 Knowledge Specifications – Adding Fractions 1.0.0 Does the student understand the importance of the unit whole or amount? The size of a rational number is relative to how one Whole Unit is defined. 2.0.0 Does the student understand the meaning of addition? Only identical units can be added to create a single numerical sum.

7. 7 / 27 Item Specifications Knowledge Specs Computational Fluency: Students can execute procedures in the domain without the need to create or derive the procedure. Fluid performance is based on recall of patterns or other well established procedures, and is fast, automatic, and error-free. How is something done? Conceptual Understanding: Captures demonstration of understanding of the mathematical concepts. Why is something done? When presented with… (Assessment Stimulus) Students should be able to… When presented with… (Assessment Stimulus) Students should be able to… 1.0.0 Does the student understand the importance of the unit whole or amount? 1.1.0. The size of a rational number is relative to how one Whole Unit is defined. Any rational number…Place it on a number line relative to the whole interval explicitly (0 and 1 labeled) or implicitly (0 and an integer other than 1 labeled) defined. Apparent contradictions involving rational number such as ¾ < ½ or ½ does not equal ½. Explain that the contradiction can be resolved if their relative wholes must be equal when comparing. 2.0.0 Does the student understand the meaning of addition? 2.2.0. Identical (common) units can be added to create a single numerical sum. Given a certain number of integers or fractions with the same denominator Determine the sum of those integers or fractions. Given a certain number of integers or fractions with the same denominator… Explain what common unit would be used to add (e.g. 3 + 5 would be three ones + 5 ones, or ¾ + ¼ would be three fourths + one fourth).

8. 8 / 27 Assessment Items (1.1.0 Proc)

9. 9 / 27 Assessment Items (1.1.0 Conc)

10. 10 / 27 Assessment Items (2.2.0 Proc)

11. 11 / 27 Assessment Items (2.2.0 Conc)

12. 12 / 27 Instruction – Video Game

13. Some Game Play

14. Results

15. 15 / 27 Technical Quality Accuracy (Valid Inference) Evidence about the content of the assessments. Evidence from other activities requiring the same construct. Consistency (Reliability) Inter-item reliability Item test-retest reliability Form test-retest reliability

16. 16 / 27 Technical Quality Analysis Samples of convenience of over 400 Students from Middle and High Schools in Southern and Central California

17. 17 / 27 Accuracy (Valid Inference) Items assessed the key ideas broadly, but within a defined conceptual area. Game integrated both Knowledge Specifications in most levels Scores on the pretest were significantly correlated with the level a student completed after playing the non-instructional version of the game for 30 minutes

18. 18 / 27 Accuracy (other influences) Self-reported math grade from previous year and self-reported math ability were significantly correlated with game level a student achieved. The amount of time a student played video games each week had the next greatest correlation with the game level a student achieved Gender was not a significant predictor of game level achieved after controlling for the above

19. 19 / 27 Consistency (Reliability) Test measures specific knowledge (Inter-item reliability is between.92 and.96) Performing well on an item is strongly correlated with scoring well on the overall test (Point-biserial mean is.52) Performance on the Pre-test items is strongly correlated with performance on the Post-test items (reliability ranges from.83 to.94) with no notable Test – Retest effects.

20. 20 / 27 Summary of Research Studies

21. 21 / 27 Studies Conducted Instruction and feedback Scoring rules and help seeking Collaboration Narrative 21

22. 22 / 27 Instruction and Feedback Does type of instruction and feedback matter in a math game? Variations: Type of instruction (minimal math, or math-focused), availability of feedback (yes, no), format of instruction Results: No effect of type of instruction, availability of feedback, or format Learning gains for lower-performing students Save Patch perceived as “gamey” 22

23. 23 / 27 Explanation of Scoring Rules and Help Seeking What is the effect of providing an explanation of scoring rules and an incentive to access feedback on learning? Variations: Explanation of scoring plus incentive, explanation of scoring, minimal scoring information Results: Providing the explanation of scoring plus incentive was most effective.  Higher normalized change scores (compared to minimal scoring information)  Higher posttest scores (adjusting for pretest scores)  Stronger effects for students with low academic motivation (effect sizes ranging from 0.7-1.2) 23

24. 24 / 27 Collaboration Does type of grouping affect learning and game play in a math game? Variation: Grouping of students (played game in pairs or as an individual) Results: Math outcomes  Low prior-knowledge students benefitted from grouping; high prior-knowledge students benefitted from individual play Game outcomes  Students working individually progressed farther in the game, and especially so for the high-prior knowledge students 24

25. 25 / 27 Narrative Is math learning and game engagement affected by: (a) presence of a narrative, and (b) gender matching with game character? Variations: type of narrative (masculine theme, feminine theme, no narrative) Results: More engagement with narrative Males + masculine theme  More positive response to game  Higher on math outcomes 25

26. Questions

27. 27 / 27 References Siegler, R., Carpenter, T., Fennell, F., Geary, D., Lewis, J., Okamoto, Y., et al. (2010). Developing effective fractions instruction for kindergarten through 8th grade: A practice guide (No. NCEE #2010-4039). Washington DC: U.S. Department of Education U.S. Department of Education. (2008). Foundations for Success: The final report of the National Mathematics Advisory Panel. Washington D.C.: US Department of Education

28. gdelacruz@cse.ucla.edu greg@ucla.eduvendlins@ucla.edu