1. The Math Forum/Math Tools: Evaluation K. Ann Renninger Swarthmore College Evaluation of Math Tools, a DL of software tools for learning mathematics on all platforms (computer and handheld) Builds on evaluation methods used to evaluate The Math Forum (mathforum.org)

2. This evaluation process: Is a participatory and collaborative process Is a participatory and collaborative process (Renninger & Shumar, 2002, in press a, in press b) *Staff member *Site participants Has a dual focus on process and outcomes, undertaken in three phases (Renninger, 2003 http://eduimpact.comm.nsdl.org/evalworkshop/_renninger.php) Has a dual focus on process and outcomes, undertaken in three phases (Renninger, 2003 http://eduimpact.comm.nsdl.org/evalworkshop/_renninger.php)

3.  Includes three overarching questions: 1.Does The Math Forum provide meaningfully different types of tools, or methods for enhancing math education? 2.What kinds of facilitation, adjustments of existing resources, and/or mentoring relationships are needed in order to meet the strengths, needs, interests, and experiences of different users? 3.What is the impact of DL resources on forms of mathematical interaction and the development of mathematical thinking?

4.  Combines methods: Online questionnaires (Renninger & Shumar, 1998, 2002, in press b; Shechtman, Chung, Rosenquist, DiGiano, & Rochelle, 2003) Online questionnaires (Renninger & Shumar, 1998, 2002, in press b; Shechtman, Chung, Rosenquist, DiGiano, & Rochelle, 2003) In-depth structured interviews (Renninger & Shumar, 2002) In-depth structured interviews (Renninger & Shumar, 2002) Videotapes of classroom practice (Renninger & Shumar, 2003, in press a) Videotapes of classroom practice (Renninger & Shumar, 2003, in press a) Email discussions (Renninger & Shumar, 2003, in press a) Email discussions (Renninger & Shumar, 2003, in press a) Discourse analysis (Shumar, 2003 http://www.ericit.org/books.shtml#DevelopingDigital) Discourse analysis (Shumar, 2003 http://www.ericit.org/books.shtml#DevelopingDigital) Student work with problems (Renninger, Farra, & Feldman- Riordan, 2000 mathforum.org/articles/rennin2_2000.html) Student work with problems (Renninger, Farra, & Feldman- Riordan, 2000 mathforum.org/articles/rennin2_2000.html)

5. Employs purposeful sampling Employs purposeful sampling Is undertaken at three levels of analysis (see Renninger & Farra, 2003 http://www.ericit.org/books.shtml#DevelopingDigital) Is undertaken at three levels of analysis (see Renninger & Farra, 2003 http://www.ericit.org/books.shtml#DevelopingDigital) 1) Individual 2) Dyadic 3) Culture

6. Explaining the three levels of analysis… For example, from Renninger and Farra (2003): 1) individual engagement, or that which the mentor and the participant each brings to his or her work with the Dr. Math service (i.e. interest, risk- taking behavior, and resourcefulness), 2) interaction between the mentor and the participant (i.e. the abilities of the mentor and the participant to connect to the problem/task in question; generate and adjust use of strategies to meet problem/task demand; and amount of support required to accurately address the problem/task demands), and 3) the norms or cultural context of the mentor and of the participant (i.e., norms and positioning).

7. References: Renninger, K.A. & Shumar, W. (1998). Why and how students work with The Math Forum’s Problem(s) of the Week: Implications for design. Proceedings of ICLS 98 (pp. 348-350). Charlottesville, VA: AACE. Renninger, K.A., Farra, L., & Feldman-Riordan, C. (2000). The impact of The Math Forum’s Problems of the Week on students’ mathematical thinking. Proceedings of ICLS 2000. Mahwah, NJ: Erlbaum. (www.mathforum.org/articles/rennin2_2000.html) Renninger, K.A. & Shumar, W. (2002). Community building with and for teachers: The Math Forum as a resource for teacher professional development. In K.A. Renninger & W. Shumar (Eds.), Building virtual communities: Learning and change in cyberspace (pp. 60-95). New York, NY: Cambridge University Press. Renninger, K. A. & Farra, L. (2003). Mentor-participant exchange in the Ask Dr. Math service: Design and implementation considerations. In M. Mardis (Ed.), Digital Libraries as Complement to K-12 Teaching and Learning. ERIC Monograph Series. http://www.ericit.org/books.shtml#DevelopingDigital

8. Renninger, K. A. (2003). A three-phase evaluation strategy for evaluating educational impact. White paper, Developing an Educational Strategy for the Educational Impact of the National Science Digital Library. http://eduimpact.comm.nsdl.org/evalworkshop/_renninger.php Renninger, K. A. & Shumar, W. (April, 2003). The role of the social in teachers’ interest development and learning with an online learning community. Paper presented as part of the symposium: Sociocultural aspects of interest development and their implications for education (S. Nolen, Chair). American Educational Research Association. Chicago, IL. Renninger, K. A. & Shumar, W. (in press a). The centrality of culture and community to participant learning at and with The Math Forum. In S. Barab, R. Kling, & J. Gray (Eds.), Designing for virtual communities in the service of learning (pp. 181-209). New York, NY: Cambridge University Press. Renninger, K. A. & Shumar, W. (in press b). What do they want? Why do they come? : Participation at The Math Forum. In J. Weiss, J. Nolan, & P. Trifonas, International Handbook of Virtual Learning Environments. Amsterdam: Kluwer Press. Shechtman, N., Chung, M., Rosenquist, A., DiGiano, C. & Rochelle, J. (2003). Math Tools User Study Report. Unpublished document. Shumar, W. (2003). The role of community and belonging in online learning: The ESCOT Project at The Math Forum. In M. Mardis (Ed.), Digital Libraries as Complement to K-12 Teaching and Learning. ERIC Monograph Series. http://www.ericit.org/books.shtml#DevelopingDigital