1. Technology-Facilitated Formative Assessment in Physical Science Connected Classrooms: Case Studies Karen E. Irving Vehbi A. Sanalan Melissa L. Shirley The Ohio State University ASTE International Conference, St. Louis, MO January 10-13, 2008 The research reported here was supported by the Institute of Education Sciences, U.S. Department of Education, through Grant R305K050045 to The Ohio State University. The opinions expressed are those of the authors and do not represent views of the U.S. Department of Education.

2. 01/11/2008 2 Project team Principal Investigators –Douglas T. Owens, OSU –Stephen L. Pape, UF –Karen E. Irving, OSU –Louis Abrahamson, BEF –Frank Demana, OSU Post doctoral researcher –Vehbi Sanalan, OSU & Erzincan University, Turkey Consultants –Christy Kim Boscardin, UCLA –Joan Herman, UCLA –Jeremy Roschelle, SRI Project Coordinator –Mike Kositzke, OSU GRAs –Sedat Ucar; Sukru Kaya; Melissa Shirley; Clare Bell; Ugur Baslanti; Sharilyn Owens; Hyesook Shin; Gonul Sakiz

3. 01/11/2008 3 The potential of the connected classroom includes… Multiple interconnected representations Conceptual development through activity- based learning experiences Immediate anonymous feedback Public displays of class knowledge Technology-facilitated formative assessment Improved self-regulated learning

4. 01/11/2008 4 CCMS project overview Interdisciplinary professional development and research project Algebra I and Physical Science First year algebra data Classroom connectivity technology Summer Institute – training T 3 conference follow-up (2 years)

5. 01/11/2008 5 Purpose & research questions Purpose To report preliminary findings on science teacher experiences in connected classrooms Research Questions How do physical science teachers implement connected classroom technology in their teaching? Does connected classroom technology facilitate formative assessment?

6. 01/11/2008 6 Technology integration: Social theory & practicality Archer’s social theory –Connected classroom as a cultural change Morphostasis/ morphogenesis Doyle & Ponder’s practicality concept –Instrumentality Does the innovation fit the context of school? –Congruence Is the innovation compatible with teacher practice? –Costs/Benefits Is it worth the effort?

7. 01/11/2008 7 Pedagogy of connected classrooms: Formative assessment The process used by teachers and students to recognize and respond to students’ learning in order to enhance that learning, during the learning. (Bell& Cowie, 2001) The gathering of assessment data & teacher use of assessment information to modify their work to improve their teaching effectiveness. (Black, 1995; Black & Wiliam, 1998).

8. 01/11/2008 8 Formative assessment Formative assessment correlates with increased student achievement –Analyzed over 40 studies on formative assessment –All of the studies reported significant and often substantial achievement gains with positive effect sizes ranging from 0.4 to 0.7 (Black, Harrison, Lee, Marshall, & Wiliam, 2003)

9. 01/11/2008 9 Difficulties implementing formative assessment Studies show formative assessment use by classroom teachers is one of the weakest aspects of teacher practice. (Daws & Singh, 1996; Assessment Reform Group, 1999)

10. 01/11/2008 10 Participants First physical science cohort: 9 teachers Purposively selected 3 teachers for preliminary analysis Criteria for selection –Success in implementation –Diversity of classroom situations

11. 01/11/2008 11 School demographics Teacher*State# studentsSchool Diversity% low SES Ms. DTX 856 Grades 7-8 57% Hispanic 35% Black 5% White 3% Asian/Pac Islander <1% Native American 82% Ms. COH 662 Grades 6-8 51% White 36 % Black 11% Multiracial 2% Unspecified 72% Ms. SOH 897 Grades 9-12 98% White8% *pseudonyms www.Greatschools.net

12. 01/11/2008 12 Teacher preparation *Pseudonyms Teacher * UG degreeYear Graduated Graduate degree Years Teaching Ms. D Animal Science 2000None5 Ms. C Elementary Education 1987Master’s Curriculum & Instruction 18 Ms. S Secondary Education 2003None3

13. 01/11/2008 13 Students *pseudonyms Teacher*ClassStudent Characteristics Student gender Ms. D7 th grade science Regular classes M Ms. C8 th grade science Lowest achieving in the 8 th grade; many inclusion & special ed M & F Ms. S9 th grade physical science Regular classes M & F

14. 01/11/2008 14 Participants - Tech skills & support *pseudonyms Teacher*Tech SkillsTech supportClassroom technology Ms. D Experienced computer user T 3 mentor offsite Only teacher in school using Navigator Navigator SmartBoard Projector Ms. C Technology novice No calculator skills initially T 3 mentors in bldg Math teachers using Navigator Navigator SmartBoard Projector Ms. S Prior use of Probe ware; prior use of graphing calculator T 3 mentors in bldg Math teachers & 1 science teacher using Navigator Navigator SmartBoard Projector

15. 01/11/2008 15 Data sources Telephone interviews (audio & notes) –Autumn –Spring Classroom observations (video & notes) –2 days –2 class periods Post observation teacher interviews Student focus groups Surveys and measures

16. 01/11/2008 16 Coding – Practicality Instrumentality – context –Students –Technology support system –Social support Congruence –Successes/Challenges –Perceived ease of adoption Cost/Benefits –Why adopt? Why not? *(Doyle & Ponder, 1977)

17. 01/11/2008 17 Coding: Formative assessment typology Communicate task criteria –What has to be done –Quality criteria Gather data –Questions –Observes Interpret data –Critiques Change instructional practice –Suggestions for improved practice –Supplies information/ corrects/makes counter-suggestion (Torrance and Pryor, 2001)

18. 01/11/2008 18 Instrumentality - context Differences –Students –Schools –SES –Curriculum topics –Technology skills –Teacher preparation –Teaching experience Similarities –Mentoring –Social support –Hardware support –Risk takers –Persistence

19. 01/11/2008 19 Congruence – inquiry teaching Teachers found connected classroom technology congruent with their prior teaching practices –My curriculum is to use the FOSS* kits. [There are] a lot of hands on labs with those. So, basically, every time we do anything with data and graphing, we just automatically implement it through the Navigator™. (Ms. C 3050.TIAu06.P23) *Full Option Science System

20. 01/11/2008 20 Congruence - assessment I am on the data team for my district ….We really noticed …that kids are not analyzing the graphs….We had terrible data as an entire district. So our district goal … is every week … having kids graph and analyze lines in graphs….This is what my students are...doing through Navigator™. (Ms. C 3050.TIAu06.P29)

21. 01/11/2008 21 Congruence - assessment And [learning checks] are great because you can… put it by specific objectives and see where you are weak…and who is specifically weak in what area. That just narrows it down to each student by each objective. (Ms. D 3026.TIAu07.P54)

22. 01/11/2008 22 Costs I got a late start. And trying to use it every day is hard. And …equipment management is hard. So once I felt more comfortable, [then] the more I used it of course…. It’s not hard to adapt, but it takes a little time to adapt your lessons. (Ms. D 3026.TISp07.P52)

23. 01/11/2008 23 Costs It’s quite a bit of planning before to get set up and ready. And during class…you have to …change your mind a lot. Because even if you may have tested stuff …, there’s [going to] be issues that don’t work. And you may have to…switch gears right in the middle of your lesson ….So, it is [important] to be quick on your feet. (Ms. D 3026.TISp07.P75)

24. 01/11/2008 24 Benefits Connected classroom technology revealed new information about student learning. –Informal methods of ‘reading’ their classes to determine the success of their lessons provided inadequate data –Connected classroom technology gathered data from almost every student in the classroom, rather than just a few.

25. 01/11/2008 25 Benefits – knowing more about student learning It is just a great tool because you can narrow it down by topic and by student and you can really work with that student on what they are having trouble with and you can see if it is… one or two kids or if it is all of your kids. …It is great on…figuring out why they are missing something… or if it is… something you need to re-teach. (Ms. D 3026.TIAu07.P72)

26. 01/11/2008 26 Benefits – knowing more about student learning Kids, especially by this age level, the seventh or eighth grade, they have learned to play a game. They can know nothing and appear that they know a lot. I think it really brought to light for me that I did not have a good sense of what they are learning by looking at them. (Ms. C 3050.TISp07.P90) I have taught for 18 years and I have been in seventh grade science for about 15 of the 18, and there are things that I have always been really sure that I think kids have understood completely. Now I see what they are thinking, and I am like, whoa, I am just amazed. (Ms. C 3050.TISp07.P40)

27. 01/11/2008 27 They didn’t know they didn’t know. I am surprised. I will give a quiz which I think, okay, they should know this. This is pretty basic stuff. And it comes back and some kids on one question, half of them got it wrong. And I am like, whoa. (Ms. D 3026.TISp07.P67)

28. 01/11/2008 28 Gathering data – questioning in the CC We were doing earth and sun relationships… revolution versus rotation….We did a learn check and … I didn’t put it up on the screen. But I was able to look on [my screen] and see who was making those mistakes still…. So it helped me because I could pinpoint, without embarrassing them, … who was still mixing those concepts up. (Ms. C 3050.TIAu06.P47)

29. 01/11/2008 29 Data interpretation – student learning Connected classroom technology serves as a platform for science data collection and interpretation as well as pedagogical data collection and interpretation.

30. 01/11/2008 30 Data interpretation – science inquiry There was really strange data that showed up once we aggregated the class. …the data showed up, and [also] the child’s name. So I looked at the girl, and I said, “Do you know what you did? Look at this.” And she said, “Oh, I put the temperature in before the time.” And I said, “Yes, and what’s wrong with that?” And she said, “The x information goes first. And I did it backwards”… But from that she learned what she was doing wrong. I would have never caught that if we were just having a classroom discussion. (Ms. C3050.TIAu06.P33)

31. 01/11/2008 31 Rapid feedback – just in time It really opens my eyes and tells me that ok, they didn’t understand this. It really helps me quickly learn that I need to go back over a few things rather than move on. ‘Cause usually when you give a paper and pencil quiz, and you get it back, and you grade it, it’s like a day before you realize they didn’t understand this question at all. (Ms. D 3026.TISp07.P67)

32. 01/11/2008 32 Increasing student engagement If I send them a learn check or a quick poll, I immediately go to screen capture. And every time…I walk by my computer, I just tap the refresh. It gives me an idea where they are. And actually it works really well to keep them on task because they really buy into the idea that I know what they are doing all the time. (Ms.C 3050.TISp07.P45)

33. 01/11/2008 33 Increasing student engagement That is great because … every kid is involved… They like seeing that they are involved and it is confidential so they do not feel weird about jumping out and saying who they are if they are not sure about it. But …the kids just love it. (Ms. D 3026.TIAu07.P56)

34. 01/11/2008 34 Results - for these case studies… Connected classroom technology can be integrated in a variety of classroom contexts. Strong social support for innovative technology use provides an environment that supports connected classrooms. Teacher practice was congruent with CC practice. Teachers integrate connected classroom technology into both traditional and inquiry teaching practices.

35. 01/11/2008 35 Results – for these case studies… Connected classroom technology helps teachers know more about their students. –More students respond –Rapid feedback –Aggregated class data in a public or private forum –Discussion of right and wrong answers –Monitor on-task behavior

36. 01/11/2008 36 Next steps… Continued analysis of classroom video –Track formative assessment cycles Look for complete and incomplete FA cycles –Define task; gather data; evaluate; plan teaching –Evaluation without data gathering? –Evaluation with partial data gathering? Look for evidence of teacher interactive and preplanned FA Look for evidence of teacher informed decision making Year 2 data collection –Second cohort of science teachers

37. 01/11/2008 37 Paper & Contact Information Paper –5 paper copies of full article available today –Electronic version available at: www.ccms.osu.edu/ASTE2008/PSCaseStudy.pdf Contact Information –Karen Irving Irving.8@osu.eduIrving.8@osu.edu –Vehbi Sanalan Sanalan.1@osu.eduSanalan.1@osu.edu –Melissa Shirley Shirley.37@osu.eduShirley.37@osu.edu

38. Physical Science Connected Classrooms: Case Studies Karen E. Irving Vehbi A. Sanalan Melissa L. Shirley The Ohio State University ASTE International Conference, St. Louis, MO January 10-13, 2008

39. 01/11/2008 39 Rapid feedback – just in time From the student perspective, I feel like if they can get that instant feedback, instead of a couple of days later, when they don’t remember what they chose anyway, I feel like this way they really understand it a lot better and they can ask questions right away instead of that time lapse where they forget everything. (Ms. S 3064.POI07)

40. 01/11/2008 40 Benefits – knowing more about student learning It also allows me to see right away, what things they know and what things they are not catching on to. Like … that one learning check today, I could tell right away that they still cannot tell the difference between a distance graph and a position graph and their meaning. (Ms. C 3050.poi07.P29)