1. Effects of high level prompts and peer assessment on online learners’ reflection levels Presenter: Zong-Lin Tsai Advisor: Ming-Puu Chen Date: January 19, 2009 Chen, N. S., Wei, C. W., Wu, K. T. & Uden, L. (2009). Effects of high level prompts and peer assessment on online learners’ reflection levels. Computers & Education, 52(2), 283-291.

2. Introduction Reflection plays a very important role in learning processes and is very helpful for promoting learning performance (Chi, de Leeuw, Chiu, & Lavancher, 1994; Lee & Hutchison, 1998; McNamara, 2004; McNamara, O’Reilly, Best, & Ozuru, 2006). Knowledge is much easier to memorize and understand if it is being constructed after a learner has gone through the internal thinking and comprehension process. This is because learners can review, test, and modify their own ideas while engaging in reflective practice in a learning process (Wolfe & Goldman, 2005); and this is why reflection is very important to learning. Dewey considered reflection as a mental processing of one’s internal problem solving activity. Some researchers considered reflection is a kind of meta-cognitive ability to which a person can adequately apply different strategies, and this ability can be acquired by proper learning and training (Borkowski, Carr, Rellinger, & Pressley, 1990). The focus of this study is to explore how to better utilize online learning to cultivate learners’ reflection ability. 2

3. Introduction Research has shown that good learning performance, whether through humans or computers, can be achieved by providing prompts to learners (Aleven & Koedinger, 2002; Chi et al., 1994; Davis, 2000). These pre-designed prompts will appropriately display to learners while they are reading the materials. This should prompt learners to engage in reflective practice. We call this kind of prompt strategy ‘‘reflection prompt”. King (1994) classified questions into high level and low level questions. High level questions are comprehension and integration questions, and low level questions are merely memorization questions. High level questions are very helpful for learners to comprehend learning material. It is our belief that providing high level prompts is a key factor for promoting reflection. Humphreys, Greenan, and McIlveen (1997) have reported that getting learners involved in assessment would improve learning motivation. Peer assessment has positive effects both on learning attitude and learning performance (Sung, Chang, Chiou, & Hou, 2005; Topping, 1998; Tsai, Lin, & Yuan, 2002; Tseng & Tsai, 2007). In view of this, we believe that peer assessment is another factor we identified as promoting reflection. The aim of this study is to explore how high level prompts and peer assessment can affect a learner’s reflection levels in an online learning environment. 3

4. Method Variables – Both high level prompts and peer assessment are independent variables and reflection levels is dependent variable. – High level prompts King (1994) classified prompt questions into three different types: memorization questions, comprehension questions, and integration questions. It is generally accepted that high level prompts such as comprehensive and integration questions are more helpful to learners for constructing new knowledge compared to the memorized question prompts. Two settings for this independent variable are: with high level prompts and without high level prompts. We carefully selected a scientific article related to human hearing as an online reading material and adopted the definitions proposed by King (1994) that high level prompts are comprehension questions or integration questions. 4

5. Method Variables – Peer assessment Peer assessment means a group of learners with similar background and ability not only play the learner role, but also act as an instructor for assessing peers’ learning (Topping, 1998; Topping & Ehly, 2001). There are two types of peer assessment, peer observation and peer feedback. There are two forms of peer assessment, peer observation and peer feedback. We would like to investigate how peer observation and peer feedback can affect learners’ reflection levels. There are three manipulations for this independent variable: Without peer assessment (neither peer observation nor peer feedback) Peer observation without peer feedback Peer observation with peer feedback 5

6. Method Variables – Peer observation The concept of peer observation is the same as Observation Learning proposed by Bandura (1986) in his social learning theory. It means learning by observing others’ behaviors and performance. We classified the two types of reflection content observed by learners into high quality and low quality. By adopting the classification of Bain, Ballantyne, Packer, and Mills (1999) on reflection contents, we considered reasoning and reconstructing as high quality reflection contents and reporting and responding as low quality reflection contents. – Peer feedback As there is no empirical study on how positive and negative feedback affects reflection levels, we want to explore this issue in this study. We classified peer feedback into two types, positive and negative feedback. 6

7. Method Variables – Reflection levels Bain et al. (1999) classified reflection into five different levels, reporting, responding, relating, reasoning, and reconstructing, according to learners’ reflection contents. Because in a real learning practice, some learners’ reflection contents would be too simple, incomplete, or have even misunderstood the question, which cannot be classified to the lowest level of reporting, so we extended the criteria into eight reflection levels. 7

8. Design Participants The participants in the experiment were 157 on-campus college students from different major disciplines, aged between 21 and 26. Participants were randomly assigned into 14 different learning conditions as shown in Table 4. Table 4. Method 8

9. Material The learning material used in this experiment is a biology article related to the human ear. A question mark represents that learners will be guided to engage in reflection while they are reading the sentence, this can be seen in the left hand box of Fig. 2.Fig. 2. Procedure Method 9

10. Results Testing between high level prompts and peer observation on reflection levels –In the first stage, high level prompts and peer observation are independent variables and reflection levels is the dependent variable. –The mean of reflection levels was above five in the groups with-high- level-prompts. It showed that their reflection levels were generally higher than responding level. –However, the reflection levels were generally lower than responding level in the other groups. 10

11. Results Testing between high level prompts and peer observation on reflection levels – A two-way ANOVA analysis was conducted on the reflection levels for high level prompts and peer observation. There were significant main effects for high level prompts (F(1, 151) = 68.58, p-value = 0.000) and peer observation (F(2, 151) = 11.33, p-value = 0.000). Meanwhile, the interaction effect between them was also significant (F(2, 151) = 3.58, p-value = 0.030). – This result indicated that reflection levels will not be affected by just high level prompts or peer observation alone. It implies that learner’s reflection levels can be greatly improved by applying both high level prompts and peer observation teaching strategies. 11

12. Results Testing between high level prompts and peer observation on reflection levels – A post-hoc analysis using Fisher’s least significant difference (LSD) test. – In the without-high-level-prompts condition, the reflection levels of the group with high quality observation was significantly higher than the groups without peer observation and with low quality observation. – In the with-high-level-prompts condition, there were no significant differences among their reflection levels in the groups. – Learners who were provided with high level prompts as well as high quality observation, their reflection levels were significantly higher than the groups without high level prompts. 12

13. Results Testing between peer observation and peer feedback on reflection levels –In the second stage, peer observation and peer feedback are independent variables and reflection levels is the dependent variable. – There was a significant main effect for peer observation (F(1, 113) = 14.78, p-value = 0.000). However, no significant main effect for peer feedback (F(2, 113) = 1.43, p-value = 0.244) was found. The interaction effect between them was also not significant (F(2, 113) = 2.53, p-value = 0.084). – The results showed that peer observation has a statistically significant influence on reflection levels and peer feedback has no significant influence on reflection levels. 13

14. Discussion There are several important implications from this study. – Firstly, for with-high-level-prompts groups, their reflection levels are significantly higher than the without-high-level-prompts and without- high-quality-observation groups as shown in Fig. 4. This implies that learners often do not know how to carry out good reflection if high level prompts were not provided. 14

15. Discussion – Secondly, for the with-high-quality-observation groups, we found that their reflection levels are significantly higher than the without high- level-prompts and without-high-quality-observation groups as shown in Fig. 5. This implies that instructors could provide high quality reflection contents to use as examples in an online learning environment. 15

16. Discussion – Thirdly, giving learners’ positive feedback, negative feedback, or no feedback did not significantly affect learners’ reflection levels. This implies instructors can expose learners to peer feedback and not worry about the effects on learners’ reflection levels. Although positive feedback or negative feedback has no significant effect on learners’ reflection levels, but peer feedback could give learners an opportunity to play the role as an instructor: from the peer learning perspective, this is good for learners to improve their own critical thinking abilities by giving comments to their peers. – Finally, instructors can integrate e-portfolio to realize reflection education. Instructors can use e-portfolio as a reflection venue for learners and consider the inclusion of reflection portfolios as one of the assessment items. In addition, how to create a good atmosphere such that learners will be more willing to express their own opinions is very important. 16

17. 17

18. 18