1. Jose Carlo A. Soriano

2. Break down Effective Help-seeking Behavior Among Students Using an Intelligent Tutoring System for Math: A Cross-Cultural Comparison

3. Intelligent Tutoring Systems Seeks to simulate the effectiveness of a good personal human tutor Individual tutoring is more effective than classroom instruction by 2 standard deviations (Bloom, 1984) Knows what specific skills the student is having trouble with Able to offer appropriate help at an appropriate time Objective: help the student learn ITS are better by 1 standard deviation (Koedinger et al, 1998; Corbett et al., 2001)

4. Intelligent Tutoring Systems

5. Help-Seeking in ITS The kind of help, and how the help is offered, affects learning (Aleven et al, 2003) Students who use High-level help most frequently have the least learning(Matthews and Mitrovic, 2008) Help-seeking is a Meta-cognitive skill Meta-cognition is “cognition about cognition” One’s knowledge of the processes in play One’s active control of it during learning

6. Help-seeking Behavior 1. Both the EU and UNESCO declared: developing metacognitive skills, or ‘teaching students how to learn’ should be among the highest educational priorities (Louizidu and Kotselini, 2007) 2. Help-seeking behavior is an achievement-related behavior (Karabenick and Knapp, 1991) 3. Higher-achieving students were more likely to ask for help when encountering personal difficulties (Taplin et al, 2001)

7. Effective Help-Seeking Student is more likely to learn when: Student seeks for help when encountering personal difficulties Student knows what kind of help is needed such that student can work effectively on his/her own Student knows how to ask for help Student is not dependent on help Student spends time understanding help given

8. Help-seeking in ITS However, students generally do not know when they need help (Aleven and Koedinger, 2000) Students “game the system” (Baker et al) Meta-cognitive tutors have been developed by Aleven et al “Scooter the tutor” developed by Baker et al To teach students “how to learn”

9. The Problem Is “effective help-seeking” the same across cultures? Very few cross-cultural comparisons Comparing ITS use between USA and Latin American students(Ogan et al, in press) Comparing Disengaged behavior between USA and Filipino students (Rodrigo et al, 2010) Implications on Meta-cognitive tutors

10. Method Find out if effective help-seeking behavior is the transferrable across cultures, or are significantly different Might encourage more cross-cultural comparisons Implications on future efforts on meta-cognitive tutoring

11. Scatterplot tutor

12. Data Costa Rica Mexico USA Philippines

13. Feature Engineering 1. Helpavoidance 2. Nothelpavoidance 3. Helpnonuse 4. Unneededhelp 5. BugmsgLongpause 6. BugmsgShortpause 7. HintmsgLongpause 8. HintmsgShortpause

14. Feature Engineering 9. HintmsgLongpauseCorrect 10. HintmsgShortpauseCorrect 11. NothelpavoidanceShortpause 12. NothelpavoidanceLongpause 13. UnneededhelpShortpause 14. UnneededhelpLongpause 15. ShortpauseHintmsg 16. LongpauseHintmsg 17. FirstattemptHintmsg

15. Feature Optimization Most features require a threshold, either p-know or a time threshold Brute-force grid search: For p-know thresholds, grid-size is 0.05 For pause thresholds, grid-size is 0.5 seconds Single-parameter linear regression for each threshold for each feature in grid

16. Feature Selection Cross-validated r was used as the goodness criterion Correlation between the predicted learning values and the actual learning value The threshold with the best cross-validated r becomes the threshold for each feature As an additional control against over-fitting, features whose best threshold had negative cross-validated r is dropped from model creation

17. Model Creation and Evaluation Models were created using Forward Selection Models were evaluated by applying each country’s model to each country’s data set A model were created after combining the four data sets

18. Brute-Force Grid Search Featurecut-offrFeaturecut-offR CRPH Helpavoidance0.150.081Nothelpavoidance0.40.087 Helpnonuse0.150.012Helpnonuse0.950.043 Unneededhelp10.006NothelpavoidanceShortpause10.108 BugmsgLongpause25.50.06NothelpavoidanceLongpause00.075 HintmsgLongpause47.50.054UnneededhelpShortpause0.50.061 HintmsgShortpause0.50.017US HintmsgLongpauseCorrect41.50.294Helpavoidance0.250.122 NothelpavoidanceLongpause45.50.284Helpnonuse10.014 UnneededhelpShortpause130.019Unneededhelp10.117 UnneededhelpLongpause00.008BugmsgLongpause570.131 MXBugmsgShortpause0.50.026 Helpnonuse10.201HintmsgLongpause0.50.003 BugmsgShortpause2.50.044HintmsgShortpause6.50.02 HintmsgShortpauseCorrect0.50.025HintmsgLongpauseCorrect10.039 NothelpavoidanceLongpause58.50.018HintmsgShortpauseCorrect40.073 ALLNothelpavoidanceShortpause0.50.265 Helpavoidance0.050.023NothelpavoidanceLongpause00.096 Nothelpavoidance0.40.024UnneededhelpShortpause120.203 Helpnonuse00.094UnneededhelpLongpause00.123 BugmsgShortpause2.50.071ShortpauseHintmsg32.50.026 HintmsgLongpauseCorrect10.022LongpauseHintmsg37.50.04 HintmsgShortpauseCorrect30.062 NothelpavoidanceShortpause200.027 NothelpavoidanceLongpause10.052 UnneededhelpShortpause350.04 UnneededhelpLongpause10.062

19. Analysis FirstattemptHintmsg – only feature that was not able to pass in any country Different nature of skills Some can be hard at the very start, some can be easily understood from the start Most skills may require a large number of actions NothelpavoidanceLongpause – had positive cross-validated correlation in all five data sets In contrast to theory, this feature has negative directionality for most countries This might be because students who exhibit the behavior simply do not understand the skill

20. Forward Selection CountryLearning =Cross-validated r CR 0.132 * Helpavoidance(0.15) + 7.385 * HintmsgLongpause(47.5) - 9.096 * HintmsgLongpauseCorrect(41.5) - 21.847 * NothelpavoidanceLongpause(0.25, 45.5) + 53.010 0.462 MX - 0.147 * Helpnonuse(1) - 0.754 * BugmsgShortpause(2.5) + 1.187 * HintmsgShortpauseCorrect(0.5) + 40.652 0.229 PH 0.021 * Helpnonuse (0.95) - 0.763 * NothelpavoidanceShortpause(0.4, 1) + 32.423 0.126 US - 1.021 * Nothelpavoidance(0.25) - 2.870 * BugmsgLongpause(57) - 6.680 * NothelpavoidanceShortpause(0.25, 19.5) + 5.605 *LongpauseHintmsg(37.5) + 12.086 0.350 ALL 0.036 * Helpavoidance(0.05) + 0.082 * Helpnonuse(0) - 0.491 * BugmsgShortpause(2.5) - 46.861 0. 153

21. Analysis Helpavoidance – negative directionality for CR and PH reinforcces Aleven et al’s findings that avoiding help is negatively correlated to learning (Aleven et al., 2006) ButmsgShortpause and NothelpavoidanceShortpause also has negative directionality for MX and US, and PH and US. Possibly effect of not spending enough time to understand bug or hint message provided

22. Cross-Cultural Evaluation CountryCRMXPHUSALL CR0.534-0.2850.0510.1510.08 MX0.040.392-0.086-0.0090.088 PH0.004-0.1740.2030.1460.038 US-0.085-0.1640.2280.4760.057 All-0.0320.1650.0470.1420.215 Rows are models, columns are data sets applied to

23. Analysis Eighteen out of 25 model applications produced positive correlation between model’s predicted learning and the actual learning Negative correlation means model did worse than chance at predicting learning LOOCV r values are very high compared to r when applied to other countries Reinforces hypothesis that help-seeking might not transfer across countries

24. Analysis MX and US – performance on each other’s data sets are low (-0.009 and -0.164) This means that our model of effective help-seeking is not effective when applied to the other country Reinforces findings in (Ogan et al, in press) which compares differences in behavior of USA and Mexico students Collaborative nature of students from Mexico may be the reason why help-seeking is different The help they ask from the tutor will only be help that they do not get from other students

25. Analysis CR and MX – did not perform well on each other’s data sets (-0.285, 0.04) Though collaborative tendencies might be common between Costa Rica and Mexico, help-seeking behavior with the ITS may differ US and PH – performed very well on each other’s data set (0.146 and 0.228) Reinforces findings in (San Pedro, 2011) wherein a carelessness detector is generalizable between the two countries In contrast to (Rodrigo, 2010) which shows disengaged behavior is different between the two countries But Help-seeking and Disengaged behavior are two different sets of behaviors

26. Conclusion Results did not expose that effective help-seeking as a whole is very culture-specific (18 out of 25 applications returned positive r) However, it is not apparent that effective help-seeking is transferrable across countries Big difference between LOOCV r and cross-cultural evaluations

27. Conclusion and Recommendations There are pairs of countries wherein effective help-seeking fail to generalize to each other’s data set. Meaning effective help-seeking from one country may not necessarily be effective in another Single effective help-seeking models used by meta- cognitive tutors may be effective in one culture but not in others Future meta-cognitive tutors might have to use a more generalizable model May have to switch models, depending on the culture where the ITS is used