1. The Role of Optimal Challenge in Adaptive E-learning: Evidence from Field Experiments with Middle School Students
De Liu
Information and Decision Sciences Carlson School of Management University of Minnesota
Joint work with Tao Li, Sean Xin Xu | Oct 17, DAE, Knoxville, TN
每个作业的平均时间。

2. E-Learning provides many opportunities for online experimentation
Rise of online and mobile learning has redefined learning
These platforms are incredible fertile playgrounds for design and experimentation.
Big data (volume + granularity), real-time personalized intervention
Applications of ML/AI tools, new challenge issues (disengagement, self-selection, fairness)
Enlarging disparity

3. A research stream aimed to help middle school students learn outside of the classrooms using online learning platforms.
Collaborate with a few middle school in Beijing
Currently focus on learning English during summer/winter breaks
Online learning platform + machine learning + learning/engagement theories
Problem Sequencing
Interleaving vs blocking designs
Adaptive challenge design
Digital Learning Divide
Self-selection & polarization
Re-engaging underachievers

4. How to choose a problem with the right challenge level for learners?
Large question banks exist for online learning
Within each topic domain, we can choose from many practice problems with different challenge levels
The question is how to choose?
Challenge Level
Challenge consistency
Strong learner
With the explosive growth of investments in e-learning (McCue 2014), ample open educational resources have been built and contributed by different parts of shareholders, such as e-learning companies, schools, government education agencies, after-school organizations and so on (Porcello and Hsi 2013). The popularity of MOOCs also accelerates this trend (Reich and Ruipérez-Valiente 2019).
E-learning eliminates the boundaries of meritorious learning resources, which are now accessible by learners throughout the world (Schneps et al. 2010). Rather than lacking enough learning materials, e-learners face the information overload since massive good-quality materials are available now, which could make learners get overwhelmed.
Therefore, how to effectively use these rich online learning resources raises the attention by academics (Schneps et al. 2010) and practitioners.
Porcello, D., and Hsi, S "Crowdsourcing and Curating Online Education Resources," Science (341:6143), pp
Reich, J., and Ruipérez-Valiente, J. A "The Mooc Pivot," Science (363:6423), pp
Schneps, M. H., Griswold, A., Finkelstein, N., McLeod, M., and Schrag, D. P "Using Video to Build Learning Contexts Online," Science (328:5982), pp
Weak learner

5. Agenda Introduction & motivation Related literature
Research hypotheses
Experiment design
Results
Discussion

6. Relationship with Personalized Learning
Personalized learning is pivotal for e-learning designs
Personalized learning
Topic Gap Detection
Adaptive Challenge Design
In the provision of personalized recommendations, personalized learning can comply with learner’ heterogeneities toward their learning abilities and close their knowledge gap.
By accommodating the feedbacks with their individual differences, personalized learning is broadly regarded as the pivot of e-learning design (Shute and Towle 2003) and has been adopted by leading e-learning platforms, such as Knewton, edX, Coursera, etc. (Bauman and Tuzhilin 2018).
Previous personalized learning studies focus on the knowledge level, such as misconception detection or further knowledge point recommendation (Bauman and Tuzhilin 2018). No doubt, that demonstrating the knowledge gaps and related prerequisites are of vital importance for personalized feedback. However, the gap detection is not the whole story but the first step. Learning is fundamentally a recursive process and requires sequential exercises to serve as the remedial materials to heal their deficits gradually. In personalized feedbacks, the corresponding exercises related to the knowledge gaps are also essential to help e-learners dynamically and adaptively make up their deficits.
Knowledge gaps are at the knowledge level, but exercises are at the problem level. Due to the long-term accumulation and richer digital resources available, the problem banks increase rapidly and thousands of problems can serve as the candidates for the target knowledge point. Mapping from the knowledge level to the problem level, we would find a great number of problems related to the corresponding knowledge. These problems, although related to the same knowledge point, varies a lot. Some of them may be straightforward and easy to solve, while others may be more complicated and require more efforts to master.
Therefore, how to select suitable problems to form the exercises is the indispensable procedure after the gap detection. In order to realize effective personalized learning, we have to consider the problem choice strategy to form the individualized exercises, identify how best to address a particular misconception, and optimize a specific learning sequence (Reich 2015).
Bauman, K., and Tuzhilin, A "Recommending Remedial Learning Materials to Students by Filling Their Knowledge Gaps," MIS Quarterly (42:1), pp
Reich, J "Rebooting Mooc Research," Science (347:6217), pp
Indispensable but neglected

7. Relationship with Adaptive Learning
Adaptively accommodate individual’s heterogeneities and learning progress (Schneps et al. 2010)
Macro adaptive: adapt to different learning goals
Self-select learning goal
Aptitude adaptive: Adapt to learner types
e.g. different learning plans for different learning styles
Micro adaptive: adapt to learning dynamics
e.g., adapt learning based on learner progress
Learning does not follow a linear progression and learners could not master the knowledge in one stroke. Schneps et al. (2010) criticize the traditional linear instruction approach as a “conceit implicit”, which instructs learners from one prerequisite to the next, in a linear and hierarchical manner. Instead, learning is a recursive process, which needs iterating exercises. Learners usually build their knowledge from conflicting ideas (Schneps et al. 2010) and update their beliefs from the ascending spiral rather than the linear progression. Knowledge is accreted, turned and restructured during the learning process (Butler and Winne 1995). Thus, the learning process of nonlinear reasoning requires e-learning to provide personalized materials to accommodate individual differences in talents, strengths and learning progress.
Adaptive learning is a broad concept, which can be categorized into three layers: macro-adaptive, aptitude-treatment interaction, and micro-adaptive approach (Park and Lee 2003). Macro-adaptive is to provide learners with multiple options and allow them to self-select their learning goals, depth of curriculum content and delivery way (Mödritscher et al. 2004; Park and Lee 2003). Meanwhile, aptitude-treatment interaction approach considers learner’s heterogeneities in intellectual ability, prior knowledge level (Park and Lee 2003), affective aptitudes and cognitive aptitudes (Kalyuga 2006). It tries to fit the learning materials with learner’s invariant characteristics, like self-efficacy, achievement motivation and learning styles (Akbulut and Cardak 2012; Truong 2016), which occupy the majority of adaptive learning studies (81.4%) according to a meta-analyses (Akbulut and Cardak 2012). Micro-adaptive approach diagnoses the learner’s specific learning needs and delivers personalized contents in match with their real-time progress, which is comparable to one-to-one tutoring.
This research

8. Prior Research on Adaptive Challenge
Maintain a medium challenge level
Endler et al. (2012) adjust problem difficulty to maintain a 50% success rate in the last two tasks
Sampayo-Vargas et al. (2013) adjust the challenge to stay within a (unspecified) success rate band.
Item Response Theory (Wauters et al. 2010)
Item challenge match learner’s ability
Wauters, K., Desmet, P., and Van Den Noortgate, W "Adaptive Item‐Based Learning Environments Based on the Item Response Theory: Possibilities and Challenges," Journal of Computer Assisted Learning (26:6), pp
Sampayo-Vargas, S., Cope, C. J., He, Z., and Byrne, G. J "The Effectiveness of Adaptive Difficulty Adjustments on Students' Motivation and Learning in an Educational Computer Game," Computers & Education (69), pp

9. Some gaps in existing online learning research
Theoretical ambiguity
What is the optimal level of challenge for learners with different aptitudes?
Is it because the level of challenge is good or simply that the challenge level is consistent?
Field experiment!
Implementation sophistication
Existing research is very underdeveloped in real-time estimation of learner's mastery levels in specific areas and they evolve
Machine learning can help!
The mechanism of effectiveness of adaptive challenge is still unclear. Its effectiveness could be interpreted as the suitable challenge level design since the previous studies focus on the item response theory, which emphasizes the match between difficulty level and prior knowledge. In the flip side, its effectiveness could also be interpreted as the challenge consistency over time, since limited studies keep the correctness within the scope of certain thresholds.
Besides, previous studies consider the challenge level of feedbacks in a homogenous and static view.
The learner’s heterogeneity, such as past performance, learning behaviors and motivations does not take into consideration by this naïve method, which could contradict with the reality since the problem could be easier for those with well-established learning backgrounds. Besides, topic heterogeneity is also ignored in the traditional method since learner’s ability is treated as a whole parameter. A learner’s ability depends on topic heterogeneity where a learner may be familiar with some topics while unskilled in others.
In addition, the problem challenge could be time-contingent since the problem might be harder in the beginning and could be much easier once the learner master the knowledge point related to this problem.

10. Agenda Introduction & motivation Related Literature
Research hypotheses
Experiment design
Results
Discussion

11. Optimal Challenge Level - Flow theory
Match between perceived challenge and perceived ability (Csikszentmihalyi 1975)
Optimal challenge level is different for different individuals
Low ability  Low challenge (flow)
High ability  High challenge (flow)
H1: For weak learners: Low challenge > High challenge (in learning performance)
H2: For strong learners, High challenge > Low challenge (in learning performance)
Csikszentmihalyi, M., and Csikszentmihalyi, I Beyond Boredom and Anxiety. Jossey-Bass San Francisco.
Csikszentmihalyi, M Finding Flow: The Psychology of Engagement with Everyday Life. Basic Books.

12. Sensitivity to Challenge Levels
Differences between flow and boredom are not found in some scenarios.
Replace the boredom to relaxation (Csikszentmihalyi 1997; Engeser and Rheinberg 2008)
H3: Learning performance disparity between low and high challenges is smaller for strong learners than for weak learners.
Csikszentmihalyi, M., and Csikszentmihalyi, I Beyond Boredom and Anxiety. Jossey-Bass San Francisco.
Csikszentmihalyi, M Finding Flow: The Psychology of Engagement with Everyday Life. Basic Books.
Csikszentmihalyi (1997)

13. Does Challenge Consistency Matter?
H4: Challenge consistency  Better performance
Fluctuations in challenges  Unexpected impediment  reassessment or abandon (Butler and Winne 1995) + Negative attribution framing (Weiner 1985)
Challenge Level
(mean)
Challenge consistency
(variance)
Fluctuating challenge  Unexpected impediment, or discrepancy between expectation and reality  reassessment of learning efforts and abandon
Wauters, K., Desmet, P., and Van Den Noortgate, W "Adaptive Item‐Based Learning Environments Based on the Item Response Theory: Possibilities and Challenges," Journal of Computer Assisted Learning (26:6), pp
Sampayo-Vargas, S., Cope, C. J., He, Z., and Byrne, G. J "The Effectiveness of Adaptive Difficulty Adjustments on Students' Motivation and Learning in an Educational Computer Game," Computers & Education (69), pp

14. Agenda Introduction & motivation Related literature
Research hypotheses
Experiment design
Results
Discussion

15. Research Context
756 students (7-th grade) from 6 middle schools in Beijing
English reading comprehension
Winter break reading assignments (Jan. ~ Feb. 2019)
7 English articles per week, each followed by 3-5 questions.
Students are expected to complete them, but no penalty if they do not (other than scolded by teachers)
13,076 completed student-article assignments, 86.7% completion rates

16. System User Interface

17. Experiment Design Dependent variable Conditions Learner aptitude:
Random Challenge
Experiment Design
Inconsistent
Low Challenge
High Challenge
Consistent
Dependent variable
Posttest accuracy (correctness): Number of correct answers / number of questions.
Conditions
Control: randomly choose problems from a pool of candidates
High Challenge: problems ranked among 20% lowest predicted accuracy for this learner at the present time
Low Challenge: problems ranked among 20% highest predicted accuracy for this learner at the present time
Learner aptitude:
Strong learner: pre-test final exam score above the median score in one's class
Weak learner: pre-test final exam score below the median score in one's class

18. Experimental Procedure
Between subject: The assigned condition is fixed during the study period.
First, we will personalize the exercise design at the knowledge point level. The system detects learner’s knowledge gaps every day and then decides the specific knowledge gaps to review
After determining the topic to be review, we will mapping the topic to all suitable related problems to form the problem candidate list.
After finding the problem candidates, we will determine the personalized exercise at the problem level according to the challenge level,
The system would predict the correctness of each problem in the candidate list for the focal learner at this time, and sort all the problems according to the predicted correctness.
To exclude the extreme outliers, system chooses the problem candidates ranked at 20% lowest correctness as the high challenge group, while the 20% highest correctness as the low challenge group. As for the control group, the problems are randomly selected from the candidate list.
According to the randomly assigned group, system will provide corresponding problems as the exercises for next time learning materials.
4 weeks

19. Accuracy Prediction
Our assignment is based on dynamic, personalized accuracy prediction
Predict accuracy for student i on English article j at day t.
Heterogeneous ensemble with 5-fold cross-validation
SVM + BP Neural Network + Ridge Regression + Ordered Logistic Regression
Predictive performance (ex post)
Accuracy (% of questions answered correctly)：R-square = 58.6%
Number of correct answers per article: 66.5% accurate.
97% within 1 deviation from the actual # of correct answers.
1. I want to predict the continuous variable: correctness. We choose the standard artificial neural network (BP network) to train the model.
2. To increase the model generalizability, we want to exclude the unimportant factors, so we choose the Ridge regression to the variable selection by punishing the unimportant factors.
3. Considering the challenge is not equally increased with correctness. For instance, increasing the correctness from 80% to 90% is much more challengeable compared with increasing from 10% to 20%. Considering the nonlinear relationship between challenge and correctness, we transfer from the correctness prediction to the number of correct answers classification. We choose the SVM and ordered logistic regression to represent this classification.
Then we use a heterogamous ensemble model to combine four model together to increase the prediction accuracy.

20. Features used for accuracy prediction

21. Knowledge gap detection
Use to select a subset of candidate problems targeted at a certain knowledge (topic) gap.
Hidden Markov Model + manually coded problem-to-topic mapping
Portray the evolvement of topic proficiency
Higher proficiency  more likely to correctly answer the problems related to the topic.

22. Randomization check
No significant difference in the pre-test final exam scores
拆分成3波：25%

23. Agenda Introduction & motivation Related literature
Research hypotheses
Experiment design
Results
Discussion

24. Posttest Learning Performance Comparisons
Random
High
Low
H1 (√)：Weak learner, low challenge > high challenge
H2 (X)：Strong learner, high challenge > low challenge
H3 (√)：Disparity between high and low challenge is lower for strong learners
H4 (√)：Consistent challenge  better performance*
Random
High
Low

25. Low > High challenge
Regression results
H1 is supported
H1: For weak learners,
Low > High challenge
Consistency  high score

26. H3: For Strong learner, disparity is smaller
Regression results
H3 is supported
H3: For Strong learner, disparity is smaller
Consistency  high score

27. Regression results H2 is not supported (sum of the two coefficients)
H2: For strong learners,
High > Low challenge
Consistency  high score

28. Challenge consistency  Better performance
Regression results
H4 is supported
Consistency  high score
Challenge consistency  Better performance

29. Agenda Introduction & motivation Related literature
Research hypotheses
Experiment design
Results
Discussion

30. Challenges in Design and Analysis
Treatment based on predicted accuracy is inherently imprecise
We removed the cases that are too "wrong"
Predicted accuracy > 75% for High challenge
Predicted accuracy < 65% for low challenge
How to analyze such a data set based on imprecise treatments?

31. Challenges in Design and Analysis (2)
The experiment design isn't ideal for teasing apart level vs. consistency effects
The control group differs in both consistency and level!
Can we construct synthetic datasets ex post to address this design flaw?

32. Challenges in Design and Analysis (3)
Self-selection issue exists (and may cause biases)
In another repetition of the experiment (where there is less teacher intervention), only 38% of students completed their assignments
In this one, 86.7% completed their assignments.

33. Thank You! De Liu deliu@umn.edu

34. Forecast Accuracy Correctness：R-square 58.6%
Number of correct answered problems in each exercise
Accuracy: 66.5% ; 97% within 1 problem forecast error
First, we will personalize the exercise design at the knowledge point level. The system detects learner’s knowledge gaps every day and then decides the specific knowledge gaps to review
After determining the topic to be review, we will mapping the topic to all suitable related problems to form the problem candidate list.
After finding the problem candidates, we will determine the personalized exercise at the problem level according to the challenge level,
The system would predict the correctness of each problem in the candidate list for the focal learner at this time, and sort all the problems according to the predicted correctness.
To exclude the extreme outliers, system chooses the problem candidates ranked at 20% lowest correctness as the high challenge group, while the 20% highest correctness as the low challenge group. As for the control group, the problems are randomly selected from the candidate list.
According to the randomly assigned group, system will provide corresponding problems as the exercises for next time learning materials.

35. Consistent result after the following manipulation adjustment
Conflicts: Predicted correctness > 0.75 for High challenge ； Predicted correctness < 0.65 for low challenge
Delete # of conflicts more than 1 times.
Delete 56 individuals including 27 in high challenge and 29 in low challenge

36. Manipulation Check
First, we will personalize the exercise design at the knowledge point level. The system detects learner’s knowledge gaps every day and then decides the specific knowledge gaps to review
After determining the topic to be review, we will mapping the topic to all suitable related problems to form the problem candidate list.
After finding the problem candidates, we will determine the personalized exercise at the problem level according to the challenge level,
The system would predict the correctness of each problem in the candidate list for the focal learner at this time, and sort all the problems according to the predicted correctness.
To exclude the extreme outliers, system chooses the problem candidates ranked at 20% lowest correctness as the high challenge group, while the 20% highest correctness as the low challenge group. As for the control group, the problems are randomly selected from the candidate list.
According to the randomly assigned group, system will provide corresponding problems as the exercises for next time learning materials.

37. Robustness Check Different definitions of strong/weak learners
Main analysis we use top-50% as strong learners, here we try different %.