1. Colin Milligan, Allison Littlejohn & Nina Hood Learning in MOOCs: A Comparison Study Research Track: Room Nr. HS 01.14 Date: Monday 22 nd February 2016, Time: 14:30

2. Outline MOOCs and Self-Regulated Learning, Study contexts, participants and method, Findings: Individual studies, emerging themes, Reflection on implications, limitations and future work.

3. Introduction and Background FORETHOUGHTPERFORMANCE SELF- REFLECTION

4. Massive Open Online Courses FORETHOUGHTPERFORMANCE SELF- REFLECTION Massively popular, Content-centric, pedagogically simplistic. Our interest: how are they being used by professionals: Formalising and updating knowledge, Preparing for career move, Networking and access to other professionals.

5. FORETHOUGHTPERFORMANCE SELF- REFLECTION Self-Regulated Learning Self-regulation is the ‘self-generated thoughts, feelings and actions that are planned and cyclically adapted to the attainment of personal goals’ - Zimmerman, 2000. FORETHOUGHTPERFORMANCE SELF- REFLECTION

6. FORETHOUGHTPERFORMANCE SELF- REFLECTION Phases and Sub-Processes of SRL Self-Regulated Learning (adapted from Zimmerman, 2000) PhaseForethoughtPerformanceSelf-reflection Sub-processes Goal setting Self-efficacy Task interest/value Learning and Task strategies Help seeking Interest enhancement Self-evaluation Self-satisfaction/affect SRL is highly context dependent – an individual may be unable to (or may choose not to self-regulate in some contexts.

7. FORETHOUGHTPERFORMANCE SELF- REFLECTION SRL & Massive Open Online Courses Four types of Motivation/underlying goals in MOOCS: fulfilling current needs, preparing for the future, satisfying curiosity, connecting with people. Zheng, Rosson, Shih & Carroll, 2015 Higher levels of self-efficacy lead to greater persistence. Poellhuber, Roy, Bouchoucha & Anderson, 2014, Wang & Baker, 2015 Learners who set goals are more likely to persist. Haug, Wodzicki, Cress & Moskaliuk, 2014 Forum participation linked to course completion. Gillani & Eynon, 2014

8. FORETHOUGHTPERFORMANCE SELF- REFLECTION What, Why & How: Qualitative Data While “… quantitative studies of learner activity within MOOC platforms provide us with a greater understanding of what learners do within MOOCs, our understanding of why MOOC participants learn as they do and how they actually learn is less well developed.” Veletsianos, Collier & Schneider, 2015

9. Research Questions and Study Design

10. Research Questions RQ1 How are MOOCs currently designed to support self-regulated learning? RQ2 What self-regulated learning strategies do professionals apply in a MOOC? RQ3 How can MOOCs be designed to encourage professionals to self-regulate their learning? Littlejohn, A, & Milligan, C. (2015). Designing MOOCs for professional learners: tools and patterns to encourage self-regulated learning. eLearning Papers, 42, 38-45.

11. Contexts & Cohorts Fundamentals of Clinical Trials (edX, 2013-4). Participants were professionals across a range of roles – medicine, healthcare, statistics, bio- scientists, pharmacists. 35 interviewees [16m, 19f], 23 countries. Drawn from 350 survey respondents. Introduction to Data Science (Coursera, 2014). Participants were software professionals across a range of roles – software engineers, data analysts, scientists. 32 interviewees [27m, 5f], 16 countries. Drawn from 768 survey respondents.

12. Method & Analysis Administer quantitative instrument: to gain a measure of the extent to which they are self-regulating in this context, Recruit volunteers for interviews, Undertake interviews, Code interviews, Sort by SRL score and sub-process score (high/low), Look for patterns of SRL: and how it differs between the two groups.

13. Instrument: SRL Questionnaire A measure of SRL for each respondent. Items were tailored to encourage participants to reflect specifically on their learning practices in the MOOC, Adapted from SRL in non-formal contexts instrument, previously validated (Fontana et al, 2015), This in turn was constructed from existing instruments: MSLQ (Pintrich et al, 1991); MAI (Schraw & Dennison, 1994); OSLQ (Barnard-Brak et al, 2010); LS (Warr & Downing, 2000); OS (Rigotti, Schyns & Mohr, 2008). Instrument available from figshare: http://figshare.com/articles/SRLMQ/866774 http://figshare.com/articles/SRLMQ/866774

14. SRL Profiles

16. Instrument: Semi-Structured Interview Explored various aspects of MOOC learning, structured around SRL sub-processes including self-efficacy, goal- setting and learning and task strategies, as well as patterns of help-seeking, self-reflection etc. Available from figshare (IDS version): https://figshare.com/articles/Interview_Script_SRL_in_MOOCs_IDS_/1300050

17. What are we looking for? What evidence of self-regulation do we see? Taking the two studies together: what similarities and differences do we recognise? how do these relate to topic/course format?

18. Findings

19. Differences: Course Topic & Format IDS Aimed at programmers, who wanted to gain skill in an emerging domain. Included projects – opportunities for in depth FCT Interdisciplinary – focused on diverse aspects of clinical trials: ethics, statistics, Adopted a very rigid structure Don’t underestimate the brand: Harvard

20. Sub-processHigh groupLow group Goal setting detailed, learning/mastery goals set, emotionally invested, and focused on role or career, some also mention certification. goals, if set at all, were typically focused on completion and certification. Self-efficacy clear and detailed descriptions demonstrating individual responsibility. less detailed descriptions, almost half indicated low self-efficacy. Learning and Task strategies note taking standard, active engagement, most did not change approach (did not feel the need to change), minority made active decision to change based on time pressures. only a minority took notes, more passive in approach, almost half changed approach as original approach had been ineffective, remainder had faced challenges but not changed: citing time pressures as a barrier (as opposed to a driver for change). Fundamentals of Clinical Trials Milligan & Littlejohn, under review

21. Sub-processHigh groupLow group Goal setting goals reference professional roles and future needs, improving skillset, gaining content knowledge, only a minority focused on completion. goals more abstract, focused on love of learning or extrinsically motivated, focused on completion/certification. Self-efficacy confidence in ability arising from specific factors: - previously familiar with content knowledge, - previous participation in a MOOC. some participants lacked ability or confidence to evaluate own learning (ss). Learning and Task strategies wide variety of strategies used, flexible in approach, adaptive. linear approach adopted, highly scheduled, inflexible. expressed disappointment in performance but didn’t change approach (ss). Introduction to Data Science Littlejohn, Hood, Milligan & Mustain, 2016

22. Similarities: Goal-setting Greater specificity of goals among high SRL group: “The main aim is to become a better data analyst and get my introduction and get the concept I need for data science, especially data science that involves around building MapReduce programmes and Python programmes.” (IDS 673) Intrinsic over extrinsic motivation among high SRL group. “I would like to have finished the class, to get the certificate, but it wasn't really for that. I think it’s more personal, like a personal goal, like I just wanted to learn from the best. So it’s great that you have a certificate, but I’m not about the piece of paper, I’m about the learning opportunity.” (FCT334)

23. Similarities: Self-Efficacy Good self-efficacy across the board, Lack of confidence in a subset who had not studied online before or who lacked background knowledge: “I’m very familiar with the course, I already have a good background, I have all the resources and knowledge about this issue” (FCT 152) “I found myself lost, this is due to the background, maybe that I was deficient” … “I always start searching on an internet engine, but it needs some sort of assistance.” (FCT 316)

24. Differences: Goal-setting The importance of the Harvard brand: “The goal is to have an in depth knowledge of this area from a very prestigious university like Harvard and having it certified with a certificate.” (FCT 152)

25. Differences: Learning and Task Strategies Note taking a standard approach for FCT while not universal among IDS learners, High SRL FCT learners kept to the rigid structure of the course, following it: “They recommend readings /books where you can dig deeper into the subject … I could have gone more in depth, although I chose not to because I didn’t think I needed more in depth just now” (FCT 143). High SRL IDS learner were more flexible in their approach: “I think my knowledge and my background and my work experience was very, very helpful, because whenever I saw something I understood I just ditched it and went to another part of the course.” (IDS 239)

26. Conclusions and Reflection

27. Conclusions Individuals self-regulate their learning to varying degrees, Other factors affect learning - motivation, prior experience etc. Course topic and format can shape expectations and learning approach adopted, Prestige is also a factor.

28. Implications for Practice Encourage learners to set goals, Get them to think about what they want Match content to learner expectations, Can one course meet diverse expectations? Ensure learners are confident to learn in your MOOC environment.

29. Reflection: Limitations Small samples: inherent in qualitative research. Broad variability of key factors within sample: e.g. motivation, experience of online learning. Limited range of SRL ability: all were self-regulating their learning to a significant degree. Lack of external measure of success: difficult to link learning and performance (time/access to data).

30. Reflection: Future Work Study further MOOC contexts: to see to what extent observations are generalisable. Link with completion and other quantitative data: to strengthen evidence and understand the impact of different learning strategies. Perform longitudinal studies: to see the impact of MOOC learning on practice. Use the SRL profiles as feedback for learners: to encourage reflection on strengths, weaknesses.

31. Thank you Dr Colin Milligan Caledonian Academy GLASGOW CALEDONIAN UNIVERSITY Glasgow, SCOTLAND colin.milligan@gcu.ac.uk @cdmilligan Slides (with notes) available from: https://figshare.com/authors/Colin_Milligan/100462 This work was originally funded by the Bill & Melinda Gates Foundation. Thanks to Obiageli Ukadike, Nabeel Gillani and Bill Howe for access and assistance, and Lou McGill for conducting interviews. Prof Allison Littlejohn Institute of Educational Technology OPEN UNIVERSITY Milton Keynes, ENGLAND allison.littlejohn@open.ac.uk @allisonl Dr Nina Hood Faculty of Education UNIVERSITY OF AUCKLAND Auckland, NEW ZEALAND n.hood@auckland.ac.nz

32. Extras

33. References Bandura, A. (1986). Social foundations of thought and action: A social cognitive theory. Englewood Cliffs, NJ: Prentice Hall. Bernacki, M. L., Aguilar, A. & Byrnes, J. (2011). Self-regulated learning and technology-enhanced learning environments: An opportunity propensity analysis. In G. Dettori and D. Persico (Eds.), Fostering self-regulated learning through ICT (pp. 1-26). Hershey, PA: IGI Global Publishers. Breslow, L., Pritchard, D. E., DeBoer, J., Stump, G. S., Ho, A. D. & Seaton, D. T. (2013). Studying learning in the worldwide classroom: Research into edX’s first MOOC. Journal of Research & Practice in Assessment, 8, 13–25. Fontana, R.P., Milligan, C., Littlejohn, A. & Margaryan, A. (2015). Measuring self-regulated learning in the workplace. International Journal of Training and Development. 19(1) 32-52. Gašević, D., Kovanović, V., Joksimović, S. & Siemens, G. (2014). Where is Research on Massive Open Online Courses Headed? A Data Analysis of the MOOC Research Initiative. International Review of Research in Open and Distance Learning, 15(5), 134-176. Gillani, N. & Eynon, R. (2014). Communication patterns in massively open online courses. The Internet and Higher Education, 23, 18-26. Haug, S., Wodzicki, K., Cress, U. & Moskaliuk, J. (2014). Self-Regulated Learning in MOOCs: Do Open Badges and Certificates of Attendance Motivate Learners to Invest More? In U. Cress & C. D. Kloos (Eds.). EMOOCs 2014 - European MOOC Stakeholder Summit, (pp. 66-72). Lausanne, Switzerland. Kizilcec, R. F., Piech, C. & Schneider, E. (2013). Deconstructing disengagement: Analysing learner subpopulations in massive open online courses. In Proceedings of the 3rd International Conference on Learning Analytics and Knowledge (pp. 170–179). New York, NY, USA: ACM. Retrieved 8 April 2015, from: http://dl.acm.org/citation.cfm?id=2460330 Littlejohn, A, & Milligan, C. (2015). Designing MOOCs for professional learners: tools and patterns to encourage self-regulated learning. eLearning Papers, 42, 38-45. Littlejohn, A., Hood, N., Milligan, C. & Mustain, P. (2016). Learning in MOOCs, motivations and self-regulated learning. The Internet and Higher Education. 29, 40-48. Littlejohn, A., Milligan, C. Fontana, R.P. & Margaryan, A. (2016). Professional learning through everyday work: how finance professionals self-regulate their learning. Vocations and Learning.

34. References Margaryan, A., Bianco, M. & Littlejohn, A. (2015). Instructional quality of Massive Open Online Courses (MOOCs). Computers & Education, 80, 77-83. Milligan, C., Littlejohn, A. & Margaryan, A. (2013). Patterns of engagement in connectivist MOOCs. MERLOT Journal of Online Learning & Teaching 9(2), 149-159. Milligan, C. & Littlejohn, A. (2014). Supporting professional learning in a massive open online course. International Review of Research in Open and Distance Learning 15(5) 197-213. Milligan, C. & Littlejohn. A. (under review). How health professionals regulate their learning in MOOCs. Poellhuber, B., Roy, N. Bouchoucha, I. & Anderson, T. (2014). The relationships between the motivational profiles, engagement profiles and persistence of MOOC participants. MOOC Research Initiative, Final Report. Ryan, R.M. & Deci, E.L. (2000). Intrinsic and extrinsic motivations: classic definitions and new directions. Contemporary Educational Psychology, 25, 54-67. Veletsianos, G., Collier, A. & Schneider, E. (2015). Digging Deeper into Learners Experiences in MOOCs: Participation in social networks outside of MOOCs, Notetaking, and contexts surrounding content consumption. British Journal of Educational Technology, 46,570 - 587 Wang, Y. & Baker, R. (2015). Content or platform: why do students complete MOOCs. MERLOT Journal of Online Learning and Teaching, 11(1), 17-30. Zheng, S., Rosson, M. B., Shih, P. C. & Carroll, J. M. (2015). Understanding Student Motivation, Behaviors and Perceptions in MOOCs. In Proceedings of the 18th ACM Conference on Computer Supported Cooperative Work & Social Computing (pp. 1882- 1895). ACM. Zimmerman, B. J. (2000). Attaining self-regulation: a social cognitive perspective. In M. Boekaerts, M. Zeidner, and P.R. Pintrich (Eds.), Handbook of self-regulation (pp13-39). Academic Press, San Diego, CA.

35. Recommendations 1.Enable professional learners to link theory learned in the MOOC with their work practice by setting personal goals, or personalizing course goals. 2.Help professional learners to reflect on the knowledge gained from the course and how it may be embedded into their work practice before the end of the course. 3.Support professional learners to continually monitor their learning to determine its ultimate value beyond their immediate learning experience. 4.Capitalize on the diversity of motivation, expectation, and prior knowledge and experience that is an inherent within all MOOC cohorts. 5.Encourage professional learners to discuss ideas from the course with co- workers in their external professional network as well as with other learners on the course. 6.Utilize the existing knowledge and experience that professional learners bring to the learning context. https://figshare.com/articles/MOOC_Design_Recommendations/1420557