Using scenarios to promote learning analytics practice for teachers Project: Building an evidence-base for teaching and learning design using learning analytics data Four NZ institutions - eight case studies Cathy Gunn, Claire Donald, Marion Blumenstein, Jenny McDonald, John Milne, Mark Nichols & Eva Heinrich

Project Aims Guide selection and analysis of data to answer learning design questions: – ‘Classify’ learning analytics data available through common LMS / VLE and elearning tools – Use analytics data to illuminate relationships between teaching/learning design and learning outcomes – Promote use of “learning intelligence” in course design and institutional practice

Approach to PD Low barriers to entry (perhaps a different name?) Avoid controversy (while issues are worked through) Practice based, practice focused (obvious relevance) Address common questions & challenges (motivate)  Scenarios derived from case studies & interviews, adaptable to institutional context, use of examples

Scenarios  actionable insights 3 Broad areas: Student retention  investigate, support Student engagement  observe interactions, understand learners, promote achievement Student performance  inform design decisions

Scenario format Problem = common challenge Strategy = use analytics data to address problem Data sources = quantitative and qualitative Evaluate and take action = actionable insights Design implications = practical steps Caveats = what to expect and what to avoid

Example: Gain insights into student (mis)conceptions and knowledge Problem Students arrive at university with high school passes in subjects they continue to study Early success builds false confidence where experience shows common misconceptions and rote learning can lead to failure

Strategy What - use analytics data to understand what students know, expose common misconceptions, and influence learning strategies How - a purpose built pre-course test or quiz to offer insights for teachers and students; provide students with constructive feedback and point to tutorials designed to promote effective learning strategies and knowledge development Example: Gain insights into student (mis)conceptions and knowledge

Data sources Analytics report - quiz scores, correct / incorrect answers, number of attempts, use of hints, click through from feedback Question design reveals common misconceptions, invite written or verbal feedback to clarify meaning of quantitative data. Use discussions, surveys, interviews or focus groups to add a qualitative dimension. Example: Gain insights into student (mis)conceptions and knowledge

Evaluate and take action Know what students bring to a course to sharpen the focus of teaching and learning design; Discuss quiz results with students; explain common misconceptions and pathways to successful study; Develop online tutorials, monitor use, invite feedback on perceived usefulness Example: Gain insights into student (mis)conceptions and knowledge

Design implications Requires a pre-course quiz and tutorials to introduce or revisit subjects previously covered; Investment of time small relative to benefits of being able to focus on what evidence shows learners need. Example: Gain insights into student (mis)conceptions and knowledge

Caveats Misconceptions can be deeply rooted, addressing them may be a long-term process of reinforcement. Additional strategies might include peer review, frequent quizzes or tests, student designed questions or study resources; Causal relationships hard to establish, but higher mean grades may be indicative.

Example: Gain insights into student (mis)conceptions and knowledge Scenario topics Support students to avoid drop out or failure Gain insights into (mis)conceptions and knowledge Track student use of resources Analyze answers to explore disciplinary knowledge Use peer evaluation to lift student performance Give students a sense of how they are performing Understand why students struggle with concepts Monitor online discussion to inform teaching Track online interactions to correlate with grades

An appeal to your imagination… Scenarios as customizable templates PD support over semester / year (if required) and open access Supportive institutional environment where controversies have been addressed, data is freely available and in usable format…

What do you think? Are scenarios a useful approach? What ‘common problems’ would you focus on? Would you use a different structure? Any other feedback?