Presentation is loading. Please wait.

Presentation is loading. Please wait.

BUILDING A PREDICTIVE MODEL A Behind the Scenes Look Mike Sharkey Director of Academic Analytics, The Apollo Group January 9, 2012.

Similar presentations


Presentation on theme: "BUILDING A PREDICTIVE MODEL A Behind the Scenes Look Mike Sharkey Director of Academic Analytics, The Apollo Group January 9, 2012."— Presentation transcript:

1 BUILDING A PREDICTIVE MODEL A Behind the Scenes Look Mike Sharkey Director of Academic Analytics, The Apollo Group January 9, 2012

2 THE 50,000 FT. VIEW We have lots of data; we need to set a good foundation… …so we can extract information that will help our students succeed

3 O UR D ATA F OUNDATION

4 INTEGRATED DATA WAREHOUSE LMS SIS CMS Applications Integrated Data Repository Integrated Data Repository Databases Reporting Tools Analytics Tools Business Intelligence Applicant

5 HOW IS IT WORKING? Continuous flow of integrated data Can drill down to the transaction level New data flows require in-demand resources Need skilled staff to understand the data model AdvantagesDisadvantages

6 B UILDING A P REDICTIVE M ODEL

7 PREDICTING SUCCESS… …BUT WHAT IS SUCCESS? Learning Program persistence Course completion ? ? Student drops out Student passes class Did the students learn what they were supposed to learn?

8 THE PLAN Use available data to build a model (logistic regression) Demographics, schedule, course history, assignments Develop a model to predict course pass/fail e.g. scale of will likely pass the course 1 will most likely fail the course Feed the score to academic counselors who can intervene (phone at-risk students)

9 THE MODEL Built different models Associates, Bachelors, Masters Predict at Week 0, Week 1, … to Week (last) Strongest predictive coefficients Course assignment scores (stronger as course goes on) Financial status (mostly at Week 0) Did the student fail courses in the past Credits earned in the program (tenure)

10 WHERE WE ARE TODAY Validation The statistics are sound, but we need to field test the intervention plan to validate the model scores What we learned The strongest parameters are the most obvious (assignments) Weak parameters: gender, age, weekly attendance Add future parameters as available Class activity, participation, faculty alerts, inactive time between courses, interaction with faculty, orientation participation, late assignments

11 THANK YOU! Mike Sharkey

12 5 CHALLENGES IN BUILDING & DEPLOYING LEARNING ANALYTICS SOLUTIONS Christopher Brooks

13 MY BIASES A domain of higher education Scalable and broad solutions The grey areas between research and production

14 QUESTION: YOUR BIASES: WHAT DO YOU THINK THE PRINCIPAL GOAL OF LEARNING ANALYTICS SHOULD BE? Enabling human intervention Computer assisted instruction (dynamic content recommendation, tutoring, quizzing) Conducting educational research Administrative intelligence, transparency, competitiveness Other (write in chat)

15 CHALLENGE 1: WHAT ARE YOU BUILDING Exploring data Intuition and domain expertise are useful Multiple perspectives from people familiar with the data More data types (diversity) is better, smaller datasets (instances) is ok Imprecision in data is ok Visualization techniques Answering a question Data should be cleaned and rigorous, with error recognized explicitly The quantity of data in the datasets (instances) strengthens the result Decision makers must guide the process (are the questions worth answering?) Statistical techniques

16 CASE 1: HOW HEALTHY IS YOUR CLASSROOM COMMUNITY (SNA)

17

18 CASE 2: APPLYING SUPERVISED LEARNING TECHNIQUES (CLUSTERING)

19 RESULTS VALIDATED, QUANTIFIED, AND ENCOURAGED MORE INVESTIGATION Hypotheses H1: There will be a group of minimal activity learners... H2: There will be a group of high activity learners... H3: There will be a group of disillusioned learners... H4: There will be a group of deferred learners...

20 CHALLENGE 2: WHAT TO COLLECT Too much versus too little Make a choice based on end goals Think in terms of events instead of the click stream Collecting everything comes with upfront development costs and analysis costs The risk is the project never gets off the ground Make hypotheses explicit in your team so they can decide how best to collect that data Follow agile software development techniques (iterate & get constant feedback) Build institutional will with small targeted gains

21 CHALLENGE 3: UNDERSTAND YOUR USER Breadth of Context Administrator Rates for degree completion, retention rate, re-enrolment rate, number of active students... (Abbreviated statistics) Instructional Design/Researcher Educational researcher, what works and what doesn't tools and processes should change... (Sophisticated statistics & visualizations) Instructor Evaluation of students, of a cohort of students, and identifying immediate remediation... (Visualization, Abbreviated statistics) Student Evaluation, evaluation, evaluation.... (Visualization)

22 WITH GREAT POWER COMES GREAT RESPONSIBILITY.... Some potential abuses of student tracking data Changing pedagogical technique to the detriment of some students Denying help to those who aren't really trying A failure of instructors to acknowledge the challenges that face students Is it ethical to give instructors access to student analytics data? Yes No Sometimes (write your thoughts in the chat)

23 CHALLENGE 4: ACKNOWLEDGE CAVEATS Analytics shows you a part of the picture only Dead tree learning, in-person social constructivism, shoulder surfing/account sharing Anonymization tools, javascript/flash blockers False positives (incorrect amazon recommendations) Misleading actions (incorrect self-assessment, or gaming the system (Baker)) Solutions Aggregation & anonymization Make error values explicit Use broad categories for actionable analytics

24 DOES LEARNER MODELLING OFFER SOLUTIONS? Learner modelling community blends with analytics. Open learner modelling (students can see their completed model) Scruitable learner modelling (students can see how the system model of them is formed) Question: I believe the student should have the right to view where analytics data about themselves has come from and who it has been made available to. Yes No Sometimes (and what are the implications on doing this? write in chat)

25 CHALLENGE 5: CROSS APPLICATION BOUNDARIES Data from different applications (clickers, lcms, lecture capture, SIS/CIS, publisher quizzes, etc.) doesn't play well together Requires cleaning Requires normalizing on semantics Requires access Data warehousing activities Is there a light on the horizon?

26 QUICK CONCLUSIONS Thus far I've learned it's important to: Know your goals Know your user Capture what you know you need and don't worry about the rest Acknowledge limitations of your approach Iterate, iterate, iterate Christopher Brooks Department of Computer Science University of Saskatchewan

27 LEARNING ANALYTICS FOR C21 DISPOSITIONS & SKILLS Simon Buckingham Shum Knowledge Media Institute, Open U. UK

28 L.A. FRAMEWORK TO THINK WITH… Discipline knowledge Educator owns and manages a single dataset

29 L.A. FRAMEWORK TO THINK WITH… Discipline knowledge Educator owns and manages a single dataset Educator owns and manages multiple datasets

30 L.A. FRAMEWORK TO THINK WITH… Discipline knowledge Educator owns and manages a single dataset Educator owns and manages multiple datasets Learners add their own datasets

31 L.A. FRAMEWORK TO THINK WITH… Discipline knowledge Educator owns and manages a single dataset Educator owns and manages multiple datasets Learners add their own datasets Hybrid closed + open datasets

32 L.A. FRAMEWORK TO THINK WITH… Discipline knowledge Educator owns and manages a single dataset Educator owns and manages multiple datasets Learners add their own datasets Hybrid closed + open datasets Hybrid closed + open analytics

33 L.A. FRAMEWORK TO THINK WITH… Discipline knowledge Educator owns and manages a single dataset Educator owns and manages multiple datasets Learners add their own datasets Hybrid closed + open datasets Hybrid closed + open analytics Focus of most LA effort beginning to move towards these more complex spaces

34 L.A. FRAMEWORK TO THINK WITH… Discipline knowledge Educator owns and manages a single dataset Educator owns and manages multiple datasets Learners add their own datasets Hybrid closed + open datasets Hybrid closed + open analytics Focus of most LA effort beginning to move towards these more complex spaces

35 L.A. FRAMEWORK TO THINK WITH… Discipline knowledgeC21 Learning Capacities Educator owns and manages a single dataset Educator owns and manages multiple datasets Learners add their own datasets Hybrid closed + open datasets Hybrid closed + open analytics critical for learner engagement, and authentic learning Focus of most LA effort beginning to move towards these more complex spaces

36 We are preparing students for jobs that do not exist yet, that will use technologies that have not been invented yet, in order to solve problems that are not even problems yet. Shift Happens LEARNING ANALYTICS FOR THIS?

37 The test of successful education is not the amount of knowledge that pupils take away from school, but their appetite to know and their capacity to learn. Sir Richard Livingstone, 1941

38 ANALYTICS FOR… C21 SKILLS? LEARNING HOW TO LEARN? AUTHENTIC ENQUIRY? social capital critical questioning argumentation citizenship habits of mind resilience collaboration creativity metacognition identity readiness sensemaking engagement motivation emotional intelligence 38

39 L.A. FRAMEWORK TO THINK WITH… Discipline knowledgeC21 Learning Capacities Educator owns and manages a single dataset Educator owns and manages multiple datasets Learners add their own datasets Hybrid closed + open datasets Hybrid closed + open analytics More LA effort needed e.g. 1. Disposition Analytics 2. Discourse Analytics Focus of most LA effort beginning to move towards these more complex spaces

40 ANALYTICS FOR LEARNING DISPOSITIONS

41 ELLI: EFFECTIVE LIFELONG LEARNING INVENTORY WEB QUESTIONNAIRE 72 ITEMS (CHILDREN AND ADULT VERSIONS: USED IN SCHOOLS, UNIVERSITIES AND WORKPLACE) Buckingham Shum, S. and Deakin Crick, R (2012). Learning Dispositions and Transferable Competencies: Pedagogy, Modelling, and Learning Analytics. Accepted to 2 nd International Conference on Learning Analytics & Knowledge (Vancouver, 29 Apr – 2 May, 2012).

42 VALIDATED AS LOADING ONTO 7 DIMENSIONS OF LEARNING POWER Changing & Learning Meaning Making Critical Curiosity Creativity Learning Relationships Strategic Awareness Resilience Being Stuck & Static Data Accumulation Passivity Being Rule Bound Isolation & Dependence Being Robotic Fragility & Dependence

43 ELLI GENERATES A 7-DIMENSIONAL SPIDER DIAGRAM OF HOW THE LEARNER SEES THEMSELF Bristol and Open University are now embedding ELLI in learning software. Basis for a mentored- discussion on how the learner sees him/herself, and strategies for strengthening the profile 43

44 ADDING IMAGERY TO ELLI DIMENSIONS TO CONNECT WITH LEARNER IDENTITY Milhouse

45 ELLI GENERATES COHORT DATA FOR EACH DIMENSION

46 …DRILLING DOWN ON A SPECIFIC DIMENSION

47 Plugin visualizes blog categories, mirroring the ELLI spider ENQUIRYBLOGGER: TUNING WORDPRESS AS AN ELLI-BASED LEARNING JOURNAL Standard Wordpress editor Categories from ELLI

48 ENQUIRYBLOGGER: COHORT DASHBOARD

49 LEARNINGEMERGENCE.NET more on analytics for learning to learn and authentic enquiry

50 ANALYTICS FOR LEARNING CONVERSATIONS

51 DISCOURSE LEARNING ANALYTICS Effective learning conversations display some typical characteristics which learners can and should be helped to master Learners written, online conversations can be analysed computationally for patterns signifying weaker and stronger forms of contribution

52 SOCIO-CULTURAL DISCOURSE ANALYSIS (MERCER ET AL, OU) Disputational talk, characterised by disagreement and individualised decision making. Cumulative talk, in which speakers build positively but uncritically on what the others have said. Exploratory talk, in which partners engage critically but constructively with each other's ideas. Mercer, N. (2004). Sociocultural discourse analysis: analysing classroom talk as a social mode of thinking. Journal of Applied Linguistics, 1(2),

53 Exploratory talk, in which partners engage critically but constructively with each other's ideas. Statements and suggestions are offered for joint consideration. These may be challenged and counter-challenged, but challenges are justified and alternative hypotheses are offered. Partners all actively participate and opinions are sought and considered before decisions are jointly made. Compared with the other two types, in Exploratory talk knowledge is made more publicly accountable and reasoning is more visible in the talk. Mercer, N. (2004). Sociocultural discourse analysis: analysing classroom talk as a social mode of thinking. Journal of Applied Linguistics, 1(2), SOCIO-CULTURAL DISCOURSE ANALYSIS (MERCER ET AL, OU)

54 ANALYTICS FOR IDENTIFYING EXPLORATORY TALK Elluminate sessions can be very long – lasting for hours or even covering days of a conference It would be useful if we could identify where quality learning conversations seem to be taking place, so we can recommend those sessions, and not have to sit through online chat about virtual biscuits Ferguson, R. and Buckingham Shum, S. Learning analytics to identify exploratory dialogue within synchronous text chat. 1st International Conference on Learning Analytics & Knowledge (Banff, Canada, 27 Mar-1 Apr, 2011)

55 De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. Discourse-centric learning analytics. 1st International Conference on Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr, 2011) KMIS COHERE: A WEB DELIBERATION PLATFORM ENABLING SEMANTIC SOCIAL NETWORK AND DISCOURSE NETWORK ANALYTICS Rebecca is playing the role of broker, connecting 2 peers contributions in meaningful ways

56 DISCOURSE ANALYSIS BACKGROUND KNOWLEDGE: Recent studies indicate … … the previously proposed … … is universally accepted... NOVELTY:... new insights provide direct evidence we suggest a new... approach results define a novel role... OPEN QUESTION: … little is known … … role … has been elusive Current data is insufficient … GENERALIZING:... emerging as a promising approach Our understanding... has grown exponentially growing recognition of the importance... CONRASTING IDEAS: … unorthodox view resolves … paradoxes … In contrast with previous hypotheses inconsistent with past findings... SIGNIFICANCE: studies... have provided important advances Knowledge... is crucial for... understanding valuable information... from studies SURPRISE: We have recently observed... surprisingly We have identified... unusual The recent discovery... suggests intriguing roles SUMMARIZING: The goal of this study... Here, we show... Altogether, our results... indicate Xeroxs parser can detect the presence of knowledge-level moves in text: Ágnes Sándor & OLnet Project: De Liddo, A., Sándor, Á. and Buckingham Shum, S. (In Press). Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study. Computer Supported Cooperative Work Journal

57 NEXT STEPS SOCIAL LEARNING ANALYTICS: Develop this framework to integrate social, discourse, disposition and other process-centric analytics DISPOSITION ANALYTICS: Extend the capabilities of the ELLI learning power platform using real-time analytics data from online learner activity DISCOURSE ANALYTICS: human+machine annotation of written discourse and argument maps

58 IN MORE DETAIL… Social Learning Analytics Buckingham Shum, S. and Ferguson, R. (2011). Social Learning Analytics. Available as: Technical Report KMI-11-01, Knowledge Media Institute, The Open University, UK. Discourse Analytics De Liddo, A., Buckingham Shum, S., Quinto, I., Bachler, M. and Cannavacciuolo, L. (2011). Discourse-Centric Learning Analytics. 1 st International Conference on Learning Analytics & Knowledge (Banff, 27 Mar-1 Apr, 2011). Eprint: Ferguson, R. and Buckingham Shum, S. (2011). Learning Analytics to Identify Exploratory Dialogue Within Synchronous Text Chat. 1 st International Conference on Learning Analytics & Knowledge (Banff, Canada, 27 Mar-1 Apr, 2011). Eprint: De Liddo, A., Sándor, Á. and Buckingham Shum, S. (2012, In Press). Contested Collective Intelligence: Rationale, Technologies, and a Human-Machine Annotation Study. Computer Supported Cooperative Work. DOI: /s x. Disposition Analytics Ferguson, R., Buckingham Shum, S. and Deakin Crick, R. (2011). EnquiryBlogger: Using Widgets to Support Awareness and Reflection in a PLE Setting. 1 st Workshop on Awareness and Reflection in Personal Learning Environments, PLE Conference 2011, July 2011, Southampton, UK. Eprint: Buckingham Shum, S. and Deakin Crick, R (2012). Learning Dispositions and Transferable Competencies: Pedagogy, Modelling, and Learning Analytics. Accepted to 2 nd International Conference on Learning Analytics & Knowledge (Vancouver, 29 Apr – 2 May, 2012). Working draft under revision:

59 SUMMARY Discipline knowledgeC21 Learning Capacities Educator owns and manages a single dataset Educator owns and manages multiple datasets Learners add their own datasets Hybrid closed + open datasets Hybrid closed + open analytics More LA effort needed We need analytics tuned to generic capacities which equip learners for novel challenges Focus of most LA effort mastery of core knowledge and skills in training is vital, but no longer sufficient


Download ppt "BUILDING A PREDICTIVE MODEL A Behind the Scenes Look Mike Sharkey Director of Academic Analytics, The Apollo Group January 9, 2012."

Similar presentations


Ads by Google