Educational Data Mining and DataShop John Stamper Carnegie Mellon University 1 9/12/2012 PSLC Corporate Partner Meeting 2012

The Classroom of the Future Which picture represents the “Classroom of the Future”? 2 9/12/2012 PSLC Corporate Partner Meeting 2012

3 The Classroom of the Future The answer is both! Depends of how much money you have... … but maybe not what you think… 9/12/2012 PSLC Corporate Partner Meeting 2012

4 The Classroom of the Future Rich vs. Poor – Poor kids will be forced to rely on “cheap” technology – Rich kids will have access to “expensive” teachers We are seeing this today! – Waldorf school in Silicon Valley – no technology – NGLC Wave III Grants – MOOCs (AI Course at Stanford) – Growth of adaptive technology companies – Online instruction – … and more… 9/12/2012 PSLC Corporate Partner Meeting 2012

5 What does this mean? My view is that we cannot stop this, I believe we must accept that economics will force this route. We should focus on improving learning technology New ways to improve teacher-student access Add more adaptive features to learning software Intelligent Tutors, at scale, using data! 9/12/2012 PSLC Corporate Partner Meeting 2012

9/12/2012 PSLC Corporate Partner Meeting Educational Data Mining “Educational Data Mining is an emerging discipline, concerned with developing methods for exploring the unique types of data that come from educational settings, and using those methods to better understand students, and the settings which they learn in.” –

9/12/2012 PSLC Corporate Partner Meeting Classes of EDM Methods (Baker & Yacef, 2009) Prediction Clustering Relationship Mining Discovery with Models Distillation of Data For Human Judgment

9/12/2012 PSLC Corporate Partner Meeting Prediction Develop a model which can infer a single aspect of the data (predicted variable) from some combination of other aspects of the data (predictor variables) Does a student know a skill? Which students are off-task? Which students will fail the class?

9/12/2012 PSLC Corporate Partner Meeting Clustering Find points that naturally group together, splitting full data set into set of clusters Usually used when nothing is known about the structure of the data – What behaviors are prominent in domain? – What are the main groups of students?

9/12/2012 PSLC Corporate Partner Meeting Relationship Mining Discover relationships between variables in a data set with many variables – Association rule mining – Correlation mining – Sequential pattern mining – Causal data mining

9/12/2012 PSLC Corporate Partner Meeting Discovery with Models Pre-existing model (developed with EDM prediction methods… or clustering… or knowledge engineering) Applied to data and used as a component in another analysis

9/12/2012 PSLC Corporate Partner Meeting Distillation of Data for Human Judgment Making complex data understandable by humans to leverage their judgment Text replays are a simple example of this

9/12/2012 PSLC Corporate Partner Meeting Knowledge Engineering Creating a model by hand rather than automatically fitting model In one comparison, leads to worse fit to gold- standard labels of construct of interest than data mining (Roll et al, 2005), but similar qualitative performance

9/12/2012 PSLC Corporate Partner Meeting LearnLab The LearnLab has played a pivotal role in the creation of the EDM community The CMDM thrust of the center focuses on Educational Data Mining DataShop is also a key tool for the EDM community

9/12/2012 PSLC Corporate Partner Meeting DataShop Open repository for educational data Many large-scale datasets both public and private Tools for – exploratory data analysis – learning curves – domain model testing

9/12/2012 PSLC Corporate Partner Meeting DataShop Import/Export of data Custom fields Easy Knowledge Model creation and validation Web services for tools integration

Demo 9/12/2012 PSLC Corporate Partner Meeting

9/12/2012 PSLC Corporate Partner Meeting Engaging the KDD/ICDM Community Some hesitation from these groups – Educational data not interesting – Too applied – Not “big” enough for eScience This was one motivation for the 2010 KDD Cup

9/12/2012 PSLC Corporate Partner Meeting KDD Cup Competition  Knowledge Discovery and Data Mining (KDD) is the most prestigious conference in the data mining and machine learning fields  KDD Cup is the premier data mining challenge  2010 KDD Cup called “Educational Data Mining Challenge”  Ran from April 2010 through June 2010

9/12/2012 PSLC Corporate Partner Meeting KDD Cup Competition Competition goal is to predict student responses given tutor data provided by Carnegie Learning DatasetStudentsStepsFile size Algebra I ,3109,426,9663 GB Bridge to Algebra ,04320,768, GB

9/12/2012 PSLC Corporate Partner Meeting KDD Cup Competition  655 registered participants  130 participants who submitted predictions  3,400 submissions

9/12/2012 PSLC Corporate Partner Meeting KDD Cup Competition  Advances in prediction and cognitive modeling  Excitement in the KDD Community  The datasets are now in the “wild” and showing up in non KDD conferences  New competitions have been done and are in the works

Opportunities Huge potential for EDM and DataShop to improve educational systems DataShop is open and staff is available to help get users started Great option for creating capstone projects 9/12/2012 PSLC Corporate Partner Meeting

9/12/2012 PSLC Corporate Partner Meeting EDM Community is Online!   EDM 2013 in Memphis TN in July  Questions: