1. VERSION 2 TO DO Upgrade logos to high res Add in “demo” slides Add in “backup” slides

2. Richard BaraniukFounder and Director Daniel WilliamsonManaging Director David HarrisEditor-in-Chief Kathi Fletcher Product Manager OpenStax Courseware

3. GOALS of OpenStax Courseware 1.broader access to high-quality courseware 2.new tools to improve learning experiences (machine learning, cognitive science models) 3.validation in real classrooms + research

4. 1.broader access to high-quality courseware 2.new tools to improve learning experiences (machine learning, cognitive science models) 3.validation in real classrooms + research

5. today’s agenda OpenStax Courseware building blocks –digital content: Connexions + OpenStax College –digital assessment: OpenStax Tutor –cognitive science –machine learning key technology components content focus areas go-to-market strategy discussion and closing statement

6. digital content open ed publishing platform established in 1999 25000 learning objects in 40 languages millions of users per month library of 25 free and open college textbooks professionally authored and peer reviewed 875 adoptions, saving 140,000 students over $14M

7. digital assessment in use at 12 colleges (Rice, Georgia Tech, Duke, UT El Paso, …) built-in research infrastructure integrated cognitive science principles (collaborators at Duke, UT-Austin, WashU) flexible platform for computer-based assessment and research

8. learning principles retrieval practice –retrieving information from memory is not a neutral event; rather it changes memory spacing –distributing practice over time produces better long-term retention than massing practice feedback –closes the learning feedback loop –must be timely learners content data

9. digital assessment experiment at Rice 2012 findings: Students using cognitive science principles in OST scored ½-1 GPA point better than those using standard practice homework flexible platform for computer-based assessment and research

10. learning analytics content analytics

11. learning/content analytics classical approach – “knowledge engineering” –domain experts pore over content, assessments, data, tagging and building rules –fragile, expensive, not scalable, not transferable modern approach – “machine learning” –learn directly from data –automatic –robust, inexpensive, scalable, transferable

12. standard practice Johnny Eve Patty Neelsh Nora Nicholas Barbara Agnes Vivek Bob Fernando Sarah Hillary Judy Janet

13. standard practice Johnny Eve Patty Neelsh Nora Nicholas Barbara Agnes Vivek Bob Fernando Sarah Hillary Judy Janet Goal: using only “grade book” data, infer: 1.the concepts underlying the questions (content analytics) 2.each student’s “knowledge” of each underlying concept (learning analytics)

14. students problems sparse factor analysis Goal: using only “grade book” data white: correct response black: incorrect response grey:unobserved infer: 1.the concepts underlying the questions (content analytics) 2.each student’s “knowledge” of each underlying concept (learning analytics)

15. students problems + students concepts each problem involves a combination of a small number of key “concepts” each student’s knowledge of each “concept” each problem’s intrinsic “difficulty” ~ Ber Sparse Factor Analysis

16. questions (w/ estimated inherent difficulty) concepts student knowledge profile 87 55 23 93 62 Patty

18. DEMO SLIDES

19. technology architecture

20. content Principles of Accounting Principles of Management American Government Microbiology content development in partnership with Words & Numbers (9 texts published to date, X in production) quality control via extensive peer review and classroom testing at partner colleges ROOM FOR ONE MORE BULLET 1.65 million students/year

21. go-to-market strategy research partners will co-develop –Salt Lake Community College, University of Georgia pilot partners will field test –The Ohio State University, Auburn University, University System of Georgia-Online Courses, Central New Mexico College, South Florida State College, Maricopa CC District, Tarrant County CC key elements –fit into existing faculty/student workflow –build an ecosystem of affiliate partners –execute advertising and marketing campaigns –employ viral new media approaches –employ direct marketing and customer relationship management system proven success 2012-2014

22. driving adoption: workflow principles Scope/sequence: content is available in complete discrete units of 100/200 level courses flexible: we provide for a blended learning experience part of the grade: assignable with metrics given APIs: interoperable across multiple platforms

23. access to drive adoption Institutional partners and pilots 1.Salt Lake Community College 2.University of Georgia 3.Georgia University System 4.Auburn University 5.College of South Florida 6.The Ohio State University 7.Mariposa Community College District 8.Shasta College District Large Base of OpenStax College Adopters 1.Approaching 1,000 adoptions 2.Over 135,000 student seats

24. Lumen Learning Logo CCOER Logo Lumen Learning Logo CCOER Logo Students and Faculty Administrators and Faculty Ecosystem Partners

25. summary – 1 What makes your proposed courseware “exemplary?” –strong research base in machine learning, cog science –15 years of experience in digital education  network of administrators/educators who already use our content/tools –once proven, can expand at minimal cost into comprehensive library of highest enrollment college courses –strong backing of Rice University –Not sure about this one: Flexible: works in multiple modes to meet various workflow requirements

26. summary – 2 How will your proposed courseware enable a “great leap forward” in improving the learning outcomes for low income, disadvantaged learners? –platform integrates cognitive science principles that have been proven to improve knowledge retention and transfer  large literature of laboratory studies  Rice 2013 experiment –machine learning learning/content analytics scale across courses  dramatically lower cost/prices will result as compared to courseware based on hand-coded ontologies

27. summary – 3 Why do you believe your team can develop your proposed courseware? –experienced team has built  OpenStax College 140,000 students in 2 years Physics textbook displacing market leaders  OpenStax Tutor 50 years of experience at IBM, Microsoft, JP Morgan Chase, Northrop Grumman, Texas Instruments, Cengage, Pearson, …  machine learning 20 years of research in Rice DSP group –$72M in research, development, and deployment funding from 15 foundations and government agencies

28. summary – 4 How will you achieve wide adoption of your proposed courseware? –proven go-to-market strategy  most successful launch of a physics text in 30 years (17.5% market share in 2 years) –WHAT ELSE (remember this is a summary)

29. summary – 5 How does this project align to the charitable purpose set forth by the Foundation? –Gates Foundation “guiding principles of Global Access” –The technology and products developed with grant funds be made available and accessible at an affordable price to people most in need  Because of the relatively low cost (due to scalability enabled by machine learning), we can sustainably make OpenStax Courseware affordable to those most in need –Knowledge and information gained from the project be promptly and broadly disseminated  As a university project dissemination is core to our mission; we have already published a number of papers in machine learning and cog sci

30. closing statement

31. curriculum (re)design personalized learning pathways cognitive science research machine learning cycles of innovation

32. closing Andrew Carnegie: “personalized courseware library” of the future Eric j appeal to carnegie – this is the “personalized courseware library” of the future

33. backup slides

34. More on tech More on ecosystem/marketing?

35. budget Overview of the $5M budget and key categories

36. privacy

37. sparfa

38. from grades to concepts students problems data –graded student responses to unlabeled questions –large matrix with entries: white: correct response black: incorrect response grey:unobserved standard practice –instructor’s “grade book” = sum/average over each column goal –infer underlying concepts and student understanding without question-level metadata

39. students problems data –graded student responses to unlabeled questions –large matrix with entries: white: correct response black: incorrect response grey:unobserved goal –infer underlying concepts and student understanding without question-level metadata key observation –each question involves only a small number of “concepts” (low rank) from grades to concepts

40. students problems ~ Ber statistical model converts to 0/1 (probit or logistic coin flip transformation) estimate of each student’s ability to solve each problem (even unsolved problems) red = strong ability blue = weak ability

41. students problems + SPARse Factor Analysis ~ Ber

42. students problems + students concepts SPARFA each problem involves a combination of a small number of key “concepts” each student’s knowledge of each “concept” each problem’s intrinsic “difficulty” ~ Ber

43. students problems solving SPARFA factor analyzing the grade book matrix is a severely ill-posed problem significant recent progress in relaxation-based optimization for sparse/low-rank problems –matrix based methods(SPARFA-M) –Bayesian methods(SPARFA-B) similar to compressive sensing

44. standard practice Johnny Eve Patty Neelsh Nora Nicholas Barbara Agnes Vivek Bob Fernando Sarah Hillary Judy Janet Grade 8 science 80 questions 145 students 1353 problems solved (sparsely) learned 5 concepts

45. Grade 8 science 80 questions 145 students 1353 problems solved (sparsely) 5 concepts

46. questions (w/ estimated inherent difficulty) concepts student knowledge profile 87 55 23 93 62

47. marketing

48. driving adoption: workflow principles Scope/sequence: content is available in complete discrete units of 100/200 level courses flexible: we provide for a blended learning experience part of the grade: assignable with metrics given APIs: interoperable across multiple platforms

49. access to drive adoption Institutional partners and pilots 1.Salt Lake Community College 2.University of Georgia 3.Georgia University System 4.Auburn University 5.College of South Florida 6.The Ohio State University 7.Mariposa Community College District 8.Shasta College District Large Base of OpenStax College Adopters 1.Approaching 1,000 adoptions 2.Over 135,000 student seats

50. Lumen Learning Logo CCOER Logo Lumen Learning Logo CCOER Logo Students and Faculty Administrators and Faculty Ecosystem Partners

51. charitable purpose mission of providing free access since 1999 free access for tutoring and review anticipate a low cost, analytic driven, version for classroom use, approx $10/student to maintain system