1. AN EXPERIENCE REPORT ASSESSING A PROFESSIONAL DEVELOPMENT MOOC FOR CS PRINCIPLES SIGCSE 2017 – MEMPHIS, TN JEFF GRAY DEPARTMENT OF COMPUTER SCIENCE UNIVERSITY OF ALABAMA GRAY@CS.UA.EDU JONATHAN CORLEY DEPARTMENT OF COMPUTER SCIENCE UNIVERSITY OF ALABAMA CORLE001@CRIMSON.UA.EDU BRIAN P. EDDY DEPARTMENT OF COMPUTER SCIENCE UNIVERSITY OF WEST FLORIDA BEDDY@UWF.EDU This project is supported by NSF Award #1240944 and Google CS4HS

2. PURPLE STYLE DEDICATED TO THE CSP “PURPLE BOOK”

3. CSP4HS 2015 TEAM

4. CS4ALABAMA TEACHERS

5. PRESENTATION OVERVIEW  Overview of Course and Face-To-Face Event  Course Demographics  Lessons Learned  Challenges Encountered Designing and Delivering the MOOC  Evaluating Impact of Face-to-Face & Online Community  Conclusions PRESENTATION OVERVIEW

6. OVERVIEW OF COURSE AND FACE-TO-FACE EVENT

7. CSP4HS OVERVIEW  2011: College Board and Google CS4HS  College Board CSP Pilot: University-level course; 40 pre-service SEMA trained each year  First Google CS4HS offered  2013: NSF CE21 grant  Focus on a PD model using NMSI approaches; adapted in other states  Cohort 1 Teacher Leaders expand curriculum  2014 and 2015: Google CS4HS MOOC  Teacher-facing online course offered in summer  Still open and available (e.g., Colorado district has 12 teachers currently going through course)

8. CSP4HS 2015  2015 summer participants: 947  Current enrollment: 1,109  All 50 states (except Wyoming)  Similar in 2014…

9. CSP4HS – A PD MOOC ABOUT CS PRINCIPLES CS Principles  A few (7) big ideas  Broader CS coverage  Performance Tasks Teacher PD focus Building a community  Face-to-Face event  Online community Course Content  6 Units over 6 weeks  2 week break in middle  Lesson quizzes  Midterm  Create PT  Hangout Office Hours  Hangouts on Air  Community of Practice (Piazza)

10. FACE-TO-FACE MEETINGS NSF CS4Alabama Project (Supported by NSF CE21)  47 teachers across Alabama  Lasted one week running 9am-4pm  Two evening activities  Parallel sessions offered to support both new participants and returning participants Other workshops  Similar event offered by 5 other state groups  Other workshops not included in our analysis

11. 2015 CSP4HS COURSE DEMOGRAPHICS

12. WHO IS PARTICIPATING Female (57%) Male (43%) Gender of Participants Grades Taught by Participants Participant School Classification Percent of Participants Teaching Subject Areas

13. PROGRAMMING LANGUAGE EXPERIENCE

14. CS TEACHING EXPERIENCE AND PREVIOUS TRAINING High w/ CSP (27%) High w/o CSP (25%) Low Moderate 5+ Courses (38%) 0-2 Courses (46%) 3-4 Courses (16%) CS Teaching Experience Formal CS Training

15. PARTICIPANT INTENTIONS Full-Course (65%) Partial (30%) Browse (5%) Intent to Complete Course Participant Plans to Teach CS Principles Possible (37%) Planned (45%) Yes (15%) No (3%) 13% had completed course at the time of writing.

16. LESSONS LEARNED FROM OUR CSP4HS EXPERIENCE

17. CHALLENGES ENCOUNTERED DESIGNING THE MOOC Maintaining and developing materials and curriculum

18. CHALLENGES ENCOUNTERED DESIGNING THE MOOC Multiple Tools for MOOC Design Google Course Builder: Core content delivery (Google) YouTube:  Video hosting Google Drive  Lesson slides  Curriculum material  Sample projects Google Forms + Flubaroo  Pre/Post survey  Quizzes and exams Google Hangouts On & Off Air  Interactive sessions w/ guest speakers  Virtual office hours (Not Google) Piazza: Announcements, Q&A, Intra-community discussion and sharing Thanks to Google for providing a host of great tools and funding us!

19. CHALLENGES ENCOUNTERED DURING THE MOOC Managing Concerns at Scale  Communication  Grading  Differentiated Learners Create PT

20. EVALUATING IMPACT OF FACE-TO-FACE & ONLINE PARTICIPANTS Face-to-Face  Some MOOC participants also involved in our statewide training session  Several other states ran similar training sessions  We do not include participants in other state’s training sessions in the analysis Online Participants  Majority of participants are online-only interacting solely through email, Piazza, and Hangouts  Some participants did not participate in the online community  We evaluated the impact of those participating in the online community We only consider participants interested in completing the course Full-Course Partial Browse Intent to Complete Course

21. IMPACT OF FACE-TO-FACE ON COURSE COMPLETION Face-to-Face participants significantly more likely to complete course materials

22. IMPACT OF FACE-TO-FACE ON COURSE ASSESSMENTS No statistical difference between populations on assessment performance. However, a significant difference was found in completion rates.

23. IMPACT OF FACE-TO-FACE ON PARTICIPATION IN ONLINE DISCUSSION Slight difference in contributions per participant in community of practice. Significantly more face-to-face participants involved in community of practice.

24. IMPACT OF COMMUNITY OF PRACTICE ON COURSE COMPLETION Those who participated in the community of practice completed significantly more of the course.  Wilcoxon rank-sum with p<.05

25. IMPACT OF COMMUNITY OF PRACTICE ON COURSE ASSESSMENT No significant difference on mid- term score. Statistically significant difference on Create-PT score. Higher completion rate for participants in online discussion.

26. CSP4HS RESOURCES OVERVIEW  CSP4HS Course Site ◦ Units 1-4 lessons complete (all 6 unitresources available) ◦ http://csp-cs4hs.appspot.com  CSP4HS Google Drive ◦ Resources for all 6 units ◦ http://tinyurl.com/CSP4HS-2015-Resources  CSP4HS YouTube Channel ◦ Over 120 videos on CSP topics ◦ http://tinyurl.com/CS4Alabama-YouTube

27. CONCLUSIONS Discussed challenges designing and running our MOOC  Designing for evolving standards  Integrating multiple components to build a cohesive course  Challenges of scale: communication, grading, differentiated learners Evaluated the impact of participation (face-to-face or an online community)  Impact on course completion  Impact on performance of participants

28. QUESTIONS / COMMENTS / HUMOROUS ANECDOTES This project is supported by NSF Award #1240944 and Google CS4HS

29. MORE DETAILS OF CSP4HS SETUP

30. COMMUNICATION AND INTERACTION Piazza  Online community discussion (including sub-communities)  Announcements  Asynchronous communication with instructors Live Guest Talks  Via Hangout on Air  Archived to YouTube Synchronized Office Hours via Hangout

31. COURSE MATERIALS Course Lessons Hosted using Google Course Builder (https://csp-cs4hs.appspot.com/)https://csp-cs4hs.appspot.com/  Text discussion of topics  Lesson video hosted on YouTube (http://tinyurl.com/CS4Alabama-YouTube)http://tinyurl.com/CS4Alabama-YouTube  Lesson slides hosted on Google Drive (http://tinyurl.com/CSP4HS-2015-Resources)http://tinyurl.com/CSP4HS-2015-Resources  Lesson quizzes using Google Forms with Flubaroo (http://www.flubaroo.com/)http://www.flubaroo.com/  Programming Assignment modeled after CSP Create Performance Task (http://apcsp.org/images/Resources/CreatePT_2016.pdf)http://apcsp.org/images/Resources/CreatePT_2016.pdf

32. COURSE MATERIALS Redirect to External Sources for Further Support of Lesson Materials:  CS10K Community (https://cs10kcommunity.org/)https://cs10kcommunity.org/  AP Teacher Community provided by College Board (https://apcommunity.collegeboard.org/)https://apcommunity.collegeboard.org/  Code.org (https://code.org/)https://code.org/  CS Unplugged (http://csunplugged.org/)http://csunplugged.org/  Ted Ed (http://ed.ted.com/)http://ed.ted.com/  And more