1. Canvas User Experience (UX) Studies Presented today by: Amy Garbrick and Maggie Frederick, Penn State University Core Research Team: ●Amy Garbrick: Director of Learning Design (College of IST) ●Andrea Gregg: Manager of Instructional Design (World Campus) ●Ronda Reid: Instructional Designer (College of IST) ●Vicki Williams: Manager, Assessment & Evaluation Research (ITS)

2. Where It Started…with one college and it grew... Canvas User Experience (UX) Studies  A simple thought: College of Information Sciences and Technology (IST) wanted to find out how students approached the IST course design in Canvas  Decided to test students: College of IST came up with a 8 tasks for standard course items and then recorded students as they were going through the tasks 2

3. Canvas User Experience (UX) Studies @ PSU 10 min Overview 3 Definitions:  eLearning Shops = Learning Design Units within a college/campuses that design and develop online courses; All support online some also support Resident Instruction (RI) courses  World Campus = Delivery unit for courses designed by eLearning Shops; Also has it’s own Learning Design unit that creates courses for Colleges/Campuses who do not have their own eLearning Shop or the capacity

4. Canvas User Experience (UX) Studies @ PSU 10 min Overview 2 different efforts: 1.Talk-alouds  Round 0: College of IST Student Interns  Round 1: College of IST WC Students  Round 2: Cross-Unit UX Study -- EMS, Liberal Arts, World Campus 2.User Survey  Hot off the press results  Student design preferences  5,000 World Campus students received survey 4

5. Canvas User Experience (UX) Studies @ PSU 10 min Overview 5 Tell us what you think… Students in the talk-aloud sessions most often went to: 1.Syllabus 2.Modules 3.Schedule 4.Homepage

6. Canvas User Experience (UX) Studies @ PSU 10 min Overview 6 Here’s what our students said... Students in the talk-aloud sessions most often went to: 1.Syllabus 2.Modules 3.Schedule 4.Homepage

7. Canvas User Experience (UX) Studies @ PSU 10 min Overview 7 “Modules are the primary source of information regarding courses, understanding that is paramount.” ~Subject 1 “Pretty much everything is under the modules. That's typically where I would go.“ ~Subject 2 Modules, Modules, Modules!

8. Canvas User Experience (UX) Studies @ PSU 10 min Overview 8 Tell us what you think.... Students in the survey said the most important component on the Syllabus was: 1.Grading Weights 2.Grading Scale 3.List of Course Materials 4.Course Schedule

9. Canvas User Experience (UX) Studies @ PSU 10 min Overview 9 Here’s what our students said... Students in the survey said the most important component on the Syllabus was: 1.Grading Weights 2.Grading Scale 3.List of Course Materials 4.Course Schedule

10. Canvas User Experience (UX) Studies @ PSU 10 min Overview 10 Tell us what you think.... Students in the survey said the most important component in the Course Schedule was: 1.Grades 2.Assignments 3.Due Dates 4.Lesson Date Ranges

11. Canvas User Experience (UX) Studies @ PSU 10 min Overview 11 Here’s what our students said... Students in the survey said the most important component in the Course Schedule was: 1.Grades 2.Assignments 3.Due Dates 4.Lesson Date Ranges

12. Why it’s Important to Talk to Learners “[A]sking students about their experiences of e-learning gives surprising, individual, and frequently contradictory results and this in itself is good reason for listening to students more often and more thoroughly if we are to avoid making assumptions about their experience from our teacher-centred view.” 12 (Sharpe & Bennefield, 2005)

13. PSU Who we are 13

14. 14

15. Recent Research WC LMS Requirements (2013, qual.) What are World Campus requirements for the LMS? 2 Int. & 4 focus groups - w/ WC students (12), faculty (13), staff (12) Identified requirements (e.g. mobile, easy, discussion, social presence) Report presented to TLT, Bb, WC; Used in LMS pilots Online Student Interface Study (2015, quant.) What are student perceptions in navigating online courses? Sub-committee of the DMT; OAR survey; 387 respondents Mostly +; Critiques: locations for due dates, assignments Student Interface Task Force using report to make recommendations Socient Study (2015, quant.) What improvements needed for outstanding student exp.? Socient survey: 1649 respondents Mostly positive w/ areas for improvement (e.g. career services, advising) LD improvements: usability, tech consistency, instructor involve. 15

16. What We’ll Cover Today  Canvas User Experience (UX) Studies  Talk-alouds  Round 0: College of IST Student Interns  Round 1: College of IST WC Students  Round 2: Cross-Unit UX Study -- EMS, Liberal Arts, World Campus  User Survey  Hot off the press results  Student design preferences  5,000 World Campus students received survey 16

17. Canvas User Experience (UX) Studies 17

18. Where It Started…with one college and it grew... Canvas User Experience (UX) Studies  A simple thought: College of Information Sciences and Technology (IST) wanted to find out how students approached the IST course design in Canvas  Decided to test students: College of IST came up with a 8 tasks for standard course items and then recorded students as they were going through the tasks 18

19. (Task 1) You have a question on the Lesson 07 Web Usability essay assignment. How would you find out more about this assignment? (Task 2) Your instructor sent an email reminding you to complete the Lesson 03 readings. How would you locate your reading assignments? (Task 3) You would like to receive a weekly email notification regarding course announcements. How would you set-up this notification? Examples of What We Asked 19

20. Tested the Test (aka Canvas UX Round 0): Bring in the Guinea Pigs Image: Megan van der Elst (http://jezebel.com/hipster-guinea-pig-professor-boo-boo-was-cute-before-it-1595022520)Megan van der Elsthttp://jezebel.com/hipster-guinea-pig-professor-boo-boo-was-cute-before-it-1595022520 20

21. Canvas UX Round 1: IST World Campus Students  Recruited 5 undergrad IST World Campus students  Willing to come to University Park for the study  IRB Approved study  Gave them a $50 gift certificate to Amazon.com  4 male, 1 female:  1 student: aged 20-23  1 student: aged 24-29  2 students: aged 30-39  1 student aged 40-49 21

22. Canvas UX Round 2: Cross-Unit Canvas UX Study 22 CROSS UNIT IST WCLD EMS LIBERAL ARTS TLT Our Team: Ronda, Tao, Tracy, Mike, Maggie, Vicki Jenn, Amy, Andrea The combination of prior studies with the Canvas transition has provided opportunities for more collaborative research and efforts to “get it right.”

23. IST, EMS, Liberal Arts, WCLD, w/ TLT  Maggie Frederick -- College of Earth & Mineral Sciences (EMS)  Jennifer Gray & Tracy Wallace -- College Liberal Arts)  Mike Brooks -- World Campus, Learning Design unit (WCLD)  Each “shop” recruited 5 WC learners; each shop used own course design 14 new participants + 5 from IST = 19 in total 23 Canvas UX Round 2: Cross-Unit Canvas UX Study

24. IST, EMS, Liberal Arts, WCLD, w/ TLT  Completed set of tasks, used talk-aloud protocol  Audio-, video-, and screen-recorded  IRB Approved study  Gave them a $50 gift certificate to Amazon.com  15 cards were funded by central ITS 24 Canvas UX Round 2: Cross-Unit Canvas UX Study

25.  Currently undergoing data analysis on audio, video, and transcripts  Should have more results soon  But, anecdotally, did see some of the same themes we saw in IST’s Canvas UX Round 0 and Round 1 testing 25 Canvas UX Round 2: Cross-Unit Canvas UX Study

26. Canvas UX Round 0 and Round 1: What Did We Find in IST Testing? 26 Image: Staples.com “Easy to find.” ~Subject 5 (Task 4) “Once I understood what modules were it was fairly easy to find.” ~Subject 4 (Task 1) “Not too bad, easy once you get familiar” ~ Subject 1 (Task 2)

27. Canvas UX Round 0 and Round 1: Modules, Modules, Modules! http://mgoblog.com/mgoboard/dantonio-elicits-sympathy-jamie-samuelson-detroit-free-press 27

28. Canvas UX Round 0 and Round 1: Modules, Modules, Modules! 28 “Modules are the primary source of information regarding courses, understanding that is paramount.” ~Subject 1 “Pretty much everything is under the modules. That's typically where I would go.“ ~Subject 2

29. Canvas UX Round 0 and Round 1: Canvas vs. ANGEL 29 “I think this is a big improvement over ANGEL.” ~Subject 1 “That is really simple to find and I find that a lot easier than the current method of ANGEL…” ~Subject 4 “That's a lot nicer than the current system.” ~Subject 5 Image: Smeal

30. User Survey 30

31. Mixed Methods Analysis Quantitative  Descriptive and tests for significance  Reliability Qualitative  Open-ended survey responses analyzed in NVivo for themes & patterns in responses  Produce Word clouds / trees, etc. 31 Canvas User Survey

32. 32 Canvas User Survey Survey Survey developed initially in collaboration with IST faculty Jess Kropczynski to identify user behaviors and preferences in Canvas screen design Modified by cross-unit team for this round of data collection

33. 33 Canvas User Survey Participants ●Sent to almost 5,000 students whose home campus is WC and who took at least one SP16 Canvas course ○ Landing page preferences (e.g. lion, recent activity, syllabus) ○ Syllabus organization preferences (e.g. full, compressed, separate schedule) ○ Navigation menu preferences (e.g. concise, full)

34. 34

35. Canvas User Survey: Key Findings Image: http://iwantmyshowback.com/wp-content/uploads/2011/05/news-hot-off-the-press-whats-new.jpg 35

36. Canvas User Survey: Key Findings Home Page Given the four alternatives you were shown previously, please indicate which Home page you prefer by dragging and placing your selections in order of preference Modules Syllabus Recent Activity Welcome 36

37. Canvas User Survey: Key Findings Syllabus Given the three Syllabus alternatives, please indicate your preference for each by dragging your preferences in the order you prefer them Syllabus View A Syllabus View B Syllabus View C 37

38. Canvas User Survey: Key findings 38

39. Canvas User Survey: Key Findings Course Schedule on Syllabus Page 39

40. Canvas User Survey: Key Findings Course Schedule: Print 40

41. Canvas User Survey: Key Findings Most important to see in the Syllabus:  Course Schedule  Grading Weights  Grading Scale  List of Course Materials Most important components in the Course Schedule:  Due Dates  Assignments  Lesson Date Ranges 41

42. Canvas User Survey: Key Findings 42 When you view your Syllabus, what information is most important to you? AnswerAverage ValueStandard DeviationResponses Course Schedule5.550.78511 Grading weight of assignments (how much each assignment is worth) 4.921.05490 Grading scale (what percentage is necessary for an A in the course) 4.781.22482 List of Course Materials (books, eReserves, software, etc.) 4.541.29473 Instructor Information4.131.36440 Resources4.131.41391

43. Canvas User Survey: Key Findings Navigation  Assignments was the most important item to have on the Navigation menu, followed by Grades and Modules.  Students felt they did not need to see Files, Pages, Conferences, or People in a Navigation menu and were least sure of the meaning of the Pages or Conferences items. 43

44. Canvas User Survey: Key Findings 44

45. Canvas User Survey: Key Findings Sense of Community The majority of the students felt the Nittany Lion graphic gave them a sense of inclusion in the larger Penn State community, but interaction with the instructor and classmates also contributed. 45

46. In Closing… “If all of us are committed to developing innovative programs and policies for education, we need to marshal all of our forces and deploy all methodologies that are available to us — qualitative as well as quantitative research; the best randomized controlled trials as well as the best ethnographies, case studies, and so on.” 46 (Sullivan, President of UVA, @ AERA Plenary address, 2014)

47. Questions? 47 Thanks for your time! Maggie Frederick mas960@psu.edumas960@psu.edu Amy Garbrick amyg@psu.eduamyg@psu.edu