1. Canvas User Experience (UX) Studies Amy Garbrick: Director of Learning Design (College of IST) Andrea Gregg: Manager of Instructional Design (World Campus) Ronda Reid: Instructional Designer (College of IST) Vicki Williams: Manager, Assessment & Evaluation Research (ITS)

2. Why it’s So Important to Talk to Learners “[A]sking students about their experiences of e-learning gives surprising, individual, and frequently contradictory results and this in itself is good reason for listening to students more often and more thoroughly if we are to avoid making assumptions about their experience from our teacher-centred view.” 2 (Sharpe & Bennefield, 2005)

3. Recent Research WC LMS Requirements (2013, qual.) What are World Campus requirements for the LMS? 2 Int. & 4 focus groups - w/ WC students (12), faculty (13), staff (12) Identified requirements (e.g. mobile, easy, discussion, social presence) Report presented to TLT, Bb, WC; Used in LMS pilots Online Student Interface Study (2015, quant.) What are student perceptions in navigating online courses? Sub-committee of the DMT; OAR survey; 387 respondents Mostly +; Critiques: locations for due dates, assignments Student Interface Task Force using report to make recommendations Socient Study (2015, quant.) What improvements needed for outstanding student exp.? Socient survey: 1649 respondents Mostly positive w/ areas for improvement (e.g. career services, advising) LD improvements: usability, tech consistency, instructor involve. 3

4. What We’ll Cover Today  College of IST Canvas UX Research Study and Results  Cross-Unit UX Study  User Survey 4

5. Where We Started…  We wanted to find out how students approached the IST course design in Canvas  Came up with a test  8 tasks  Rate tasks from 1 (easy) to 5 (difficult) 5

6.  (Task 1) You have a question on the Lesson 07 Web Usability essay assignment. How would you find out more about this assignment?  (Task 2) Your instructor sent an email reminding you to complete the Lesson 03 readings. How would you locate your reading assignments?  (Task 3) You would like to receive a weekly email notification regarding course announcements. How would you set-up this notification? What We Asked…Basic Questions You’d Find in Any Course 6

7.  (Task 4) You remember you have the Lesson 09 quiz this week, but need to check the submission due date. How would you find out when your quiz is due?  (Task 5) You would like to get an idea regarding what assignments and work you have to do next week in the course. Where would you look to check your upcoming workload and assignments? What We Asked…Basic Questions You’d Find in Any Course 7

8.  (Task 6) You have a question regarding how to participate in the Lesson 10: DA Human Multitasking discussion forum. Where would you go to find out more information on how to participate?  (Task 7) You want to see if your Lesson 01 quiz was graded. How would you go about checking your grade?  (Task 8) You want to contact your team regarding your team project. Where would you go to do this? What We Asked…Basic Questions You’d Find in Any Course 8

9. Bring in the Guiney Pigs Image: Megan van der Elst (http://jezebel.com/hipster-guinea-pig-professor-boo-boo-was-cute-before-it-1595022520)Megan van der Elsthttp://jezebel.com/hipster-guinea-pig-professor-boo-boo-was-cute-before-it-1595022520 9

10. Canvas UX Study: IST World Campus Students  Recruited undergrad IST World Campus students  Willing to come to University Park for the study  Gave them a $50 gift certificate to Amazon.com  4 male, 1 female  1 student: aged 20-23 1 student: aged 24-29 2 students: aged 30-39 1 student aged 40-49 10

11. What Did We Find? One Word… Easy 11 Image: Staples.com

12. So, What Did Students Say? 12 “Easy to find.” ~Subject 5 (Task 4) “Once I understood what modules were it was fairly easy to find.” ~Subject 4 (Task 1) “Not too bad, easy once you get familiar” ~ Subject 1 (Task 2)

13. “Very straight forward.” ~ Subject 2 (Task 7) So, What Did Students Say? 13 “Very easy to find.” ~Subject 4 (Task 5) “Easy to find.” ~Subject 5 (Task 8) “Easy to locate information.” ~Subject 3 (Task 6)

14. Summary of Scores  Scoring system: 1 (easy) – 5 (difficult)  5 of 8 tasks scored <2  1.6, 1.6, 1.2, 1.2, 1  3 of 8 tasks scored in the 2 range  2.4, 2.4, 2.2 14

15. So, What Did Students Say? Modules, Modules, Modules! http://mgoblog.com/mgoboard/dantonio-elicits-sympathy-jamie-samuelson-detroit-free-press 15

16. Modules, Modules, Modules! 16 “Modules are the primary source of information regarding courses, understanding that is paramount.” ~Subject 1 “Pretty much everything is under the modules. That's typically where I would go.“ ~Subject 2

17. Modules, Modules, Modules! 17 “Well, I've learned that most everything is under modules.“ ~Subject 4 "I'm going to go down to the modules since that is where everything is located.“ ~Subject 5

18. So, What Did Students Say? Canvas vs. ANGEL 18 “I think this is a big improvement over ANGEL.” ~Subject 1 “That is really simple to find and I find that a lot easier than the current method of ANGEL…” ~Subject 4 “That's a lot nicer than the current system.” ~Subject 5

19. Bottom Line: Educate on the LMS  Prioritize educating students on the LMS 19

20. Suggested: Educate on the LMS 20

21. Suggested: Educate on the LMS 21

22. Other Efforts  Other efforts…  Cross-Unit Canvas UX Study  User Survey 22

23. Cross-Unit Canvas UX Study 23 CROSS UNIT IST WCLD EMS LIBERAL ARTS TLT The combination of prior studies with the Canvas transition has provided opportunities for more collaborative research and efforts to “get it right.”

24. IST, EMS, Liberal Arts, WCLD, w/ TLT  Maggie Frederick (EMS); Jennifer Gray & Tracy Wallace (Liberal Arts); Mike Brooks (WCLD)  Each shop recruited 5 WC learners; each shop using own course design (14 new participants + 5 from IST = 19 in total)  Week of Spring Break  Talk-aloud protocol 24 Cross-Unit Canvas UX Study

25. IST, EMS, Liberal Arts, WCLD, w/ TLT  Completed set of tasks, used talk-aloud protocol  Audio-, video-, and screen-recorded  Gave them a $50 gift certificate to Amazon.com  15 cards were funded by ITS – Thanks to Terry O’Heron and Jennifer Sparrow! 25 Cross-Unit Canvas UX Study

26.  Currently undergoing data analysis on audio, video, and transcripts.  But, anecdotally, found a lot of the same themes we already covered.  Modules  Liking Canvas more than ANGEL  Confusion with Canvas terminology 26

27. IST, EMS, Liberal Arts, WCLD, w/ TLT Research interest: What are learners preferences across different shop designs? Method: Mixed Methods, including a survey where students compare and evaluate screenshots across designs Informs both design decisions & future research 27 Canvas User Survey

28. 28 Canvas User Survey Survey: Survey developed initially in collaboration with IST faculty Jess Kropczynski to identify user behaviors and preferences in Canvas screen design Modified by cross-unit team for this round of data collection

29. 29 Canvas User Survey Participants: Sent to almost 5000 students whose home campus is WC and who are taking at least one SP16 Canvas course Landing page preferences (e.g. lion, recent activity, syllabus) Syllabus organization preferences (e.g. full, compressed, separate schedule) Navigation menu preferences (e.g. concise, full)

30. 30

31. 31

32. 32

33. Mixed Methods Analysis Quantitative  Descriptive and tests for significance  Reliability Qualitative  Open-ended survey responses analyzed in NVivo for themes & patterns in responses  Produce Word clouds / trees, etc. 33 Canvas User Survey

34. Canvas User Survey: Key Findings 34

35. Canvas User Survey: Key Findings 35

36. Canvas User Survey: Key Findings 36

37. In Closing… “If all of us are committed to developing innovative programs and policies for education, we need to marshal all of our forces and deploy all methodologies that are available to us — qualitative as well as quantitative research; the best randomized controlled trials as well as the best ethnographies, case studies, and so on.” 37 (Sullivan, President of UVA, @ AERA Plenary address, 2014)

38. Questions? 38  Thanks for your time!