1. Media & Learning Design (M&LD) Research & Evaluation Presentation to M&LD Steering Committee By Christos Anagiotos (cxa5065@psu.edu) & Phil Tietjen ( prt117@psu.edu ) April 4 2012

2. OUR CHARGE  Review prior approaches to evaluation within M&LD  Incorporate evaluation in new M&LD projects more systematically  Investigate state of the art in media & learning evaluation  Investigate what other Universities are doing in regards to the use of Media in online courses.

3. Evaluation elements  Learning outcomes  Learning experience  Usability

4. Research: Media enhances learning  Retention  Transfer  Cognitive Flexibility

5. Course Surveys

6.  Criminal Justice  Public Administration

7. Sample Questions  By watching the library learning tutorials, I learned things I previously did not know about the library  These tutorials will help make my research easier  Video cases allowed me to relate to the content  Video cases helped me in doing my assignments for the course

8. Focus Groups

9.  World Campus orientation videos: focused groups with students  M&LD participants: Focused groups with Instructional Designers

11. Identified Problems

12. Problem = Low Response Rate Surveys

13. Response = Embedded Evaluation Learning Analytics

18. Learning analytics is the  measurement,  collection,  analysis and  Reporting of data and their contexts, for purposes of  understanding and  optimizing learning and environments in which it occurs

19. Universities that are using Learning Analytics  University of Phoenix  Cabelas University  Baylor University  Sinclair Community College  University of Baltimore  Purdue University  Regis University (Library’s Distance Learning Department)  University of Rutgers-Newark (Law Library)  Khan Academy

20. POTENTIAL OF LEARNING ANALYTICS A. Compare users (e.g. evaluation) B. Predict student performance (Predictive Analytics) C. Understand student’s needs D. Identify media flaws E. Personalization of educational material

21. What data can we collect from current WC sources 1. ANGEL 1. Outside ANGEL - Google analytics - Flash Media Server

22. What does ANGEL offer?  All data is connected to the student (PSU ID, IP address)  Individual analytics (very complicated to get group analytics) Examples:  Log in time, Log out time, Time spend in each website, Items downloaded

23. Google Analytics (Outside ANGEL) Collect anonymous information about the user Data is connected to IP Address Data is NOT to the PSU ID  Records much more data than ANGEL  The data is presented in a more user friendly way

24. Media Flash server (M&LD Videos, Outside ANGEL) Collect anonymous information about the user Data is connected to IP Address Data is NOT to the PSU ID We can currently measure:  Log in/ log out time  Duration per visit, per visitor  Streaming duration  Play, pause hits

25. How to make sense of data collected? EXAMPLES

26. Example 1: from Media Flash Server: Course Ed. Leadership 802:  Average Length of videos : 10 minutes  Average watch time: 4 minutes

27. Example 1: Possible explanations  The content is not valuable or useful to the viewers  The user already got the info from other sources (readings, discussions etc)  Users are tired or bored after watching the same person talking for more than 4 min.  The content may not be clear enough to the user

28. Example 2 A video was watched 46 times by 12 users in 7 days. Possible Explanations:  High relevance to the user (e.g. used for an assignment)  Entertaining  Confusing

29. Comparison of data collected ANGEL: Pros: Data connected to the user PSU ID Cons: Very limited amount of data, tough to use Google Analytics (Outside ANGEL): Pros: Large amount of data & Great detail Cons: Data not connected to the individual users’ PSU ID Flash Media Server (Outside ANGEL): Pros: Decent amount of data Cons: Data for the videos ONLY Data not connected to the individual users’ PSU ID

30. Combining the data we already collect We can gather :  Data directly connected to each user (PSU ID) from the 3 sources  Group data  Data for every activity in the course website

31. OTHER FORMS OF DATA THAT WC DO NOT COLLECT 1. Social Network Analysis (Student networks) 2. Record student screens

32. 1. Social Network Analysis (Student Networks)  Students’ social networks facilitate learning processes (Dawson, 2010).  These tools are making learner networking visible  Able to “see” (identify) students who are network-poor (apply interventions)

33. Visualization of Social Networks Analysis

34. 2. Record student screens e.g. Team Viewer software

35. What’s next in Learning Analytics? Personalization of educational material Knewton - Pearsons partnership (video): (Knewton Adaptive Learning Platform).

36. Confidentiality issues  How much data we collect?  Students’ consent  Who has access to the data?

37. Recommendations  Coordinate with IDs to implement regular evaluations  Establish regular meetings with IDs to discuss and analyze results  Develop internal visualization-reporting tools  Make the connection to student performance  Publicize our findings, let people, outside PSU, know what we are doing in M&LD.

38. Some other ideas for evaluation

40. Thank You Questions?