1. Special Course in Software Techniques Directions for Future Research on Algorithm Visualization Aalto University June 7-23, 2011 Clifford A. Shaffer Department of Computer Science Virginia Tech

2. What makes for a good AV? "Good AVs are all alike; every failed AV fails in its own way." -- Leo Tolstoy 2

3. “State of the Field” Rationale Many faculty wish to use AVs Hundreds of visualizations are freely available on the Internet Studies on the effectiveness of AVs –Many studies show no significant difference –But AVs have been shown to help in some implementations –One conclusion is that creating/using effective AVs is possible but not easy

4. Research Questions What AVs are available? How do educators find AVs? Who makes them? Is there adequate coverage? What is their quality? Is the field active, and improving? Is there adequate infrastructure for storing and disseminating AVs?

5. What AVs are Available? A collection of links available at http://algoviz.org Links to over 500 visualizations Nearly all AVs now written in Java –Applets vs. applications Stand-alone vs. collections

6. Who Makes Them? Single authors, one-off implementations (1-5) –30% Small shops, sustained over a few years –Typically a faculty member and a few students –5-10 visualizations –10% Larger teams, longer term investment –Team built, maybe funded –25% Major Projects –integrated package or shared look-and-feel –35%

7. Is There Adequate Coverage? No –Sorting, search trees, and linear structures overwhelmingly dominate –Coverage for more advanced topics is spotty

8. What Is Their Quality? A majority have no pedagogical value –These give the user no understanding of how the data structure or algorithm works –Will be of little use in the classroom We would recommend less than one quarter of what we have seen for any purpose Even the better visualizations usually have serious deficiencies –Animation only: Users are passive observers –Tree structure visualizations tend to show what happens, but not how –Limited interactivity

9. Is the Field Improving? Pros: –A growing body of literature on best practices to create effective AVs –Community starting to organize (AlgoViz) Cons: –Recent projects are no more in tune with coverage gaps than old projects –No apparent movement in creating repositories

10. Is the Field Active? Appears to be a reduction in “one-off” development. (Drop in student projects) –Fewer CS students –Less interest in Java –But these trends might reverse But steady activity in the larger groups.

11. AVs: The Problem AVs have high faculty and student favorability ratings But most faculty don’t use them much in courses

12. Informal Survey Results Warning: Self-selected responders Are AVs useful? –Strongly Agree: 12 –Agree: 17 –Neutral: 1 A (bare) majority indicated that they used some sort of visualization with class

13. Survey: Impediments to Use Lack of knowledge/time to find good AVs: 13

14. Survey: Impediments to Use Lack of knowledge/time to find good AVs: 13 Time to make good AVs: 2 Difficulty integrating in class: 9 Lack of time within class constraints: 2 Uncertainty about quality outcomes: 1 Content not relevant to my classes: 1

15. Overcoming Impediments Reassurance about what AVs are good Ideas on how to use AVs Reassurance about how a given AV can be used successfully in class Ability to connect to developers

16. AVs: The Solution is Community http://algoviz.org/ –Build a community of users/developers –Better disseminate best practices information Project Support –NSF CCLI grant –NSF NSDL grant –Connections to NSDL/Ensemble project

17. AlgoViz.org A collection of links to over 500 AVs Annotated bibliography of over 500 research papers Forums, field reports OpenAlgoViz

18. Phase I: The Passive Side What problem do we solve here? –Information? We provide that (to a limited degree) Nearly done!

19. AlgoViz Usage Statistics

20. Phase II: The Active Side Key problem: How to help instructors through the impediments to using AVs We identified community as a solution How do we build an online educational community?

21. New Paradigm for Digital Library Create a new model of “dissemination” to lower barriers to access Move away from the “digital library” model of users coming to collections Focus on “community-driven” content development –Discussion, review, ratings –Think Amazon, but we have critical mass issues

22. Community Value Added Catalog entry ratings Catalog entry reviews Field reports Forum discussions Interactions with developers

23. Community Building: AlgoViz Awards Exemplars of good AVs Solicited votes from the public on a slate of nominees 2010: Six winners Hall of Fame: 6+1 –Sorting out Sorting 2011: Two winners

24. Notification Connect to community by informing members of site changes –News posts, –New AVs, bibliography, field reports, forum posts Email, FaceBook, Twitter, RSS feeds Fine grained User control of information stream –User profiles

25. Registration and Access (1) Site registration requirements deter users from participating Link spammers and other malicious entities can overwhelm a site So we need a balance –Minimum registration requirements possible Goals: –Link together a given user’s site activities –Be able to block a user

26. Registration and Access (2) Solutions: –Allow unrestricted access to view –Minimize registration burden –FaceBook connect, etc. –Initially moderate, then declare user to be “trusted” Social Capital –Ranking, badging

27. Lessons Learned Building site infrastructure and collection is relatively easy –Developers are motivated to provide information Building community infrastructure is relatively easy Building community is hard

28. Are AVs of Pedagogical Value? Instructors generally think so Students usually say they “like” them 28

29. Metastudy: 2002 Reviewed 24 prior studies on pedagogical effectiveness related to AVs –Generally of an individual system or AV Results of 24 studies: –11 found significant (positive) results –10 did not find a significant result –2 entangled prediction with visualization –1 study found a negative result! 29

30. Classification The studies represented a wide range of activities and methods Looking deeper, reclassify the independent variables: –Epistemic Fidelity: 10 –Cognitive Constructivism: 14 –(others too few to measure) CC has the highest percentage of positive studies 30

31. Results CC: 71% statistically signficant EF: 30% statistically significant 31

32. CC Activities Construct own input sets Make predictions about future states Program the algorithm Answer questions about the algorithm Construct own visualization 32

33. Level of Effort Compared when the two treatments required similar “cognitive effort” vs. different levels of effort –Equivalent effort: 33% significant –Not equivalent: 71% significant Viz vs. reading, exercises maybe on either side Construct algorithm/visualization takes time? Time vs. effort vs. engagement considerations? 33

34. Procedural vs. Conceptual Knowledge Procedural only: 67% [10/15] Procedural and Conceptual: 67% [2/3] Conceptual only: 38% [3/8] 34

35. Study Measures Post-test only: 54% Pre- to Post-test difference: 78% –But most of these studies came from one source 35

36. Study Conclusions How students use AV is more important than what they see Pre-test/post-test experiments on procedural knowledge show most improvement Technology is effective when it is used for active engagment 36

37. Engagement Taxonomy Naps Working Group 2002 –No viewing –Viewing –Responding –Changing –Constructing –Presenting Relates to Bloom’s Taxonomy 37

38. Extended Engagement Taxonomy Myller, et al. –No viewing* (textbook) –Viewing* (video) –Controlled Viewing (slideshow) –Entering Input (Define the input to execute) –Responding* (answer questions) –Changing* (direct manipulation) –Modifying (??) –Constructing* (create the AV) –Presenting* (Teach the material) –Reviewing (Give a review of AV) 38

39. List of Features From Saraiya, et al. (2004) Ease of use (one time only) Appropriate feedback State changes made explicit Multiple views User control Canned examples vs. data input Pseudocode

40. 2009 Evaluation Urquiza-Fuentes/Velazquez-Iturbide 33 successful evaluations Evaluation: –Usability –Learning outcomes Many studies compared Viewing, Changing, or Constructing vs. Non-Viewing A few compared Changing or Constructing vs. Viewing Learning improvements in 75% of studies 40

41. What is a Hypertextbook? It means different things to different people We mean a close integration of three aspects: –Static text and graphics –Dynamic content: animated slideshows, interactive activities and visualizations, simulations, etc. –Assessment activities: questions, proficiency exercises, programming exercises, etc., with immediate feedback on progress If it does not have lots of these three things working together, then we do not consider it to be a hypertextbook.

42. Why Hypertextbooks? Improve exposition –Take advantage of the computer to depict dynamic action Increase engagement –Take advantage of the computer to support interactive learning with feedback

43. Data Structures and Algorithms In particular, Data Structures and Algorithms topics are good candidates for hypertextbook treatment –Students have trouble “getting” this material because it is inherently dynamic, but often portrayed with static methods. 43

44. What’s new? In one sense, nothing –People have wanted this for decades, and often in much the same terms –But they haven’t made it yet We have some advantages –Past experience –Technology: right place at right time –A plan for getting the necessary resources

45. Won’t this Fail for the Same Reason that AVs Fail? AVs have been something of a disappointment –AVs are everywhere these days –Instructors and students “like” them –But instructors do not use them so much Impediments –Lack of knowledge about good AVs –Hard to get them integrated into classes

46. Hypertextbooks vs. AVs Conclusion: It is easier to put a full unit into your course than to “fit in” a new form of pedagogy The problems of AVs can be overcome by: –A well-known set of resources –Complete resources, ready to plug into a course –Either complete topics or whole courses –Mechanisms to pick an choose what you want, and to make modifications if necessary

47. From Marc Brown’s Dissertation Much of the success of the BALSA system at Brown is due to the tight integration of its development with the development of a textbook and curriculum for a particular course. BALSA was more than a resource for that course – the course was rendered in software in the BALSA system.

48. Prior Work (1) Content efforts: –Ginder et al (2002): Theory of computation –Rodger: JFLAP software supplements standard textbook –Crescenzi, et al (2007): courseware, tied to a textbook –JHAVE, TRAKLA2, Animal all include tutorials for various topics –Titterton, et al (2010): Lab-centric courseware for CS1 –Alharbi et al. (2010): Operating Systems interactivities and questions –Kim and Jung (2010): Korea eTextbooks

49. Prior Work (2) Definition and Infrastructure Ross (2008): Perl/Dreamweaver: a few chapters Karavirta (2009): Xaal Roling and Vellaramkalayil (2009): AVs and Moodle Rossling et al (2006): ITiCSE Working Group on VisCoSH

50. Case Study Virginia Tech Hashing Tutorialhttp://research.cs.vt.edu/AVresearch/hashing In each of two semesters –One section with lecture/textbook –One section with “closed lab” tutorial ANOVA shows significant difference between treatment groups, in favor of the online tutorial This gives some evidence that a hypertextbook can work, pedagogically

51. Technology A potential objection to our project is the fact that prior efforts haven’t succeeded due to technical obstacles One advantage now is that technology has advanced in significant ways –Ubiquitous access to computing: laptops, mobile devices, Internet, classrooms –HTML5

52. Project Vision Create a hypertextbook –text/graphics –dynamic/interactive exposition –assessment with feedback Complete materials for Data Structures and Algorithms courses Creative Commons: Pick and Choose Broad contribution from AV community

53. Levels of Interaction Text and Graphics “Animated slideshow” –for dynamic exposition Interactive proficiency exercises –TRAKLA inspired Interactive simulations/explorations –From probing equations to running performance simulations Assessment questions –Keep student engaged and on track

54. Creating the Hypertextbook We imagine a multistage process 1.Get an initial body of text/graphics up (this is easy since we have a textbook) 2.Build the Storyboard 3.Convert the Storyboard to the actual hypertextbook

55. The Storyboard The storyboard carefully details every point where an AV or other interactive/dynamic piece is needed, along with every assessment activity Begin by augmenting basic text with text specifications Over time, augment these specifications with existing AVs, mockups, whatever works Gradually replace the mockups with real implementations The Storyboard is a management tool, recruiting tool, and progress guide

56. Implementation Considerations for Dynamic and Interactive Content Animated slideshows could be implemented using many standard packages –PowerPoint, Beamer, OO Impress, KeyNote None are portable Conversion to PDF doesn’t really work for animation Flash and Java need plugins –Which leaves out iPad, mobiles HTML5/JavaScript appears to solve all of our technical problems

57. Implementation Considerations for Creative Commons Connexions (http://cnx.org)http://cnx.org LeMill (http://lemill.net)http://lemill.net Both are large creative commons for educational resources with infrastructure for authoring and reuse of material We have plenty of text!

58. Assessment Activities Questions –Multiple choice, short answer, etc –Could be occasional, could be at end of section TRAKLA-style proficiency exercises VILLE-style questions –Write a function –Trace a program How to assess correctness?

59. Implementation Considerations for Assessment This is going to be hard! What we need: –Storing, selecting, retrieving, and delivering questions –Processing student answers –Storing, organizing by class, and retrieving student assessment results –Making decisions about progress

60. Management Considerations Will need a lot of contributors Will need review mechanisms for contributed material Will need a process for handling derived/modified works

61. AV Development Library A nice technical issue ;) A set of functions to help developers build AVs Its existence will help consistency in look- and-feel