How Well Do Online Forums Facilitate Discussion and Collaboration Among Novice Animation Programmers? Christopher Scaffidi, Aniket Dahotre, Yan Zhang School of Electrical Engineering and Computer Science Oregon State University

2 Animation programming Effective for motivating students to learn programming Increasingly used in freshman CS courses Many tools available –Logo –Alice –KidSim –AgentSheets –… Introduction  Study  Statistical results  Social network results  Discussion

3 Scratch as a particular programming tool Turing- complete language Events, loops, conditionals, sprites, sound… Drag-and-drop programming + Online community Introduction  Study  Statistical results  Social network results  Discussion

4 Massive online community 1,019,863 registered members (Feb 2012) –Post animations, discuss animations, use online forum Learn by building and reflecting upon shared objects –Using constructionism (à la Papert) to support social constructivism (à la Vygotsky) –Essential ingredients: Learners Learner collaboration to create objects Sharing of created objects Reflective discussion about the objects Introduction  Study  Statistical results  Social network results  Discussion

5 Key results from previous study Analysis of user comments about 100 posted animations –27% compliment, 27% feature suggestions/critique, 26% chit-chat off-topic, 20% replies by animation owner –0% of the comments indicated that the animations under discussion were shared creations Where is the collaboration and social support for learning?? Introduction  Study  Statistical results  Social network results  Discussion

6 Which made us start to wonder Maybe the post-animation-for-discussion model isn’t effective for anchoring collaboration –Maybe we need to think harder about what social constructionism really calls for, in this context? Or maybe the model is fine, but the implementation is just getting in the way –E.g., maybe the short, Twitter-like user interface in that part of the site isn’t conducive to forming collaboration –Maybe forums might be better; they allow more space Introduction  Study  Statistical results  Social network results  Discussion

7 Specific questions for this study Do different kinds of initiating posts lead to differences in ensuing discussions? –Does posting an animation obtain more interaction? –Does inviting collaboration obtain more interaction? –Do different topics obtain different levels of interaction? –Do all users obtain the same level of interaction? What kinds of collaboration take place via the forums? –Does interaction resemble collaboration in some sense? –If so, what does collaboration practically entail? Introduction  Study  Statistical results  Social network results  Discussion

8 % of all forum posts, categorized based on respective forum’s primary focus Word Games, 34% Show Off, 31% Introduction  Study  Statistical results  Social network results  Discussion

9 Acquiring data for analysis Focused on forums… –Where most discussions were started by users –And did not simply involve word games Which led to… –11 Show Off, Trouble, and Collab Request forums –Containing 397,477 posts in 36,886 discussions –We scraped them all Introduction  Study  Statistical results  Social network results  Discussion

10 Analysis methods—Statistical Four statistical hypotheses –Link to animation will obtain different levels of interaction –Inviting collaboration will obtain different level of interaction –Posts in different kinds of forums (Show Off / Trouble / Collab Requests) will obtain different levels of interaction –Community managers’ and ordinary users’ posts will obtain different levels of interaction Operationalize “level of interaction” with four measures of interaction –# replies, # distinct users involved, # animations posted in reply, # collaborative words used Introduction  Study  Statistical results  Social network results  Discussion

11 Analysis methods—Social network analysis, plus qualitative analysis Compute social network graph (directed graph) –One node per user –One edge indicating user responded to another user Compute statistics characterizing overall structure –Standard statistics, eg, avg in-degree, connectedness Grounded theoretical (qualitative) analysis of typical interactions –Identify 30 moderately sized clusters –Create a coding scheme for categorizing each cluster –Another researcher independently categorizes clusters with same coding scheme Introduction  Study  Statistical results  Social network results  Discussion

12 Do different kinds of initiating posts obtain differences in ensuing discussions? Link to animation led to different levels of interaction Introduction  Study  Statistical results  Social network results  Discussion

13 Do different kinds of initiating posts obtain differences in ensuing discussions? Inviting collaboration led to different levels of interaction Introduction  Study  Statistical results  Social network results  Discussion

14 Do different kinds of initiating posts obtain differences in ensuing discussions? Posts in different kinds of forums were followed by different levels of interaction Introduction  Study  Statistical results  Social network results  Discussion

15 Do different kinds of initiating posts obtain differences in ensuing discussions? Different users’ posts led to different levels of interaction Introduction  Study  Statistical results  Social network results  Discussion

16 What kinds of collaboration take place via the forums? Social network analysis of user interactions within the Collab Request forums –2074 user nodes –average in-degree only 6.27 –average of 20.5 persons per weak component –bi-directional connection to an average of 0.40 people Fragmented community, with any collaboration likely within small groups consisting of a few people Introduction  Study  Statistical results  Social network results  Discussion

17 Social network One directed edge per pair of users who ever replied to each other Introduction  Study  Statistical results  Social network results  Discussion

18 Social network Exactly same graph; different visualization –Showing only edges where ≥20% of user’s replies went to other user –Omitting arrowheads –Run Gephi layout engines Introduction  Study  Statistical results  Social network results  Discussion

19 Identifying discussions for analysis Users with in-degree approximately in the 80 th percentile –In-degree of 10 or 11 –Not dependent on the visualization Randomly select 30 discussions initiated by these users Code with grounded theoretical analysis Introduction  Study  Statistical results  Social network results  Discussion

20 Forms of interaction observed Category#Example Requests for media3 Peanut butter jelly time...I need the banana! Requests for ideas4 Need an idea? Or want to share one? Look here!... Requests for feedback 2 I am currently working on a big animation… can't dicide which ideas to use… A true story about 18 American soldiers in WW2 who fended off 500 German soldiers Requests for participation 2 This will be a weekly paper that comes in on Sunday.… I need these people for the Scratcher Times. Tips2 Does anyone know a program that compresses audio files? Forum meta- discussions 4 I'm probably going to get banned soon. Community meta- discussion 2 My new website is opening up soon and I have put a Scratch games section on the Games page. Non-Scratch topics11 It's getting pretty cold, and it's snowing nearly the whole day Introduction  Study  Statistical results  Social network results  Discussion

21 Forms of interaction observed Introduction  Study  Statistical results  Social network results  Discussion

22 Forms of interaction observed ~ 1/3 of discussions contributed to creating specific projects ~ 1/3 of discussions were Scratch-related but not project- specific ~ 1/3 of discussions were not Scratch- related at all Introduction  Study  Statistical results  Social network results  Discussion

23 Putting in perspective 12% of all forum posts were in Collab Request forums –Of these users, fewer than 20% had in-degree ≥ 10 Of these users’ discussions, ~ 1/3 contributed to a project –Requests for feedback –Requests for ideas –Requests for media –Requests for participation Interpretation: collaboration seems fairly rare (We have found the needle in the haystack, and this is what it looks like.) Introduction  Study  Statistical results  Social network results  Discussion

24 Most interesting conclusions Requests for collaboration led to highest interaction (vs. posts in forums focused on getting help or showing off) Posting a link did not lead to more replies or more users Collaboration in the forums is very rare –But when it does occur, it involves requests for ideas, feedback, media, and participation Introduction  Study  Statistical results  Social network results  Discussion

25 Possible implications for distance education Requests for help obtained the lowest level of interaction –Learners might be better able to obtain help if they explicitly request collaboration, not just information Posting an anim link did not lead to more replies or users –More effective ways are needed to structure and stimulate critical reflection and discussion Collaboration was rare –More effective ways are needed to structure and stimulate and organize collaboration among learners Introduction  Study  Statistical results  Social network results  Discussion

26 Thank you… To SIGCSE for this opportunity to present To you for your attention, questions, and potential collaboration Introduction  Study  Statistical results  Social network results  Discussion