1. Goren-Bar & Goori June 20041 Design and Evaluation of Web-Based Collaborative Learning Dina Goren-Bar & Tal Goori Department of Information Systems Engineering 23/6/2004

2. Goren-Bar & Goori June 20042 Agenda Introduction & Motivation Research Objectives TEPCEL Framework TEPCEL Implementation TEPCEL Evaluation Current Status Experiment Results Conclusion Further Research

3. Goren-Bar & Goori June 20043 Introduction What is Collaboration What is Collaborative Learning? What is ALN Research Justification  Why should we perform a research in the field of Collaboration via ALN?  Shift from outcomes to interactions

4. Goren-Bar & Goori June 20044 Research Objectives Define a framework for the design of collaborative e-Learning tasks which combines the technological, educational and process oriented approaches. Evaluation of the collaborative process generated during the implementation of a collaborative assignment as part of the HCI course.

5. Goren-Bar & Goori June 20045 TEPCEL Framework An acronym for Technological, Educational and Process oriented Collaborative E- Learning Framework Enables the design and evaluation of synchronous as well as asynchronous collaborative learning environments and tasks. Combines several approaches including the tools, outcome and process, document centric, and session centric approaches into one integrated framework

6. Goren-Bar & Goori June 20046 TEPCEL Framework - 5 stages Objectives Definition Collaborative Features Settings Collaborative Assignment Definition Collaborative Tools Definition Evaluation Design Stages Implementation & Evaluation Stages Each stage is characterized by a set of attributes that enable the design & evaluation of the collaborative learning environment and tasks.

7. Goren-Bar & Goori June 20047 TEPCEL Framework - Attributes Attributes SamplesStage NameStage # - The cause for the CL processObjectives Definition1 - The size of the CL group - The duration of the CL process Collaborative Features Settings 2 - The subject of the CL process - The nature of the outcome\s of the learning process Collaborative Assignment Definition 3 - Available tools and technologiesCollaborative Tools Definition 4 - Evaluation criteriaEvaluation5

8. Goren-Bar & Goori June 20048 TEPCEL Implementation A web collaboration asynchronous environment. An undergraduate course for Information Systems Engineering students. The research was conducted in two students groups during years 2002- 2003.

9. Goren-Bar & Goori June 20049 TEPCEL Implementation Year 2003Year 2002 (pilot) 80150# of students 915# of groups 22# of collaborative assignments 6-1010# of participants in each group  Each group received the same assignment structure with slightly different content.  Each group was further divided up into dyads.  Each subtask in each group was performed by two students (dyads).  The students were unable to control their collaborative group belonging.

10. Goren-Bar & Goori June 200410 TEPCEL Implementation Experimental Settings  Each asynchronous private group workspace (within the course website) was compounded of: A list of all group members A list of five sub-tasks Collaborative Assignment description A threaded asynchronous communication Files Storage Group email capabilities An automatic email mechanism that informs the group members of a file upload operation performed by one of the dyads requesting for feedback.

11. Goren-Bar & Goori June 200411 TEPCEL Implementation

12. Goren-Bar & Goori June 200412 TEPCEL Evaluation - Tools Feedback Questionnaire  To identify the level of collaboration within the groups and the successes of the collaboration process.  The questionnaire referred to the collaboration assignments and aimed to identify student 's self - preference towards collaboration

13. Goren-Bar & Goori June 200413 Current Status The 1 st research results were used as a pilot study The main conclusions from our Pilot Study were (Goren-Bar & Goori, 2004) :  Feedback on uploaded files increased significantly from 2.6 to 6.5 (P<.05)  Students learned from experience, identified the main factors that influence successful collaboration and acted accordingly Study on 2003 focused on the collaboration process (tested by a new feedback questionnaire) Evaluation of the 2 nd study in progress

14. Goren-Bar & Goori June 200414 Feedback Questionnaire – Q1 Grading Collaboration Methods The students graded 5 collaborations methods (from 1 least preferred to 5 most preferred) to work with: Legend 1 – CL within ALN with 6-10 students 2 - F2F CL with 6-10 students 3 - Work alone 4 - F2F CL with 4 students at most 5 – CL within ALN with max 4 students Anova: df=4, F=2.016388, P=0.0928

15. Goren-Bar & Goori June 200415 Feedback Questionnaire – Q2 Collaboration Disadvantages Number of selections Legend 1.Unequal workload division between group members 2.My influence on the final product is less than when I work alone 3.The coordination between group members requires too much effort 4.The noninvolvement of some group members damages the quality of the work 5.Group members who think different than me lead to undesired directions 6.Group collaboration does not promote individual's unique ideas and skills Most of the students felt that the workload division between the group members was unequal (68%= 34/50) X 2= 22.61427, df=4, P=0.000151

16. Goren-Bar & Goori June 200416 Feedback Questionnaire – Q3 Collaboration Advantages Legend 1 - Group collaboration enables equal workload division between group members. 2 - The quality of deliverables created during group collaboration is better than those when working alone. 3 - Group collaboration enhances the variety of proposed ideas and solutions. 4 - Group collaboration reinforces the relations between group members. 5 - Discussing new ideas with other group members enables me to improve my knowledge. 6 - Group collaboration incites students' interest in the learning materials. Group collaboration enhances brainstorming (3= the variety of proposed ideas and solutions 66% and 5= discussing new ideas & issues with other group members enables me to improve my knowledge 62%) Interesting: option 1 which states that Group collaboration enables equal workload division between group members is an advantage and disadvantage! (question 2 – 34 students) X 2 = 34.344, df=4, P=0.000

17. Goren-Bar & Goori June 200417 54321Option 11112MinAssignment 1 33333Max 1.922.222.182.622.88Average 0.7240.8150.8490.6350.328STDEV 11111MinAssignment 2 33333Max 1.922.22.122.322.7Average 0.7240.6390.8240.6530.544STDEV 0.50.4470.3390.0100.0137P(t)TTEST* Legend 1 - "I am pleased with the task output I have delivered with my partner" 2 - "I am pleased with the final output my group delivered" 3 - "I have gained new knowledge as a result of the collaborative work" 4 - "The delivered presentation at the end of the assignment reflected the level of invested effort" 5 - "I would have gained better results had I performed the whole assignment by myself" * Probability associated with a Student's paired t-Test, with a one-tailed distribution Feedback Questionnaire – Q4&5

18. Goren-Bar & Goori June 200418 The T - Test results indicate that in assignment 1 the students were more pleased with the task output they have delivered with their partner (P(t) = 0.0137). Students were less pleased with the final output their group delivered (P(t) = 0.01) in assignment 2. The implementation of assignment II was less successful due to bad timing (last weeks of the semester). Feedback Questionnaire – Q4&5

19. Goren-Bar & Goori June 200419 * Probability associated with a Student's paired t-Test, with a one-tailed distribution Feedback Questionnaire – Q6&7 In questions 6 & 7 students were asked to provide the number of group's members who did not collaborate or their level of involvement was low. Only in group 5 in assignment 2 we can see a significant reduction in the number of uninvolved students (P(t) = 0.0477). This stands in contrast to our hypothesis that the level of collaboration will improve from the first CL assignment to the second.

20. Goren-Bar & Goori June 200420 Feedback Questionnaire – Q8 98654321Group ID 2135183153301917Total 3.5 2.253.15.331.91.7Average 1.8713.2062.6053.2812.9832.752.0791.418STDEV 53.45%91.60%115.78%105.85%56.29%91.62%109.42%83.42%Nor STDEV 5.25 3.3754.657.954.52.852.55Average*150% 11221332 # Of stud. 50% Over the Avg. 1.75 1.1251.552.651.50.950.85Average*50% 13431441 # Of stud. 50% under the Avg. 33.33%40.00%75.00%50.00%20.00%70.00% 30.00% Percent of students who were beyond\un der 50% average

21. Goren-Bar & Goori June 200421 The level of collaboration improves when the group results are homogeneous (meaning we strive for a low percentage). When the percentage is high it indicates that the group was not homogenous from the collaboration perspective, the effort was carried out by a small number of students within the group and most of the students were pretty much not involved in the process. We have defined the threshold for measuring homogenous group as 30%. Group 4 performed good collaboration (20%) and also groups 1 and 9 implemented good collaboration where most of the group members were involved in the process (30%, 33% respectively) Feedback Questionnaire – Q8

22. Goren-Bar & Goori June 200422 Conclusions Most students resented from working in big groups of 6-10 participants. Most students would prefer to:  Work alone as this is the traditional working method.  Collaborate F2F or within the web medium but in smaller groups of 4 participants at most. Most of the students felt that the workload division between the group members was unequal - “how can we balance the workload” This aspect will be handled in the framework Almost all the students were pleased with the task output they have delivered with the partner - We assume that students were feeling uncomfortable with the thought of "complaining" about their best friend.

23. Goren-Bar & Goori June 200423 Conclusions In order to identify the level of collaboration based of students' credit we have defined three indicators.  "collaboration champions" presents the students who were 50% beyond the group average.  "non collaborative" which represented the number of students who were 50% under the group average  The Normalize Standard Deviation presents the variance of the feedbacks the students received based on the collaborative characteristics. I.e. a high value of N. STD states that the number of students who collaborated was low. A low value of N. STD states that the more students collaborated.

24. Goren-Bar & Goori June 200424 Further Research System Usability Scale (SUS) Threaded asynchronous communication – Content Analysis based on:  Types of messages  Number of messages Threaded asynchronous communication - Social Network Analysis  To analyze the collaboration interactions within the collaborative environment.  Participation distribution within group’s pairs  Distribution of initiator or replier within the type of messages and group members