Goren-Bar & Goori June Design and Evaluation of Web-Based Collaborative Learning Dina Goren-Bar & Tal Goori Department of Information Systems Engineering 23/6/2004

Goren-Bar & Goori June Agenda Introduction & Motivation Research Objectives TEPCEL Framework TEPCEL Implementation TEPCEL Evaluation Current Status Experiment Results Conclusion Further Research

Goren-Bar & Goori June Introduction What is Collaboration What is Collaborative Learning? What is ALN Research Justification  Why should we perform a research in the field of Collaboration via ALN?  Shift from outcomes to interactions

Goren-Bar & Goori June Research Objectives Define a framework for the design of collaborative e-Learning tasks which combines the technological, educational and process oriented approaches. Evaluation of the collaborative process generated during the implementation of a collaborative assignment as part of the HCI course.

Goren-Bar & Goori June TEPCEL Framework An acronym for Technological, Educational and Process oriented Collaborative E- Learning Framework Enables the design and evaluation of synchronous as well as asynchronous collaborative learning environments and tasks. Combines several approaches including the tools, outcome and process, document centric, and session centric approaches into one integrated framework

Goren-Bar & Goori June TEPCEL Framework - 5 stages Objectives Definition Collaborative Features Settings Collaborative Assignment Definition Collaborative Tools Definition Evaluation Design Stages Implementation & Evaluation Stages Each stage is characterized by a set of attributes that enable the design & evaluation of the collaborative learning environment and tasks.

Goren-Bar & Goori June TEPCEL Framework - Attributes Attributes SamplesStage NameStage # - The cause for the CL processObjectives Definition1 - The size of the CL group - The duration of the CL process Collaborative Features Settings 2 - The subject of the CL process - The nature of the outcome\s of the learning process Collaborative Assignment Definition 3 - Available tools and technologiesCollaborative Tools Definition 4 - Evaluation criteriaEvaluation5

Goren-Bar & Goori June TEPCEL Implementation A web collaboration asynchronous environment. An undergraduate course for Information Systems Engineering students. The research was conducted in two students groups during years

Goren-Bar & Goori June TEPCEL Implementation Year 2003Year 2002 (pilot) 80150# of students 915# of groups 22# of collaborative assignments # of participants in each group  Each group received the same assignment structure with slightly different content.  Each group was further divided up into dyads.  Each subtask in each group was performed by two students (dyads).  The students were unable to control their collaborative group belonging.

Goren-Bar & Goori June TEPCEL Implementation Experimental Settings  Each asynchronous private group workspace (within the course website) was compounded of: A list of all group members A list of five sub-tasks Collaborative Assignment description A threaded asynchronous communication Files Storage Group capabilities An automatic mechanism that informs the group members of a file upload operation performed by one of the dyads requesting for feedback.

Goren-Bar & Goori June TEPCEL Implementation

Goren-Bar & Goori June TEPCEL Evaluation - Tools Feedback Questionnaire  To identify the level of collaboration within the groups and the successes of the collaboration process.  The questionnaire referred to the collaboration assignments and aimed to identify student 's self - preference towards collaboration

Goren-Bar & Goori June Current Status The 1 st research results were used as a pilot study The main conclusions from our Pilot Study were (Goren-Bar & Goori, 2004) :  Feedback on uploaded files increased significantly from 2.6 to 6.5 (P<.05)  Students learned from experience, identified the main factors that influence successful collaboration and acted accordingly Study on 2003 focused on the collaboration process (tested by a new feedback questionnaire) Evaluation of the 2 nd study in progress

Goren-Bar & Goori June Feedback Questionnaire – Q1 Grading Collaboration Methods The students graded 5 collaborations methods (from 1 least preferred to 5 most preferred) to work with: Legend 1 – CL within ALN with 6-10 students 2 - F2F CL with 6-10 students 3 - Work alone 4 - F2F CL with 4 students at most 5 – CL within ALN with max 4 students Anova: df=4, F= , P=0.0928

Goren-Bar & Goori June Feedback Questionnaire – Q2 Collaboration Disadvantages Number of selections Legend 1.Unequal workload division between group members 2.My influence on the final product is less than when I work alone 3.The coordination between group members requires too much effort 4.The noninvolvement of some group members damages the quality of the work 5.Group members who think different than me lead to undesired directions 6.Group collaboration does not promote individual's unique ideas and skills Most of the students felt that the workload division between the group members was unequal (68%= 34/50) X 2= , df=4, P=

Goren-Bar & Goori June Feedback Questionnaire – Q3 Collaboration Advantages Legend 1 - Group collaboration enables equal workload division between group members. 2 - The quality of deliverables created during group collaboration is better than those when working alone. 3 - Group collaboration enhances the variety of proposed ideas and solutions. 4 - Group collaboration reinforces the relations between group members. 5 - Discussing new ideas with other group members enables me to improve my knowledge. 6 - Group collaboration incites students' interest in the learning materials. Group collaboration enhances brainstorming (3= the variety of proposed ideas and solutions 66% and 5= discussing new ideas & issues with other group members enables me to improve my knowledge 62%) Interesting: option 1 which states that Group collaboration enables equal workload division between group members is an advantage and disadvantage! (question 2 – 34 students) X 2 = , df=4, P=0.000

Goren-Bar & Goori June Option 11112MinAssignment Max Average STDEV 11111MinAssignment Max Average STDEV P(t)TTEST* Legend 1 - "I am pleased with the task output I have delivered with my partner" 2 - "I am pleased with the final output my group delivered" 3 - "I have gained new knowledge as a result of the collaborative work" 4 - "The delivered presentation at the end of the assignment reflected the level of invested effort" 5 - "I would have gained better results had I performed the whole assignment by myself" * Probability associated with a Student's paired t-Test, with a one-tailed distribution Feedback Questionnaire – Q4&5

Goren-Bar & Goori June The T - Test results indicate that in assignment 1 the students were more pleased with the task output they have delivered with their partner (P(t) = ). Students were less pleased with the final output their group delivered (P(t) = 0.01) in assignment 2. The implementation of assignment II was less successful due to bad timing (last weeks of the semester). Feedback Questionnaire – Q4&5

Goren-Bar & Goori June * Probability associated with a Student's paired t-Test, with a one-tailed distribution Feedback Questionnaire – Q6&7 In questions 6 & 7 students were asked to provide the number of group's members who did not collaborate or their level of involvement was low. Only in group 5 in assignment 2 we can see a significant reduction in the number of uninvolved students (P(t) = ). This stands in contrast to our hypothesis that the level of collaboration will improve from the first CL assignment to the second.

Goren-Bar & Goori June Feedback Questionnaire – Q Group ID Total Average STDEV 53.45%91.60%115.78%105.85%56.29%91.62%109.42%83.42%Nor STDEV Average*150% # Of stud. 50% Over the Avg Average*50% # Of stud. 50% under the Avg %40.00%75.00%50.00%20.00%70.00% 30.00% Percent of students who were beyond\un der 50% average

Goren-Bar & Goori June The level of collaboration improves when the group results are homogeneous (meaning we strive for a low percentage). When the percentage is high it indicates that the group was not homogenous from the collaboration perspective, the effort was carried out by a small number of students within the group and most of the students were pretty much not involved in the process. We have defined the threshold for measuring homogenous group as 30%. Group 4 performed good collaboration (20%) and also groups 1 and 9 implemented good collaboration where most of the group members were involved in the process (30%, 33% respectively) Feedback Questionnaire – Q8

Goren-Bar & Goori June Conclusions Most students resented from working in big groups of 6-10 participants. Most students would prefer to:  Work alone as this is the traditional working method.  Collaborate F2F or within the web medium but in smaller groups of 4 participants at most. Most of the students felt that the workload division between the group members was unequal - “how can we balance the workload” This aspect will be handled in the framework Almost all the students were pleased with the task output they have delivered with the partner - We assume that students were feeling uncomfortable with the thought of "complaining" about their best friend.

Goren-Bar & Goori June Conclusions In order to identify the level of collaboration based of students' credit we have defined three indicators.  "collaboration champions" presents the students who were 50% beyond the group average.  "non collaborative" which represented the number of students who were 50% under the group average  The Normalize Standard Deviation presents the variance of the feedbacks the students received based on the collaborative characteristics. I.e. a high value of N. STD states that the number of students who collaborated was low. A low value of N. STD states that the more students collaborated.

Goren-Bar & Goori June Further Research System Usability Scale (SUS) Threaded asynchronous communication – Content Analysis based on:  Types of messages  Number of messages Threaded asynchronous communication - Social Network Analysis  To analyze the collaboration interactions within the collaborative environment.  Participation distribution within group’s pairs  Distribution of initiator or replier within the type of messages and group members