1. Bieber et al., NJIT ©2003 1 Using PLA to Liberate Learning (PLA: participatory learning approach) Michael Bieber, Jia Shen, Dezhi Wu, Vikas Achhpiliya Information Systems Department College of Computing Sciences New Jersey Institute of Technology http://web.njit.edu/~bieber November 2003

2. Bieber et al., NJIT ©2003 2 Outline Motivation PLA: Participatory Learning Approach A bit of theory Experimental results Interesting issues

3. Bieber et al., NJIT ©2003 3 Motivation To increase learning of course content Learning through active engagement –involve students as active participants –with the full problem life-cycle –through peer evaluation Minimize overhead for instructors

4. Bieber et al., NJIT ©2003 4 Outline Motivation PLA: Participatory Learning Approach A bit of theory Experimental results Interesting issues

5. Bieber et al., NJIT ©2003 5 PLA Process Each student creates 2 exam problems Instructor edits the problems if necessary Each student solves 2 problems Students evaluate (grade) the solutions to the problems they authored, writing detailed justifications Ph.D. students evaluate each problem a second time Instructor gives a final grade optional: Students can dispute their solution’s grade, by evaluating it themselves and writing detailed justifications Instructor resolves the dispute All entries posted on-line

6. Bieber et al., NJIT ©2003 6

7. 7 Exam Process Control Assign ID Edit questions Assign who answers questions Assign level-2 graders Course Design Determine Final Grades Set up on-line environment Dispute final grade Level-1 and Level-2 graders grade solutions Make up problems Read - other problems - other solutions - grade justifications - disputes Solve problems Instructor Control ProcessStudent Learning Process Resolve Disputes Process Flow: Learning from doing the PLA activities additional learning from reading everything peers write

8. Bieber et al., NJIT ©2003 8 Exam Process Control Assign ID Edit problems Assign who solves problems Assign level-2 graders Course Design Determine Final Grades Set up on-line environment Dispute final grade Level-1 and Level-2 graders grade solutions Make up problems Confirmation ID, understand process Read - other problems - other solutions - grade justifications - disputes Solve problems Instructor Control ProcessStudent Learning Process Resolve Disputes

9. Bieber et al., NJIT ©2003 9 Evaluation (grading) Evaluation includes: –Written critique or “justification” (positive or negative) –Optional: separate sub-criteria to critique Solution result is correct and complete (40%) Solution was well explained (30%) Solution demonstrated class materials well (10%) Solution cited appropriate references (20%) –Grade (optional; recommended to save instructor time) Evaluation/grade may be disputed (optional) –Student must re-evaluate own solution when disputing

10. Bieber et al., NJIT ©2003 10 Instructor should provide… Detailed instructions and timetable Solution: what is expected Critiquing and grading guidelines

11. Bieber et al., NJIT ©2003 11 Outline Motivation PLA: Participatory Learning Approach A bit of theory Experimental results Interesting issues

12. Bieber et al., NJIT ©2003 12 Constructivism (Learning Theory) The central idea is that human learning is constructed, that learners build new knowledge upon the foundation of previous learning {learning throughout the exam process} Two classic categorizations –Cognitive Constructivism (Piaget’s theory) –Social Constructivism (Vygotsky’s theory)

13. Bieber et al., NJIT ©2003 13 Cognitive Constructivism (Piaget 1924) Knowledge is constructed and made meaningful through individual’s interactions and analyses of the environment. --> knowledge is constructed in the mind of individual Knowledge construction is totally student- centered.

14. Bieber et al., NJIT ©2003 14 Learning Learning is a constructivist, often social activity occurring through knowledge building (Vygotsky, 1978) Knowledge building activities include contributing to, authoring within, discussing, sharing, exploring, deploying a collective knowledge base (O’Neill & Gomez 1994; Perkins 1993).

15. Bieber et al., NJIT ©2003 15 Learning People learn as they navigate to solve problems (Koschmann et al, 1996) and design representations of their understanding (Suthers 1999) Learning requires cognitive flexibility (Spiro et al. 1991), and results from interaction with people having different experiences and perspectives (Goldman-Segall et al. 1998)

16. Bieber et al., NJIT ©2003 16 Expert-like Deep Learning Categorizing knowledge and constructing relationships between concepts are likely to promote expert-like thinking about a domain (Bransford 2000). To design appropriate problems for their peers, students must organize and synthesize their ideas and learn to recognize the important concepts in the domain. This results in deep learning (Entwistle 2000) : –seeing relationships and patterns among pieces of information, –recognizing the logic behind the organization of material –achieving a sense of understanding

17. Bieber et al., NJIT ©2003 17 Where is Knowledge Constructed in PLA? In all PLA stages: constructing problems, solutions, grade justifications, dispute justifications When reading everything their peers write –Students also are motivated to learn more when peers will read their work (McConnell, 1999).

18. Bieber et al., NJIT ©2003 18 Assessment & Learning Main goals of tests: –To measure student achievement –To motivate and direct student learning The process of taking a test and discussing its grading should be a richly rewarding learning experience (Ebel and Frisbie 1986) Assessment should be a fundamental part of the learning process (Shepard 2000)

19. Bieber et al., NJIT ©2003 19 Outline Motivation PLA: Participatory Learning Approach A bit of theory Experimental results Interesting issues

20. Bieber et al., NJIT ©2003 20 Course Information NJIT CIS677: Information System Principles Graduate level core course (Masters/Ph.D.) Aim: study how IS/IT can be used effectively Both on-campus and distance-learning sections software: Virtual Classroom/WebBoard Traditional Exam: –Three-hour, in class, 3-4 essay questions, 6 pages of notes Used PLA 5 times between Fall 1999 and Summer 2002 We compared control groups without PLA and treatment groups with PLA Also, we used with shorter essay questions in CIS 365, undergraduate course on file structures in Fall 2002, with similar survey results.

21. Bieber et al., NJIT ©2003 21 Enjoyability QuestionsSAANDSDMeanS.D.# I enjoyed the flexibility in organizing my resources 26.2%48.9%16.7%3.6%4.6%3.881.00221 I was motivated to do my best work 23.5%42.9%28.2%3.4%2.1%3.82.92238 I enjoyed the examination process 17.2%42.3%22.6%10.5%7.4%3.511.13239 SA - strongly agree (5 points); A - agree (4); N - neutral (3); D - disagree (2); SD - strongly disagree (1); the mean is out of 5 points; S.D. - standard deviation Cronbach’s Alpha=0.68

22. Bieber et al., NJIT ©2003 22 Perceived Learning QuestionsSAANDSDMeanS.D.# I learned from making up questions 17.9%42.5%21.3%13.8%4.5%3.551.08240 I learned from grading other students answers 17.7%48.1%19.4%9.3%5.5%3.631.06237 I learned from reading other people’s answers 15.8%45.0%22.1%11.3%5.8%3.541.07240 I demonstrated what I learned in class 13.6%50.2%22.6%10.9%2.7%3.61.95221 My ability to integrate facts and develop generalizations improved 21.8%49.2%25.6%2.1%1.3%3.88.83238 I learned to value other points of view 17.6%51.9%27.6%1.3%1.6%3.82.81239 I mastered the course materials 7.4%51.6%31.4%6.9%2.7%3.54.84188 Cronbach’s Alpha=0.88

23. Bieber et al., NJIT ©2003 23 Recommendation: Do Again! QuestionSAANDSDMeanS.D.# Would you recommend in the future that this exam process used? 20.7%40.1%24.5%8.9%5.8%3.601.10237 Similar results for CIS365: undergraduate file structures course using short essay questions (Fall 2002)

24. Bieber et al., NJIT ©2003 24 Outline Motivation PLA: Participatory Learning Approach A bit of theory Experimental results Interesting issues

25. Bieber et al., NJIT ©2003 25 What students liked best Active involvement in the exam process Flexibility Reduction in tension

26. Bieber et al., NJIT ©2003 26 Trade-offs Trade-offs for students (traditional vs. PLA) –Timing: Concentrated vs. drawn-out (2.5 weeks) –Access to information: limited vs. the Internet –Experimental integrity: we couldn’t justify the process to the students fully Trade-offs for professors –Fewer solutions to evaluate, but each is different –Timing: Concentrated vs. drawn-out process –Much more administration

27. Bieber et al., NJIT ©2003 27 Timing PLA for exams took 2.5 weeks For frequent activities PLA processes could overlap –e.g., quizzes, homeworks –Students could be creating problems for one quiz, while solving problems for the prior quiz, while evaluating solutions from the quiz before that Benefits to overlapping PLA activities: –working with materials from several classes at the same time –could reinforce class materials –could result in synthesis (combined understanding)

28. Bieber et al., NJIT ©2003 28 Scope Which activities? –so far: exams –what about: quizzes, homeworks, larger projects, in-class projects Which problem types? –so far: short and long essay questions –what about: multiple choice, short answer, computer programs, semester projects –Sub-problems: computer program design & implementation semester project outline & execution

29. Bieber et al., NJIT ©2003 29 Scope, cont. Course Level –Graduate, undergraduate, secondary school (high school, junior high) Disciplines –IS/IT, business, science, engineering, humanities, medical, all of secondary school

30. Bieber et al., NJIT ©2003 30 Scope, cont. Degree of Evaluation (assigning grades) –Currently: solutions –What about: quality of problems quality of evaluations/grades –All could be disputed Degree of Participation –students could evaluate each –students could arbitrate disputes

31. Bieber et al., NJIT ©2003 31 Evaluation Results Written critique (positive or negative) Grade (optional; recommended to save instructor time) Recommendation to accept or reject the “artifact” (problem, solution, evaluation) If rejected, optionally: –the artifact would have to be redone and re-evaluated –the evaluator or instructor would substitute an acceptable artifact, and the PLA process continues The evaluation/grade could be disputed

32. Bieber et al., NJIT ©2003 32 Full Collaboration Groups for: –Problems, solutions, evaluation, dispute arbitration Requires group process support –Group roles: leader, scheduler, etc. –Process: work on each activity together or separately, internal review –Grading of individual group members –Process Tools: brainstorming, voting, etc.

33. Bieber et al., NJIT ©2003 33 What can go wrong Students are late; students drop the course Entries posted in wrong place Inadequate critiques –“Good” –“I agree with the other evaluator” and of course, technical difficulties…

34. Bieber et al., NJIT ©2003 34 PLA Environment Software Guide the process Form groups Assign problem solvers, evaluators, dispute arbitrators On-line templates to ensure full entries Guide people to post entries in correct place Incorporate group process tools Handle problems as much as possible –Remind people who are late –Reallocate who does what Based on a workflow management tool…

35. Bieber et al., NJIT ©2003 35 Anonymity/Privacy Issues Should student entries be anonymous? Will students reveal their IDs? Is it fair to post critiques if not anonymous? Is it fair to post grades if not anonymous? Will anonymity work in small classes?

36. Bieber et al., NJIT ©2003 36 Issue: Perceived Fairness Should students evaluate/grade peers? –But they must evaluate others in the workplace… It’s the instructor’s job to evaluate and grade –PLA is a (constructivist) learning technique Students have no training in evaluation –Evaluation is a skill that must be learnt (and taught) Many evaluators = inconsistent quality –safeguards in the PLA process

37. Bieber et al., NJIT ©2003 37 Grading Issues Disputing high grades: –Award bonus points if students dispute (and justify with a critique) grades that are too high Encouraging honest grading: –For successful disputes, deduct points from evaluators

38. Bieber et al., NJIT ©2003 38 Grade Inflation Detailed grading guidelines for sub-criteria: great: 20 points very good: 18 points good: 14 points OK: 10 points poor: 6 points Student does “good” on 5 problems, grade = 70 U.S. students will protest vigorously Evaluators will hesitate to assign “good” Result: pressure for highly skewed grading rubrics

39. Bieber et al., NJIT ©2003 39 Other Cross-Cultural Issues In some cultures: –Students are so competitive, they would only give failing grades to peers –Students would not hurt peers’ feelings, and would only give good evaluations Some systems only have pass/fail, so numeric grades are mostly irrelevant

40. Bieber et al., NJIT ©2003 40 PLA: Contributions Systematic technique to increase learning –Constructivist approach, actively engaging students in the entire problem life-cycle –Minimizes overhead for students and instructors Experimental evaluation Supporting software PLA liberates learning from its traditional instructor-controlled structure! Thank you! Questions, please?