1. 1 Evaluating the Effectiveness of Problem Based Learning John Sedgwick and Guglielmo Volpe Department of Economics, Finance and International Business London Metropolitan University j.sedgwick@londonmet.ac.uk, g.volpe@londonmet.ac.uk DEE Conference – Cambridge 6-7 September 2007 Research funded and supported by the Economics Network of the Higher Education Academy www.economicsnetwork.ac.uk

2. 2 u Aims of Project u Evaluating Problem Based Learning: the literature u Structure of PBL Project u Evaluation of Experience u Evaluation of Performance u Students Perception u Staff Evaluation u Projects Next Steps Structure of Talk

3. 3 Evaluating PBL: the literature Evaluation of Outcomes Knowledge: mixed results or insignificant differences between PBL and other methods (Albanese and Mitchell, 1993); greater knowledge with PBL (Smits, 2002), (Farrell, 2003) Lifelong Learning Skills: PBL superior to conventional methods (Bransford et al., 1989) Learning Process: deeper approaches to learning ( Sobral, 1995) Team Skills: PBL facilitates development of collaborative skills (Cockrell et. al., 2002) Control Group Analysis: in some contexts PBL may lead to worse outcomes for some students (Newman, 2004)

4. 4 u Structure of Project u Industrial Economics Module u Spring Semester Module u Taught at North and City Campus to two separate cohorts of students u PBL at North Campus while Traditional Approach at City Campus u Different lecturers in the two campuses u Same assessment but slightly broader syllabus at City campus u Control for students background and characteristics u Evaluate and compare students experience and performance Control Group Analysis

5. 5 u Methodology: evaluation framework (Newman, 2004) Control Group Analysis ObjectiveMeasure Students participationClass registers Class participation Logs of meetings Students satisfactionQuestionnaires at end of semester Focus group Module evaluation form Students outcomes and students performance Assessment performance End of semester questionnaires Statistical analysis of performance Follow up to test knowledge retention

6. 6 Students Performance AverageMedianFDCBA Overall57.959.515.0%5.0%30.0%40.0%10.0%City 55.758.07.9%13.2%39.5%23.7%15.8%North Exam57.363.512.5% 16.7%33.3%25.0%City 54.656.510.5%18.4%26.3%28.9%15.8%North Coursework53.951.015.0%25.0%15.0%25.0%20.0%City 56.056.510.0%5.0%42.5%20.0%22.5%North

7. 7 Students Performance Coefficientst StatP-value Intercept55.9043917.117192.38E-17 Difference in Year 2&3 average0.9009695.5008585.11E-06 PBL in Semester A7.0797532.1835980.036683 Mature student-5.45856-1.688650.101323 International student-4.16997-1.42070.165385 Disability-8.72376-1.527940.136667 Semester B starter4.9606571.2160630.23314 Regressor: Overall performance in module by North Campus students OLS method; R 2 =0.650, Adjusted R 2 = 0.582; Observations: 38 Does experience in PBL lead to a better performance?

8. 8 Students Performance Coefficientst StatP-value Intercept60.7646327.272631.78E-32 Difference in Year 2&3 average 0.9472127.5121467.56E-10 PBL at North Campus0.5980590.2230270.824388 Mature students-8.10651-3.363550.001451 International students-4.69812-1.757740.084677 Semester B starters1.7362570.4578380.648976 Regressor: Overall performance in module by all students OLS method; R 2 =0.576, Adjusted R 2 = 0.535; Observations: 58 Is PBL correlated with a better performance in the module?

9. 9 Students Evaluation North campus (PBL) City Campus Overall Score75%8%68%10% 1. Lecturer well organised97%0%83%0% 2. Timely assessment information97%0%92%8% 3. Clear presentation83%0%69%15% 4. Appropriate level of difficulty68%6%77%8% 5. Lecturer encourages questions90%0%85%0% 6. Clear syllabus and assessment77%0%54%8% 7. Right number of topics84%0%62%8% 8. Module recommended70%7%54%15% 9. Tutorial complements lecture77%0%77%15% Students evaluation at end of semester

10. 10 Students Evaluation – end of semester questionnaire City Campus Students (15)North Campus Students (33) SA & ASD & DS A & AS D & D I would have liked to have experienced PBL (traditional method) in this module 53%33% 30% I prefer the traditional lecture/seminar (PBL) approach to any other method 53%20%42%18% A PBL (traditional approach) approach would be appropriate for this module 53%33%52%24% Final year students should be required to learn more independently (less lectures/seminars) 43% 64%24% I expect to learn more than the North (City) campus students 33%13%33%6% I will be able to retain more knowledge of IE than the NC (CC) students in the future 20%13%47%6% All students should follow the same teaching method 67%13%70%9% A PBL approach is more appropriate for final year modules 53%20%70%12% How much/well you learn does not depend on the teaching method 47% 23%47%

11. 11 Students evaluation u Students views from questionnaires and focus group (PBL students) u You are understanding while learning, as usually in lectures you do not retain the information u I hear and I forget, I see and I remember, I do and I understand u I believe with PBL students are forced to independently attain a wider knowledge because they need to research more and by so doing we come across more things than we would get in class teaching u PBL is good as it relates to real world issues and that tends to stay in my head. Group learning is also important as it is easier to understand a discussion than a lecture u PBL is not necessarily better. The tutor who is delivering the module and his ability to teach is important! u PBL is not good at all especially for final year students who need much more support at that time

12. 12 Staff Evaluation u For me, these results are much stronger than I was getting in the module a few years ago, so I feel very encouraged u The experience was such a good one in terms of classroom buzz u I enjoyed the experience and will repeat it next year u The PBL experience induced me to introduce the approach in other modules

13. 13 Knowledge Retention Evaluation of post exam knowledge retention test (mark out of 5) Modules mark City TutorNorth TutorAverage City Student69222 North Student 16754.64.8 North Student 2672.53.32.9 North Student 3582.532.8 Both City and North Campus students were contacted after completion of module They were asked to answer three questions without consulting any notes Attempts were marked blindly by both City and North campus tutors

14. 14 Overall Evaluation GreenYellowRed u Positive correlation between PBL and performance u Students appreciation of approach u Students perception of deeper / independent / dynamic learning u Students desire to be involved with approach u Students perception that approach pushes them more than other approaches u Staff satisfaction and appreciation of its value u Students development of transferable skills: u Working with others u Self-directed learning u Knowledge retention and deeper understanding u Differences in CW and exam performances but not statistically significant u PBL students seems to retain more knowledge u Statistical correlation between performance and PBL not significant u Lack of prior experience in PBL can hinder performance u Difficulty in handling a large class

15. 15 Steps to complete research… u Still two steps to complete research: u Full analysis of focus group with identification of key themes u Further evaluation of questionnaires by distinguishing responses by gender and age

16. 16 Overall Evaluation u The PBL students perform better in the coursework but less well in the exam compared to the non-PBL students u Overall the non-PBL students have performed better but the difference in performance is not statistically significant u Students who experience PBL in the first semester perform better than those without such an experience u PBL is a positive but not significant determinant of performance across the two campuses u The majority of students appreciate the value of the PBL approach in particular for final year students u About half of the City students would have liked to experience PBL in the module u Students express some concerns about their ability to engage with PBL without having any prior experience u Staff enjoy the approach and the ability to be much closer to the students learning process