Assessing scenario-based learning John Mitchell Electronic and Electrical Engineering With thanks to: Sally Day, Rosalind Duhs, Lewis Elton, Tony Kenyon, Oleg Mitrofanov, Cyril Renaud, Miguel Rio, Ed Romans, Seb Savory, Jan Smith, Benn Thomsen. SYMPOSIA ON ASSESSMENT FOR LEARNING

Developing scenario-based learning for formative and summative assessment It is not in fact teaching but assessment that plays the largest part in guiding students learning. 1 it can be argued that giving learners feedback is just about the most important dimension of the work of teachers in post-compulsory education … but perhaps all told, formative feedback is the vital dimension, as given at the right time and in the best possible way it can lead learners steadily towards successful achievement in summative assessment contexts. 2 1 Snyder B.R, (1971) The Hidden Curriculum. New York; Alfred A Knopf 2 Race P. (2005) Making Learning Happen, Chapter 5, Learning through Feedback, London, Sage

Proforma Very useful for general feedback Common areas of mistakes Less easy to use for more open task

Scenarios Follow the model developed by Civil Engineering of one week, self contained, group work scenarios (problems) Aim of supporting learning rather than teaching new material Pilot process of 3 first year scenarios and two second year scenarios just completed

Scenario Assessment A mix of instruments used –Milestones –Presentations Individual technical reports Traditional reports Critical Assessments of other teams solutions –Group technical report User manual Due diligence document –Individual Narratives

History of Student-Centred Learning in E&EE 2004 – Problem Based Learning Modules Begin 2008 – First Scenario Weeks Piloted STUDENT–CENTRED LEARNING: WHAT DOES IT MEAN FOR STUDENTS AND LECTURERS? Geraldine ONeill and Tim McMahonIn: Emerging Issues in the Practice of University Learning and Teaching

Problem Based Learning 4 Problems over 12 Weeks (Term 1) 6 Students per group, groups stay the same, Students expected to spend 10 hours per week total (inc private study) on a problem No lectures! Reports and Presentations

Rational for assessment developments Students didnt perceive that they were receiving feedback Highlighted in the national student survey Feedback on my work has been prompt. Agree – 41% I have received detailed comments on my work. Agree – 49% Feedback on my work has helped me clarify things I did not understand. Agree – 53% The NSS has highlighted that students are notably less positive about assessment and feedback on their assignments than about other aspects of their learning experience. Exploring the National Student Survey, The HEA NSS - Engineering and Technology, UCL perceive

Problem Based Learning Iterative Assessment Cycle –Report submitted at end of problem for summative assessment –Opportunity to use feedback to improve report –At End portfolio of reports submitted for formative assessment Narrative Commentary introduced Peer Assessment

Summative Feedback Referencing: –Referencing of figure that have been taken from sources is good. However, there are few references in the text. This must be improved. Other Comments: –Good report which is well presented. There are some parts that are better than others, and although some attempts have been made to pull things together summary of the specifications would have been useful. Grade: B

Narratives The following are suggested questions to guide you in developing this piece of writing: –How did my group go about tackling this problem? –Was this a successful approach? –If so, what made it successful and how can I use that in future? –If not, what were the major difficulties, and how would I avoid them in future? –How good was our solution to the problem? –If I could tackle this problem again, how could I improve what we did? –How have I learned to cope with all the information I had to find for myself? –Has the group shared this information effectively? –How have we handled difficult issues in our group? –Did I do my best to ensure that the team worked well? –Do I recognise the importance of deadlines? –Did I and/or my group stick to the deadlines? –What else do I feel is of importance to include?

Narratives – advice given Merely documenting the steps you have gone through in the problem solving process will not earn you a very high mark – you will need to demonstrate what you have learned from the process. But often ignored!

Narratives Oh yeah, the narrative [wry chuckle]. The narrative, I didnt like the way the narrative was 50% of the marks because we spent a lot of time on those reports, a lot of time doing research into suitable system diagrams and everything else and I didnt like the way half the marks just went to the narrative…

Opportunities for Feedback in Scenario Assessment Formative Summative

Encouraging Brainstorming

Milestones

Presentations Presentations mid-way as guides rather than later as assessment Student dedicate significant time to presentations Too much focus on Information not enough on message Opportunity to learn from other groups and provide mid-point feedback

Competitions

Reports Attempted to get away from usual reports –User Manuals/Documentation –Very specific guidance given

Marking Grid Marking Proforma1 st Grade /52 nd Grade /5 Problem Definition This could have been improved, You need a lot more details 2 Design description It is not clear why some choices have been made. And a bullet point time line does not give any idea on some of the design. Your design is based on an error in your data-collection. 2 Theoretical Analysis The theory is adequate but you have not used it to its full potential. 3 Results and Conclusion There is almost no analysis in the conclusion, and you do not compare your results with the theory. 1 Individual input You claim to have done your part of the work over 1 day. You should have spend more time trying to understand the rest of the issues. 2 Reflective commentary More comments on what you have learned are needed 3 Total Mark20

Written Feedback

Student Debrief and Feedback Session Not yet fully utilised General comments to the whole group Citing good and bad practice

Assessment Weighting Course marking scheme (1-5 range) Currently a low percentage of final grade All aspects rewarded but none too heavily Group vs Individual? Peer Assessment?

Conclusions Student always appreciate feedback Even if they then go on to ignore it! Use assessment to guide learning rather than just to judge learning Linking feedback to further assessment seems vital Scenario Brief MilestonesFeedbackAssessment Summative Feedback