STEM Annual Conference 2012, Imperial College, April 2012 Evaluating the Case for Employing E-Portfolio Tools During the Final Year Engineering Project Ray Sheriff and Felicia Ong

STEM Annual Conference 2012, Imperial College, April 2012 Contents The Final Year Engineering Project –Process and Technology Could e-Portfolio tools be applied to the FYEP? –Qualitative and Quantitative Research Results –Supervisors and Students Final Year Project: Skills and Personal Reflectivity (FYP:SPA) –A PebblePAD Application Conclusion –What have we Learnt?

STEM Annual Conference 2012, Imperial College, April 2012 The Final Year Engineering Project Process and Technology

STEM Annual Conference 2012, Imperial College, April 2012 The Project Process The Final Year Engineering Project (FYEP) requires students to apply a range of personal, practical and technical skills –This represents a new way of working for many students At Bradford, the FYEP spans two semesters –Equivalent to 300 hours or 30 credits Progress tends to be gauged through regular face-to-face meetings between supervisor and student –Provides opportunity to: Assess progress Monitor engagement Discuss Technical Issues Assessment occurs at the end of the project by: –Poster and interview (20%) –Concluding technical report (80%)

STEM Annual Conference 2012, Imperial College, April 2012 The Adoption of Technology Technology in higher education is increasingly being used to enhance many aspects of learning and teaching, for example: –PowerPoint in the lecture theatre is now commonplace –Learning Management Systems such as Blackboard are widely used throughout HE The adoption of technology is being driven by factors such as: –Increased availability of broadband Internet communications –Enhanced capabilities of Web tools and applications –Widespread take-up of affordable, easy to use electronic devices

STEM Annual Conference 2012, Imperial College, April 2012 E-Portfolio Tools e-Portfolio Tools make use of the Web to: –Develop a portfolio of skills –Facilitate communication between individuals –Provide a means of record keeping e-Portfolio tools have the possibility to enhance the FYEP experience from the perspectives of both student and supervisor To date, e-portfolio tools have primarily been employed for personal development planning University of Bradford provides all students with access to an e-portfolio tool (PebblePAD) for the duration of their studies

STEM Annual Conference 2012, Imperial College, April 2012 Aims and Objectives Aim –To determine whether e-portfolio tools can be used to enhance the FYEP experience Obj-1 –To identify existing practice among supervisors and students Obj-2 –To raise level of awareness of capabilities of e-portfolio tools Obj-3 –To evaluate the performance of e-portfolio tools when used for FYEPs

STEM Annual Conference 2012, Imperial College, April 2012 Could e-Portfolio tools be applied to the FYEP? Qualitative and Quantitative Research

STEM Annual Conference 2012, Imperial College, April 2012 Methodology Apply a mixture of qualitative and quantitative techniques Consult both supervisors and students Perform initial data gathering through on-line questionnaires and semi-structured face-to-face interviews Conclude with a pilot phase involving PebblePad, the Universitys e-portfolio tool

STEM Annual Conference 2012, Imperial College, April 2012 Methodology: Qualitative Data Data gathered through semi-structured face-to-face interviews with project supervisors –Eight interviews were conducted over seven weeks during the summer of 2011 –All interviewees were members of academic staff from the School of Engineering, Design and Technology (SoEDT) –The interviews, which were recorded and transcribed, followed a common structure covering: Background and Experience Meetings with Students The Role of Technology Project Skills and Assessment

STEM Annual Conference 2012, Imperial College, April 2012 Methodology: Quantitative Data Data were gathered through on-line supervisor and student questionnaires –Generated using the Google Docs application –Available over the final two weeks of May 2011 Questionnaires were divided into four sections: –Background –Meetings –Technology and the FYEP –Project Skills and Assessment

STEM Annual Conference 2012, Imperial College, April 2012 Example On-Line Questionnaire Pages Student Supervisor

STEM Annual Conference 2012, Imperial College, April 2012 Results Supervisors and Students

STEM Annual Conference 2012, Imperial College, April 2012 Semi-Structured Face-to-Face Interviews 234 minutes of recordings were made with FYP supervisors –An interview on average lasted minutes 150 most commonly occurring words

STEM Annual Conference 2012, Imperial College, April 2012 Sample Quotes – Members of Academic Staff So every week I meet with all my project students at the same time and every student gives a short presentation about the work they have done the week before,... I think it is very much easier to gauge where the student is in a face-to- face meeting and to respond to particular technical issues, to explain complicated points and so on … I think one thing that probably is definitely missing is presentation skills or public speaking skills especially when we are trying to assess them on posters and on how confident they were and how they answered … I suspect PebblePAD might be useful but again I have not got a clear vision of how it is supposed to be used and how it might be implemented. if they are using the log-book, there is at least some written indication of what they have done.

STEM Annual Conference 2012, Imperial College, April 2012 On-Line Questionnaire: Supervisors Thirteen of SoEDTs academic staff completed the on-line questionnaire –Overall, a response rate of 27% was achieved Just over half of the academic staff that completed the on-line survey had been supervising FYEPs for more than 10 years None of the academic staff had been supervising students for fewer than 3 years More than 60% of academic staff supervised between 5 and 6 FYEP students each year –The minimum number being between 3 and 4 All bar one of the respondents indicated that was their main means of communicating with students

STEM Annual Conference 2012, Imperial College, April 2012 Supervisors – Sample Results

STEM Annual Conference 2012, Imperial College, April 2012 Observations - Supervisors Most met with students on a weekly basis Where would students benefit from Training? –90% agreed that Time Management was important Other areas to receive good support included Project Management (84%), Research Methods (76%) and Presentation Skills (76%) Familiarity with Technology –Limited expertise in a number of packages Most supervisors are at least competent in the use of the Universitys Learning Management System (LMS) package Blackboard Other applications such as Facebook, as well as the e-portfolio tool PebblePAD have very little exposure Project Practice –No strong opinion on whether Personal Development Planning (PDP) would add value with 46% of returns agreeing and 30% disagreeing –Adding a reflective account gained some support with 61% in favour –Maintaining log-books as part of good practice had strong support (84%)

STEM Annual Conference 2012, Imperial College, April 2012 On-Line Questionnaire: Students 31 students completed the survey –Corresponds to a response rate of roughly 13% –Mechanical and Medical Engineering had the greatest student representation (39%) Followed by Electronics and Telecommunications (26%)

STEM Annual Conference 2012, Imperial College, April 2012 Students – Sample Results

STEM Annual Conference 2012, Imperial College, April 2012 Observations - Students Most met with their supervisors on a weekly basis Where would students benefit from training? –90% indicated an interest in Research Methods Other areas to receive good support included Report Writing (86%), Presentation Skills (83%), Time Management (80%) and Project Management (80%) Familiarity with Technology –High level of expertise in the Universitys LMS, Blackboard, with 92% being at least competent Students indicated at least competence in Facebook (70%) and Skype (73%) PebblePAD was untried by 87% of returns Project Practice –Supportive of the use of PDP as part of the FYEP experience, with 80% agreeing that this would add value to the project Reflective writing also received good support with more than 75% in agreement Maintaining log-books as part of good practice was supported by 90%

STEM Annual Conference 2012, Imperial College, April 2012 Final Year Project: Skills and Personal Reflectivity (FYP:SPA) A PebblePAD Application

STEM Annual Conference 2012, Imperial College, April 2012 Pilot Trial Phase An application, Final Year Project: Skills and Personal Reflectivity (FYP:SPA), was developed to conclude the study –Based on an existing PebblePAD application developed for 1 st Year Induction SaPRA: Skills and Personal Reflective Activity FYP:SPA allows students to self-evaluate their competence on a scale of 1 to 5 under various activities, clustered under six Skill Statements: –Academic Writing –Academic Reading –Communication and Presentation –Individual and Learning Strategies –Library and Referencing –Research Resources within each skill set are used to inform students of freely available training opportunities As a student undertakes training and collects evidence, self-evaluation can be re- performed to demonstrate personal development and learning progression The student then has the option to share this development in the form of a portfolio of work with their supervisor

STEM Annual Conference 2012, Imperial College, April 2012 Example Skill Statement: Academic Writing

STEM Annual Conference 2012, Imperial College, April 2012 Concluding Remarks What have we Learnt?

STEM Annual Conference 2012, Imperial College, April 2012 Conclusion The FYEP represents a new way of working for most students During the course of the FYEP, regular meetings occur between supervisor and student – is the main means of contact outside of meetings E-portfolio tools facilitate communication and record keeping –The use of log-books has strong support from supervisors and students There are barriers to be overcome if e-portfolio tools are to be considered relevant to todays FYEP experience –The questionnaires illustrated a lack of awareness of the capabilities of the Universitys e-portfolio tool, PebblePAD, among supervisors and students A lack of exposure to the potential benefits of e-portfolio tools will need to be addressed if FYEP usage is to move from early adoption to mainstream The FYP:SPA application offers the opportunity to introduce a structured approach to training and personal development The developed approach could equally be applied to other disciplines or to other levels of study

STEM Annual Conference 2012, Imperial College, April 2012 Contact Details For further information please contact: –Ray Sheriff –Felicia Ong

STEM Annual Conference 2012, Imperial College, April 2012 Acknowledgement This presentation is based on work that was performed as part of the National HE STEM Programme