Developing and Assessing the Research Skills of Students in Engineering Courses Dr Said Al-Sarawi Research Skill Development and Assessment ALTC Project Member School of Electrical and Electronic Engineering University of Adelaide A seminar for Curtin Engineering Faculty

Outline  Motivation  Students and staff issues  RSD Framework in Australia  Benefits of using RSDF  What is RSD Framework?  Case Studies – From 1 st Year to Master Level  Other Dimensions of RSD Framework  RSD at the Program Level !  RSD in conclusion  Discussion and future work

Motivation (1/2)  PhD completion rate were doubled for students who had participated in undergraduate research (Bauer & Bennett, 2003)  Performance based research funding (old RQF, ERA)  Performance based learning and teaching support – Learning and Teaching Performance Fund (DEST, 2006)  Undergraduate research has been associated with higher level of student satisfaction and generic skills development (Kardash, 2000)  Factors affecting skills implementation are (Lucas et al, 2000) Scepticism of the message, the messenger and its vocabulary Scepticism of the message, the messenger and its vocabulary The skills demanded lack clarity, consistency and recognisable theoretical base The skills demanded lack clarity, consistency and recognisable theoretical base The skills are dependent on discipline area The skills are dependent on discipline area

Motivation: Student and Staff issues (2/2) StaffStudent How can I develop and assess students’ research skills? What is meant by Research and what skills are needed? Is there a systematic, explicit approach that can be used? How I can acquire and develop these skills? Is this issue only limited to the engineering discipline? How are they relevant to my career aspirations? How can I remove the “subjectiveness” from the assessment? How will my skills be assessed?

RSDF in Australia The University of Adelaide (John Willison) −Human Biology (Eleanor Peirce & Mario Ricci) −Electrical Engineering Masters by Coursework (Said Al-Sarawi and Brian Ng) −Clinical Nursing (Frank Donnelly) −Petroleum Engineering (Steve Begg) −Introductory Academic Program (Richard Warner) −English (Joy McEntee) −Dentistry (Vicki Skinner and Leonard Crocombe) −Oral Health (Sophie Karanicolas and Cathy Snelling) −Software Engineering (Li Jiang) −Veterinary Science (Susan Hazel)

RSDF in Australia (Conts.) Macquarie University (Psychology: Judi Homewood) Monash University (Business Ethics: Jan Schapper; Sue Mayson: Business; Glen Croy: Tourism) University of Melbourne (Business Law: Eu-Jin Teo) University of South Australia (Introduction to Tertiary Learning, 2008: Rowena Harper)

The facets of student research In researching, students: 1. embark on an inquiry and so determine a need for knowledge/understanding 2. find/generate needed information using appropriate methodology 3. critically evaluate information/data and the process to find/generate 4. organise information collected/generated 5. synthesise and analyse and apply new knowledge 6. communicate knowledge and the processes used to generate it, with an awareness of ethical, social and cultural issues. (Willison & O’Regan, 2007)

Level 1 Facet A) Embark on Inquiry Respond to questions / tasks arising explicitly from a closed inquiry. Facet E) Synthesis, analysis, application Ask questions of clarification / curiosity. Learning to Frame Research Questions

Level 2 Facet A) Embark on Inquiry Respond to questions / tasks required by and implicit in a closed inquiry. Facet E) Synthesis, analysis, application Ask relevant, researchable questions. Learning to Frame Research Questions

Level 3 Facet A) Embark on Inquiry Respond to questions / tasks generated from a closed inquiry. Facet E) Synthesis, analysis, application Ask rigorous, researchable questions based on new understandings. Learning to Frame Research Questions

Why use RSD approaches? (1/2)  Benefits of using the RSD for students (according to Eleanor Peirce and Mario Ricci, Medical Sciences, Uni of Adelaide)  Their research skills in our course have improved.  They understand much more clearly what is expected of them.  They know exactly where they need to develop, thanks to feedback.

Benefits for Lecturers (2/2) We can give feedback on assessment tasks more accurately and efficiently; we can give the same quality feedback with less writing, and faster. We can get a much better idea of where our students are from a quick analysis of the RSD results than from a detailed analysis of standard marks. We can easily match assessment tasks with course objectives, and course objectives with the University’s Graduate Attributes. (according to Eleanor Peirce and Mario Ricci, Medical Sciences, Uni of Adelaide)

Case Study 1: Human Biology - 1 st Year O Week RSD III A F Read the two short then complete tasks 1 and 2. Task 1: Integrate the information presented in the two articles to write your own dot-point notes on the worksheet attached. To do this: Identify 3-4 key ideas from the articles Use these key ideas to formulate headings and underline each Make bullet-point notes and list them under these headings. After each point, indicate its source, i.e. whether the idea came from article 1, article 2, or both Provide a title that embodies the content of your notes. Task2: Which of the two articles do you consider to be the better source? On what characteristics/features of the article have you based your choice?

Case Study 1: Human Biology - 1 st Year

Literature Research Skill Stream Laboratory Research Skill Stream Field and literature research Semester 1 Semester 2 Case Study 1: Human Biology - 1 st Year O Week RSD III A F Lit Week RSD 1 IIIIII A F Lit Week RSD 1 IIIIII A F Small Group Inquiry IIIIIIIV A F Lab RSD 1 III A F Lab RSD 2 III A F Individual Open Inquiry IIIIIIIV A F

Novelty in the engineering case The differentiation points:  Clearly state the need to research skill development  The students are required to identify a gap in knowledge for each of the chosen topics (the unknown)  Rigorous literature research

Case study 2: Photonic and Communication – Master Course  Masters (coursework) course at EEE Course run by senior colleague in EEE Course run by senior colleague in EEE 2 units of lectures/exams 2 units of lectures/exams 1 unit of literature research project 1 unit of literature research project Students seek supervisors individually Students seek supervisors individually Topic chosen by student, but requires approval by supervisor Topic chosen by student, but requires approval by supervisor  Demographics Almost all are international students; usually East Asian background Almost all are international students; usually East Asian background Technically capable, but generally lack prior experience in conducting research Technically capable, but generally lack prior experience in conducting research  Goal: produce a high quality review paper on chosen topic  Mostly negative experiences in 2005 Cohort lacked basic literary research skills Cohort lacked basic literary research skills Unstructured approach towards project Unstructured approach towards project Low quality final review papers Low quality final review papers

Process Details 1. Initial diagnostic task Supervisor supplies two technical papers Supervisor supplies two technical papers Different levels (magazine, journal) for contrast Students summarise and compare the key points from both sources in one structured, bulleted list Students summarise and compare the key points from both sources in one structured, bulleted list Detailed supervisor feedback in written form Detailed supervisor feedback in written form 2. Knowledge accumulation phase More articles added to the reading list More articles added to the reading list Continually add to an organic structured, bulleted list Strategy on further reading Strategy on further reading Student applies critical evaluation on suitability of sources with supervisor input Fortnightly workshops for group presentations & discussions Fortnightly workshops for group presentations & discussions Supervisor supplies feedback; optional: external advice (CLPD) 3. Writing phase Student writes review paper based on list Student writes review paper based on list Supervisor feedback on first draft approx a week before submission Supervisor feedback on first draft approx a week before submission

Outcome and Evaluation  Outcomes from 2006 Total of 6 students plus1 external (industry) student as control Total of 6 students plus1 external (industry) student as control Qualitatively, much improved papers compared to 2005 Qualitatively, much improved papers compared to 2005 Coherent structures, logical arguments, conciseness, respect of referencing practice Coherent structures, logical arguments, conciseness, respect of referencing practice Student perspective Student perspective struggled to cope with demands of research among the pressures of regular coursework struggled to cope with demands of research among the pressures of regular coursework English as second language remained a great barrier English as second language remained a great barrier  Framework “matrix” useful for quantitative assessment  Unexpected benefit – reduction in plagiarism Turn-it-in software reported major improvements from 2005 Turn-it-in software reported major improvements from 2005 Reports tend to be on conservative side. Reports tend to be on conservative side.

Case 3: Final Year Project (in Progress) The set of project deliverables are: Semester 1Formative literature assessment (diagnostic value?) Proposal seminar design document/interim report Critical design review (peer review) Semester 2final report final seminar project log books/wiki/blog/ log … etc “We have been concerned about a degree of disconnect between the desired leaning outcomes and the assessment methods used in these projects.” HoS of School of EEE, The University of Adelaide.

RSD: Useful First-year to PhD?

Variety of RSD approaches Five distinct approaches have emerged in the use of the RSD so far: Rubrics base to assess the profile of skills for each student, as demonstrated by Human Biology rubrics. this is the most common approach, and is used by most disciplines.Human Biology rubrics A lock-step approach, whereby students are kept 'in formation' and progressively and corporately develop one level of skills at a time. This is shown by the Nursing RSD rubrics.Nursing RSD rubrics Grading with a specific grading, by incorporation into SOLO taxonomy to define grading within a specific level set, as used by Dentistry. Skill Evaluation, to evaluate the skills and levels required by existing assessments. Identification of research skills of higher degree students, Masters by Research and PhD students and/or their supervisors/advisors to locate their present skill set and plot future directions and development needs. (SOLO: Structured Observed Leaning Outcomes)

Degree of Autonomy Degree of Academic Rigour Degree of Conceptual Demand Degree of disciplinary knowledge required Status of knowledge being pursued Other Dimensions of RSD

RSD Framework at Program Level ! No studies that consider student outcomes of the explicit development and assessment of research skills over a whole undergraduate or masters-by-coursework program Undergraduate Level Bachelor Oral Health, Adelaide Uni, AU Bachelor of Media Studies, Adelaide Uni, AU Bachelor of Science, Adelaide Uni, AU Bachelor of Nursing, Adelaide Uni, AU Bachelor of Business, Monash Uni, AU Bachelor of Science (Psychology), Macquarie University, AU Postgraduate Level Master of Engineering (Advanced) (Electrical), Adelaide Uni, AU Bridging Program for International PhD students, Adelaide Uni, AU Masters and PhD (Nursing), PhD (Nursing) Trinity College Dublin, Ireland Master of Business, Monash University, AU

26  Provides the Big Picture and relates this to the assessment details for course coordinators, lecturers, tutors, and especially students  Informs assessment-first curriculum redesign  Same ‘facets’ for multiple assessments, various levels  Explicit &Transparent assessment criteria  Coherent & Incremental skill development  Revisited & (potentially) Cyclic Conceptual Framework RSD in conclusion

Discussion and Future work  How to integrate the framework into other Programs?  How will the implementation of RSD framework affect academic workload?  How to monitor and assess student’s progress?  How can this be implemented for larger class sizes?  How can the framework adopted for non-literary research skills?

References This session was funded by an Australian Learning and Teaching Council Grant 1. Bennett, N., Dune, E. & Carré, C. (2000). Skills Development in Higher Education and Employment. (Society for Research into Higher Education & Open University Press). 2. Stevens, C. & Fallows, S.J. (2000). Integrating Key Skills in Higher Education: employability, transferable skills and learning for life. Routledge, ISBN Lucas, U., Cox, P., Croudace, C. and Milford, P. (2004). “Who Writes This Stuff?”: Students’ Perceptions of Their Skills Development. Teaching in Higher Education, 9(1), Willison, J.W. & O’Regan, K. (2007). Commonly known, commonly not known, totally unknown: A framework for students becoming researchers. Higher Education Research and Development, 26(3). RSD Web Site Acknowledgement