Presentation on theme: "Assessing engineering skills (in a way that also supports students in developing them) Kristina Edström & Jakob Kuttenkeuler Lyngby, 21 December 2009."— Presentation transcript:
Assessing engineering skills (in a way that also supports students in developing them) Kristina Edström & Jakob Kuttenkeuler Lyngby, 21 December 2009
Large project groups (5-20 students) Entire final year 1/4 time speed Multi-disciplinary Very open-ended A final year experience- based, design- build, project course
Open-ended tasks Task 2002: Design, build and fly a solar powered aircraft that can take 2 kg payload.
Ahh, the course where you teach the students to build cool stuff!?
What is the purpose of project work in education? Project goals Learning objectives Project
Powerful principle 1: the purpose is student learning
The purpose is learning A Helsinki teacher said: ”It is a pain to have students build anything. It takes them 3 months to do what the technicians can do in 3 days.” No. It takes them 3 months to learn. They don’t build for the purpose of building, the purpose is to learn (from building). Project goals Learning objectives
After the course the participant is expected to be able to: analyse technical problems in a systems view handle technical problems which are incompletely stated and subject to multiple constraints develop strategies for systematic choice and use of available engineering methods and tools make estimations and appreciate their value and limitations pursue own ideas and realise them practically make decisions based on acquired knowledge assess quality of own work and work by others work in a true project setting that effectively utilises available resources explain mechanisms behind progress and difficulties in such a setting communicate engineering – orally, in writing and graphically... always the same learning objectives
Teachers advice & coach but avoid to impose solutions Students create new things
CDIO -conceive -design -implement -operate Students create new things
New year - New group - new task Neither the students nor the teachers know the answers Students create new things
Applied use of theoretical skills Whatever is designed has to be realised Students create new things
Individual grades A-F Assessing individual performance in a group setting Students work on many different tasks Teachers see only fragments of the actual performance Assessment challenges
Let’s hear… How would you assess these learning outcomes? (5 minutes)
Staying focused … Course Project Product Performance”according to plan” Learning objectives Constant battle between product focus and maximizing learning
Powerful principle 2: feedback and reflection on experience The process is rich with peer feedback and self-reflection activities because experience results in learning only if reflected upon
Powerful principle 3: reversing the ’burden of proof’ Each individual student is responsible for collecting and presenting evidence of how well they fulfil the learning outcomes (portfolio) this enhances reflection (learning) this makes the course sustainable in terms of teaching resources
SEPTEMBERJUNE Faculty communicate course goals instruct students to collect evidence in “portfolios” Students express personal individual goals plan own activities Assessment – the Introduction
SEPTEMBERJUNE Faculty repeat course goals lecture on giving/receiving feedback Students write summary read summaries, write feedback, suggest peer grades read feedback & reflect revisit/revise personal goals follow-up on the process Mid Course Poll
Summary template Summarize your work mapped on the learning objectives on max 1 A4 Learning objective 1. Learning objective 2. … Learning objective 10. References: Xxxx (reports, presentations, protocols, pictures, sponsring…)
Sample Summary L10. Present technical work (orally, in writing and using other tools). Status: Satisfied. Ref:     , - I am author of 7 reports of which  as main author. - Prepared and given the presentation on the preliminary design  together with Jocke. References: 1. Meeting minutes from … 2. Presentation, Preliminary design at design review #1 3. Experiment 4, Planning, execution and results 4. Report A 12, Hydrostatic stability - analysis 5. Report A107, Engine, design and mounting …
Sample Summary L7. Effectively choose and use available engineering methods. Status: Approaching. Ref: , - I am trying but find it hard to find the balance between rough estimates and sophisticated computerized methods. Further, the word “effectively” does not apply on me. References: 1. Meeting minutes from … 2. Presentation, Preliminary design at design review #1 3. Experiment 4, Planning, execution and results 4. Report A 12, Hydrostatic stability - analysis 5. Report A107, Engine, design and mounting …
SEPTEMBERJUNE Faculty recap course goals Students write summary read summaries, write feedback & suggest grades follow up discussions on the contribution from each student reflective exercises write reflective documents Final Assessment
What do you think, should the success of the product influence the grades?
The purpose is learning A Sydney teacher said: ”We have them build a rubber band-driven car and compete. If the car runs 15 meters they get the highest grade, 12 meters the next grade and so on...” No. They should be graded on how well they reach the learning outcomes, not the product performance. Project goals Learning objectives
Grades The grades are set in relation to the intended learning outcomes based on a holistic assessment of portfolios (reports, protocols, presentations, sketches, hardware, …) given feedback received feedback + recommended grades from peers participation and to be honest, a little logged time continuous observations by two teachers, independently
Grading criteria For grade A you should also Distinguish yourself in several of the above task areas and learning outcomes Show special personal engagement, responsibility and initiative for the project and group work For grade B you should also Work actively with analysis, practical implementation, administration and communication Clearly show that you reached the learning outcomes For grade C you should also Work in most of the fields analysis, practical implementation, administration and communication Clearly show personal initiative and engagement in the course For grade D you should also Work with several types of tasks in the project To some extent take on responsibilities in the course Clearly show that you approach most of the intended learning outcomes For grade E you should Actively participate in the course seminars and project meetings Actively participate in the course activities, read and answered emails from course leaders and delivered the course assignments Spend time on task corresponding to 20 credits Show that you approach the intended learning outcomes to a significant extent
Powerful principle 4: ’for the good of the project’ The project and the group drives the specifications, the needs, the deadlines... not the teachers! makes everything students do in the course meaningful, reporting comes natural for the first time makes the course format sustainable and inexpensive
Let’s hear some student voices Interviews with students in the 2004 & 2005 cohorts (not the students in the picture...)
You knew theories before, empty phrases. But now I have seen them in reality. These things are so easy to say. Like [...]. I mean, you don’t have to be a rocket scientist to realise that, everyone knows it. But it’s one thing to know and another thing to apply, and we really got first-hand experience from applying it. It is so obvious, you can stop anyone on the street and they would say ‘of course, everyone knows that’. But it is a completely different thing to experience it in reality. Interviewer: What did you learn about working in teams?
Experience-based learning is necessary since If taught: the content/goals are perceived as obvious and trivial If taught: when a real situation occurs one might fail to recognise it but remember Experience only results in learning if reflected upon
“[Changing the project leader] wouldn’t have furthered the project. It could only have suffered. But if you completely drop [considerations for] the product - and maybe you should, actually – it might have furthered the course. It's hard to tell...you simply tend to put your focus on the product you are making.” Interviewer: So you chose not to switch project leader? Tension between project and learning...
In the beginning I think there should have been some technical seminars to give a faster start of the project. Technical specialists who could have given a few lectures. To help you see possible designs for instance? Yes, technical solutions. And whom we could have contacted later with questions. Hmm. I wonder if you may risk the main idea of the course? Yes... that is a risk... If they say ‘this is what you should do’... Yes, you are right. I can see that it’s been painful though. Yes, but maybe that’s what is good for us. But you think it would have been better with a more efficient start. Yes, but that is perhaps because it had led to a better end result, I mean the boat. But maybe the learning wouldn't... Interviewer: How do you think this course could be improved? Tension between project and learning... Conceptions of teacher’s and student’s roles are challenged...
I think some [of my fellow students] feel that we haven’t used much of our previous knowledge. Because it’s not like exam tasks, it’s not extremely difficult, but you have to think more widely. We are building this thing in parts that have to work together and as I see it that is pretty advanced [engineering]. And it is [application of knowledge from several other courses]. Here we use mostly the fundamentals, but it is advanced to apply them, the situations are difficult because it’s real problems. In the previous courses I had to learn a lot of advanced stuff, but never how to use them. Interviewer: What was it like to apply knowledge from previous courses? Previous course experiences create a narrow view on knowledge...
Interviewer: How do you think this course could be improved? They should have been more like teachers. We had to do all the hard work ourselves and we don’t feel that we got as much help from the teachers as we could have had. [...] When we went and asked them ‘does this look alright’, they tried to answer as vaguely as they could. Just because they tried to make us solve things ourselves I think. Student’s views on knowledge are challenged... Conceptions of teacher’s and student’s roles are challenged...
Not that these were the only calculations needed, but the only ones that could be made. All the calculations assuming kinematic equilibrium seem to give various degrees of unreasonable results. This is not just a pity and shame, but it is also terribly bad pedagogy now towards the end of an education. I would really have liked to see that the theory we have learnt was possible to use. We cannot even calculate the strength since everything is so tiny. Quote from a mid-course evaluation Students with a dualistic view on knowledge are seriously challenged...
Student views must always be interpreted We notice that students’ conceptions of learning or attitude towards knowledge is challenged in students’ eyes, learning is often overshadowed by the project per se The teacher or course will often be blamed, as students think the teacher should have saved them from the inconvenience. But relevant challenges are not ”flaws” that should be eliminated. They are key learning opportunities and we have no intention to protect the students from them. It is then not appropriate to behave in conformity with student expectations. But knowing they exist was valuable...
Lessons learned Peer assessment is delicate to compose – not enough to use the right instruments, one has to play them carefully too We keep students informed and invite them to discuss aspects of the course – they need to be able to trust the process
Powerful principle 1: the purpose is student learning As opposed to reaching project goals (but note that the project still drives learning & creates a motivational context) As opposed to teacher popularity, or giving students an ’easy ride’ (but note that students must still have trust in the process)
The greatest thing I have learned from this course is humility. I'll approach similar tasks more humbly in the future. We thought we were better than we were. No, not better, but we have taken courses with well-defined problems, where there is an answer, the key. And that went well. But now you realised that as soon as you are confronted with reality, it’s quite another story. The beautiful sound of students growing into engineers...
“At the beginning of the course I was somewhat worried about finishing the education and starting to work as an engineer. Those worries are gone now. My confidence in approaching technical problems and solving them has grown a lot.” “Feedback was exchanged on everything between napkin scribbles at lunch to things you had built. This was valuable since it both gave me, and trained me to give, critique. It also helped me to see how other people are thinking and how they solve problems.” “One of the best things during the project was that written documentation was called for and that we in much lived up to those demands. It allows you to cross check things and check the work of yourself and others, and things are always available.”...and more of the same...
Powerful principles: 1.The purpose is student learning 2.Feedback and reflection on experience 3.Reversing the ’burden of proof’ 4.’For the good of the project’
The key idea of CDIO is that engineering skills and disciplinary knowledge are interdependent and should be learned together. The teamwork and communication skills should be so strongly interwoven with the students’ application of technical knowledge, that the two are inseparable. INTEGRATED LEARNING
It is the assessment system that constitutes the real learning objectives to the students (rather than the espoused learning objectives). Assessment is the most powerful tool we have to guide and support student learning. Engineering skills must be given proper attention in the assessment system, as legitimate and required learning outcomes of the course. WHAT WE ASSESS IS WHAT WE GET
LET’S ADD UP THEREFORE We will not get integrated learning if we separate disciplinary knowledge and engineering skills in assessment. In order to fully communicate the intention to the students they must be assessed together. CONCLUSION Integrated learning calls for integrated assessment. + WHAT WE ASSESS IS WHAT WE GET INTEGRATED LEARNING
REFERENCES - Edström, El Gaidi, Hallström and Kuttenkeuler (2005). Integrated assessment of disciplinary, personal and interpersonal skills - student perceptions of a novel learning experience, Proceedings of the 13th Improving Student Learning, OCSLD, Oxford, UK. - Hallström, Kuttenkeuler and Edström (2007). The route towards a sustainable design-implement course, Proceedings of the 3rd CDIO Conference, Cambridge, MA.
The teacher’s fundamental task is to get students to engage in learning activities that are likely to result in their achieveing the desired outcomes in a reasonably effective manner....remember that what the student does is actually more important in determining what is learned than what the teacher does. [Shuell, quoted in Biggs & Tang 2007]
Formulating objectives Designing activities What work should the student do, to reach the objectives? Designing assessment What should the student do, to demonstrate that they reached the objectives? What should the student be able to do as a result of the course? CONSTRUCTIVE ALIGNMENT
How important is assessment? ”If we want to discover the truth about an educational system, we must first look to its assessment procedures.” [Rowntree 1987] ”It is not the curriculum that shapes assessment, but assessment which shapes the curriculum.” [Brown & Knight, 1994] ”Students can escape bad teaching; they can’t avoid bad assessment.” [Boud 1994]