1. Meta-cognitive Education or University Challenge? Brian Whalley Queens University Belfast b.whalley@qub.ac.uk Do we really need examinations? or Beware External Examiners Bearing Comments NB, as well as the slide show, there are notes and references in the Notes section.

2. Some things Im going to cover (but not necessarily the order)* A Victorian system? A quick analysis of examinations at Level1 What do examinations do? Deep Learning Feedback in (assessment) systems Metacognition (Ill explain about this) Where should assessment fit? Criteria-based assessment Just-in-Time Teaching, preflights What type of assessment? Deep Assessment (a meaningful concept?) *Education is a highly parallel and occasional, not serial, system

3. Trial and error - how can we provide good learning experiences? 'You know what a learning experience is? A learning experience is one of those things that says, 'You know that thing you just did? Don't do that. (Douglas N Adams, 1992) Photo: Chris Ogle How to avoid the panic? Do examinations provide good learning experiences?

4. A Victorian system? On the basis of University Challenge (my rules) Please answer the Qs on the folded sheet, You have 3 minutes to do this This is not anonymous as your answers will determine whether you get lunch –(1 course, 2 courses, 3 courses) Ill collect them in, mark them and then tell you what you get for lunch –(or not, if you have not done your revision) But Im not going to tell you how near you got to a better meal Oh, and theres no veggie option!

5. Questions regarding Level 1 (semester 1): Do you have end of semester exams? Are these essay type? Two questions in 2 hours? Do these exams count towards final degree –(and if yes, what %) Do you give students guidance (or practice) on what to expect in exams? Are any essays set which are designed? Do you give students remarks or discussion on how well they have done post exams? –(and how is this done)? Do you have MCQs (or variant) rather than essays.

6. Assessment (some more issues) Do you want me to be the sole maker? Do you want my values despite ± 10%? Do you want a second markers values despite ± 10%? Do these values suggest you really understand the questions? Should I give you marks for just trying? What standards should I apply? And what about criteria for pieces of work?

7. 7 Principles of Good Practice in Education: 1 Encourages contacts between students and tutors 2 Develops reciprocity and cooperation among students 3 Uses active learning technologies 4 Gives prompt and appropriate feedback 5 Emphasises time on task 6 Communicates high expectations 7 Respects diverse talents and ways of learning (Chickering and Ehrmann 1996)

8. 6 Competencies students need to gain Competence – encouragement by challenge and remarks to achieve skills levels Confidence – promoting remarks to show themselves, and others, their achievements Critical thinking – which is what we have been wanting all along in 'Thinking skills, used in problem solving Creativity – in what students do and how they do it Collaboration – bringing in team-working and ethics Commonality – of purpose, to achieve specified (and unspecified) objectives Curtiosity – which is more than curiosity. Marcia Mentkowski Mihály Csíkszentmihályi

9. Itiel Dror (University of Southampton, now Cognitive Consultants International) Control – handing appropriate responsibilities to students Challenge – student abilities Commitment – student commitment to learning opportunities What might be the best ways (note plural) to develop these? But our usual question is not this. Using cognitive psychology might be a help Are these competencies, abilities and responsibilities measurable? 3 Controls on Educational Implementation

10. The Ten Principles of Assessment and Feedback 1.Help clarify what good performance is via aims, criteria, standards 2. Encourage time and effort on challenging learning tasks. 3. Deliver high quality feedback information that helps learners self-correct. 4. Encourage positive motivational beliefs and self- esteem. 5. Encourage interaction and dialogue around learning (peer and teacher student) (Nicol 2007)

11. 10 principles continued 6. Facilitate the development of self-assessment and reflection in learning. 7. Give learners choice in assessment – content and processes. 8. Involve students in decision-making about assessment policy and practice. 9. Support the development of learning communities. 10. Help teachers adapt teaching to student needs. How do we use exams (and coursework) to deliver these?

12. What actually is deep learning Something that is promoted by the 7P, 6C, 3C, 10P Experiential? Thought-exchanging (dialogue) Game-playing Exemplar-exploring Art-expressing Problem-solving (Using the 200 hours effectively) Something that is assessed in ways other than unseen examinations?

13. Assessment - whats it for? Quick post-it storm (nerd-storm) –What do you think assessment is for? As opposed to what it is –How do we measure (if we are to give marks)? –Is it measurable (adequately)? In particular If we promote Deep Learning how do we produce Deep Assessment?

14. Criterion Referenced Assessment Provide criteria for each piece of assessed work Avoid the connoisseur approach (I know this is 57%) Show students what is needed to accomplish tasks Provide remarks with the marks Marks and remarks go hand in hand Give a sense of achievement Thus, criteria need to be assignment specific and so Link all this back to Just in Time Teaching (JITT) and After Action Reviews (AAR) –Providing proper feedback and feed-forward

15. External Examiners may say (after they have agreed about the marking:) –You over-assess what does this actually mean? –You have too many pieces of assessment –Why are there so many poor 2nd (3rd) answers in the scripts? –You do not have enough > 60% –Your fail rate is too high ……… What they don t (usually) ask: How do you provide feedback in this module How do students benefit from feedback? How do you assess your learning outcomes? How do you assess the skills stated for the module?

16. Metacognition reflecting on ones own knowledge base as well as that of students 1.What is the meaning of the musical term allegro ? A. loud, B. soft, C. quick, D. slow. 2. In Greek Myth, who is the God of wines? A. Demeter, B. Hestia, C. Artemis, D. Dionysus. 3. How many years was Margaret Thatcher Prime Minister? A. 9, B. 10, C. 11, D. 12. 4. What was Lady Chatterley s first name? A. Eleanor, B. Constance, C. Margaret, D. Elizabeth. 5. What year was the Lockerbie disaster? A. 1986, B. 1987, C. 1988, D. 1989. 6. London s population (2001) is approximately, in millions: A. Seven, B. Eight, C. Nine, D. Ten In the questions about to be revealed you have to answer as you think appropriate: A, B, C, or D BUT you will be awarded marks as follows: Correct answer 1 Incorrect-2 Unanswered 0 So you may want to pass However, later, you will have a further chance to gain marks

17. And a few more 7. What is the most common element on Earth ? A. Oxygen, B. Nitrogen, C. Iron, D. Hydrogen. 8. There is no Nobel Prize in? A. Physics, B. Chemistry, C. Biology, D. Medicine. 9. What year was John F Kennedy shot? A.1962, B. 1963, C. 1964, D. 1966. 10. What countries does Luxembourg have borders with ? A. Belgium, Holland, Germany; B. Germany, Switzerland, Belgium; C. Holland, Belgium, France; D. Belgium, France, Germany. 11. What was the last album the Beatles recorded together? A. Let It Be, B. Rubber Soul, C. Magical Mystery Tour, D. Abbey Road

18. To mark 1.When you have finished round one: Have another go for those you did not answer, this time, guess, you ll get 1 for correct and 0 for incorrect. (use a second column, different colour pen) 2.Here are the answers: mark accordingly to the two rule sets (1,0,-2) & (1,0) 3.Round 1: Negative suggests 1/3 answers were wrong - you overestimated your knowledge 4.Round 2: >25% correct suggests you should have answered more in R1 - you didn t know you knew the answers!

19. MCQ or Certainty? Are MCQs just glorified University Challenges? Useful for focusing expectations rather than (or just than) knowledge per se? Use them as preflights to indicate knowledge/or understanding - ie formative rather than summative.

20. Using metacognition Now we have learned something about your cognitive abilities - and received some remarks about the processes How do we apply this to students? –Better to give MCQs before you embark on some task to show students what they know and what they need to know –Tony Gardner-Medwin and confidence-based assessment (rather than just MCQs) –Students need to know the rules applied (in general not just MCQs) –The delayed knowledge effect (sub-conscious) EVEN! –Should exams be set well before the end of the semester so students can get remarks with their marks.

21. Rewarding the experience If students have problems with a troublesome concept or fail to grasp tacit knowledge do we penalise them? How do we provide scaffolding in the learning experience? (The right amount of scaffolding) If deep learning is experiential, then how do we reward it? How many marks do we give for just having done something? (Attainment) I dont think an essay or MCQ scores well –Report writing is better Use criteria referencing Treat exams as problem solving exercises

22. Towards a proper educational feedback system What do I mean by this? Forget about Kolb cycles and Cowan loops* or, perhaps better, use them to give a system that works for all students (and staff) Construct (with Alignment*) a system (for a module) that maximises learning and understanding (rather than memory recall) To produce proper alignment we really need more specific instructions per task/activity And to provide remarks with the marks is use a feedback in an educational control system

23. Feedback and Feedforward in a control system sense Do exams provide a reference input (or even output?) A MCQ might do this. Criterion Referencing might too.

24. Educational ramp up To raise the level of attainment We need controlled input, ramping up. –Output, achievement, results attainment Ee Marks, no remarks for Exam events Ese Ee Delayed marks and remarks for Essay event Ese T T ask set with Criterion Referenced assessment Semester start Semester end

25. Ee end start Attainment Ee Delayed marks and remarks for Exam event, Ee Ee Can I be more constructive? A T MR A T Task Activity Marks Remarks A T MR R Ee M A T MR Gradient = learning rate Two Activity c/work Two Activity c/work + exam How do we increase the gradient?

26. Delays and Troublesome Knowledge & Sticking Points Troublesome knowledge (Land) Tacit knowledge (Polanyi) Delays doing activities Sticking Points Preflights (Just-in-Time Teaching; G Novak) –Feedforward (controlled use of advance info) After Action Reviews +ve Feedback or remarks

27. Problem specified Manipulation of system Problem solution Clues Guidance Help Provision of Feedback on solutions Rules System Examples Procedures Task Resources Support Assessment T Tacit knowledge required S Sticking point(s) likely T S Fig 5.1 A temporal sequence describing a rule-based learning design, In Oliver et al. 2007. Describing ICT-based learning designs that promote quality learning outcomes R R

28. What drives learning Not the assessment (itself) –This (best) drives checking (controller in the system). Its an indicator Driving learning (+ve feedback) –Internal (student doing the activity) –External (comments from others, peers, tutor, manuals) Inhibitors to learning (-ve feedback) No remarks (feedback) Exams with no remarks (feedback) before the end of the module

29. If we believe in these principles of deep learning then (in general): Why do we set essay exams? –and seem to rely on closed exams? –and not used open book or seen? Why do we set exams at the end of modules? Why do we not give feedback on exams Why do we believe MCQs are ok; better? Why there are several cetls involved with active learning etc but few with Assessment? ……… and you can probably think of more

30. Some possible conclusions Students learn best by experience Build this into assessment Ask when, and how much, assessment is needed Start by designing the feedback which reinforces the education Use the best-fitted assessment method Do students know the criteria? Get students (esp peer viewing/discussion) to look at criteria and then the answer(s) Give exams (and remarks) before the end of the module Fit these into the 10 Principles (etc)

31. Gibbs and Simpsons (4) Conditions Conditions under which assessment supports students learning 11 conditions of which 4: 1. Capture sufficient study time and effort (in and out of class) 2. Are spread evenly across topics and weeks 3. Lead to productive learning activity (deep learning) 4. Communicate clear and high expectations

32. How might we use exams? Set questions which relate to problem solving (rather than being able to answer them by memorising) This might be solving an actual problem* By providing checks or reinforcements of what has been experienced –if not experientially learned By using criterion referencing more –Ie by showing students what we expect * which is what (tends to) happen in e.g. physics, maths, chemistry and engineering.

33. Finally! In the light of what I have been talking about, Do you consider that you need to reconsider your (first year) exam process? Was the discussion helpful?

34. Exams Jeremy? but probably not as you knew them b.whalley@qub.ac.uk