Assessment Feedback Systems and Interactive Learning Environments The Open University, Walton Hall, Milton Keynes MK7 6AA Professor Denise Whitelock denise.whitelock@open.ac.uk DMW Congress Colombia November 2014
DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 I hate marking but want the tasks and feedback to assist student learning DMW Congress Colombia November 2014
The e-Assessment and automatic feedback Challenge Constructivist Learning – Push Institutional reliability and accountability – Pull DMW Congress Colombia November 2014
DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 www.storiesabout.com www.storiesabout.com/creativepdp c.mckillop@rgu.ac.uk DMW Congress Colombia November 2014
DMW Congress Colombia November 2014
MCQs: Variation on a theme Example of LAPT Certainty-Based Marking UK cabinet ministers demo exercise showing feedback, University College, Tony Gardner-Medwin Drug Chart Errors and Omissions Medicines Administration Assessment, Chesterfield Royal Hospital DMW Congress Colombia November 2014
MCQs: High Stakes Assessment Example of practice “Thinking Skills Assessment" (TSA) question, Admissions Interview, Cambridge Assessment, Steve Lay Example of practice “Thinking Skills Assessment" (TSA) feedback, Admissions Interview, Cambridge Assessment, Steve Lay DMW Congress Colombia November 2014
Scaffolding and High Stakes assessment Math for Science Tutor less course Competency led No point to cheat Web home exam Invigilation technologies DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 Self diagnosis Basic IT skills, first year med students (Sieber, 2009) Competency based testing Repeating tests for revision Enables remedial intervention DMW Congress Colombia November 2014
Theoretical drivers for Assessment and Feedback Piaget (1930) Individual manipulating surroundings -- mental representations can be tested Induction of rules and their testing ... Nuffield science Cognitive psychology (70s 80s) How are these mental representations stored? Classification development and how this can go wrong. .... Misconceptions and mal rules........ Student modelling .... AI Bruner (1982) assessment tasks will match competency levels depending on level of help...SCAFFOLDING Pask conversation model used by Laurillard DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 Scaffolding and Interactivity; The Open University Science Foundation Course Interaction, Feedback loops Tell, Explore, Check Predict, Look and Explain Entering the discourse of a subject via audio feedback Scaffolded text feedback (Bruner & Woods) SHOW ME button DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 Interactive Tasks Games Simulations Making the abstract concrete Directing the sequence of an animation DMW Congress Colombia November 2014
Interactivity and Cognitive Change Scores DMW Congress Colombia November 2014
Theoretical drivers: Social Constructivism Vygotsky (1978) individuals shape cultural settings which shapes minds no longer individual Activity theory (Engstrom 1987) Tool mediation Situated Cognition (Lave and Wenger 1991) Authentic assessment Peer interaction and assessment Learning conversations Laurillard (2002) DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 Elliott’s characteristics of Assessment 2.0 activities Characteristics Descriptor Authentic Involving real-world knowledge and skills Personalised Tailored to the knowledge, skills and interests of each student Negotiated Agreed between the learner and the teacher Engaging Involving the personal interests of the students Recognise existing skills Willing to accredit the student’s existing work Deep Assessing deep knowledge – not memorization Problem oriented Original tasks requiring genuine problem solving skills Collaboratively produced Produced in partnership with fellow students Peer and self assessed Involving self reflection and peer review Tool supported Encouraging the use of ICT DMW Congress Colombia November 2014
Authentic Assessment: Building e-portfolios on a chef’s course Food preparation for e-portfolio Modern Apprenticeship in Hospitality and Catering, West Suffolk College, Mike Mulvihill Evidence of food preparation skill for e-portfolio Modern Apprenticeship in Hospitality and Catering, West Suffolk College, Mike Mulvihill DMW Congress Colombia November 2014
Peer Assessment and the WebPA Tool Loughborough (Loddington et al, 2009) Self assess and peer assess with given criteria Group mark awarded by tutor Students rated: More timely feedback Reflection Fair rewards for hard work Staff rated: Time savings Administrative gains Automatic calculation Students have faith in the administrative system DMW Congress Colombia November 2014
Mobile Technologies and Assessment MCQs ,PDAs Valdiva & Nussbaum(2009) Polls,instant surveys Simpson & Oliver (2007) Draper (2009) EVS DMW Congress Colombia November 2014
Gains from Formative Assessment Mean effect size on standardised tests between 0.4 to 0.7 (Black & Williams, 1998) Particularly effective for students who have not done well at school http://kn.open.ac.uk/document.cfm?docid=10817 Can keep students to timescale and motivate them How can we support our students to become more reflective learners and engage in formative assessment tasks? DMW Congress Colombia November 2014
Collaborative formative assessment with Global Warming DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 Global Warming DMW Congress Colombia November 2014
Next: ‘Yoked’ apps via BuddySpace Student A Student B (‘yoked’, but without full screen sharing required!) DMW Congress Colombia November 2014
Global Warming: Simlink Presentation DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 LISC: Aily Fowler Kent University ab-initio Spanish module Large student numbers Skills-based course Provision of sufficient formative assessment meant unmanageable marking loads Impossible to provide immediate feedback leading to fossilisation of errors DMW Congress Colombia November 2014
The LISC solution: developed by Ali Fowler A call system designed to enable students to: Independently practise sentence translation Receive immediate (and robust) feedback on all errors Attend immediately to the feedback (before fossilisation can occur) DMW Congress Colombia November 2014
How is the final mark arrived at in the LISC System? The two submissions are unequally weighted Best to give more weight to the first attempt since this ensures that students give careful consideration to the construction of their first answer but can improve their mark by refining the answer The marks ratio can vary (depending on assessment/feedback type) the more information given in the feedback, the lower the weight the second mark should carry DMW Congress Colombia November 2014
Heuristics for the final mark If the ratio is skewed too far in favour of the first attempt… students are less inclined to try hard to correct non-perfect answers If the ratio is skewed too far in favour of the second attempt… students exhibit less care over the construction of their initial answer DMW Congress Colombia November 2014
What about emotional support in the feedback? Difficult at times to receive written feedback Not just a cognitive response How can Bales help? DMW Congress Colombia November 2014
Coding into Categories Bales analysis Psychology 1950s Analyses talk Includes socio-emotive categories Flander’s (1970) categories inappropriate as also includes classroom control DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 Bales Categories Four main groupings A. Positive reactions; agreeing and boosting the other person B. Directing/teaching C. Questions: requesting information, clarification etc D. Negative reactions: disagreement DMW Congress Colombia November 2014
Coding the tutor comments We needed a tried and tested model for categorising the comments in order to code them, so that we could quantify the types of comments that were being provided. We chose to use Bale’s Interactional Categories, as a means of doing this. We then counted the number of incidences of each type of comment within the four main categories. The 12 categories are subcategories of four main types of response, as can be seen. This type of analysis could form the basis of an analysis by computer. Also, by coding each type of response we were then able to go back and identify typical terms or phrases for each of the12 individual categories. Examples of these were……. [next slide] Bales’ Interaction Process DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 Identifying trends: H801 Graph to show conflated Bale’s categories against mean number of incidences in H801 scripts When it came to identifying trends then, In this graph, we compared the distribution of the four main categories of comments within the four levels of pass. The results can be seen in this graph. There is, more or less, a pattern for each standard of pass with regard to the types of comments given by tutors. [talk through graph] (The comments had been coded by two people with a ratability of approximately 89%) So category A shows ‘praise and agreement’ B shows ‘direction and evaluation’ C shows questions and D shows ‘disagreement’ So looking at the four levels of pass….. Etc [talk through graph] So the main objective of this phase of the analysis was to identify a set of trends in the tutor interactions that matched the grade awarded. DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 Identifying trends: H801 Pie Chart to show the mean number of incidences per pass per conflated Bales' Interactional Category for all four levels of pass in H801 scripts Key: A = Positive reactions B = Responses C = Questions D = Negative reactions The results can be seen here once again. This chart illustrates the distribution of each category of comment across all levels of pass. This doesn’t however enable us to identify trends. This gets interesting when we look at the break-down once again… [next slide] DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 Identifying trends Pie Charts to show the mean number of incidences per conflated Bales Interactional Category for ‘Pass 1’ and ‘Pass 4’ in the following courses: B820 S103 H801 Pass 1 We plotted the following charts, and a shift in the distribution of comments among the four interactional categories can be seen. This would suggest (as shown in the previous bar chart) that the number of comments that are questions, increases as the student’s score decreases, and vice versa. Pass 4 DMW Congress Colombia November 2014 Key: A = Positive reactions C = Questions B = Responses D = Negative reactions
DMW Congress Colombia November 2014 What is Open Mentor? “An open source mentoring tool for tutors” “Open source” = free and easy to use, and to embed in an institutions infrastructure and working practices “mentoring” = designed to help people learn how to give feedback effectively, through reflection and social networks “tutors” = primarily intended for teaching staff, but with clear applications for those involved in quality DMW Congress Colombia November 2014
How Open Mentor handles comments “Good work” “Yes, well done” “Yes, but is this useful?” “Can you explain what you mean” “This does not follow” A = positive reactions A = positive reactions B = attempted answers, and not a positive reaction C = questions D = negative reactions DMW Congress Colombia November 2014
Explaining OpenMentor’s Rules Four categories A – Positive Reactions B – Attempted Answers C – Questions D – Negative Reactions DMW Congress Colombia November 2014
‘A’ - Positive Reactions DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 ‘B’ Attempted Answers DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 Is the rule set generic? DMW Congress Colombia November 2014
DMW Congress Colombia November 2014
DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 What can we learn from modelling tutors marking to construct a formative e-assessment tool? Open Comment project builds on the work of OpenMentor Free text entry for History and Philosophy students Immediate feedback (in context) to students Influenced by ELIZA (Weizenbaum, 1963) DMW Congress Colombia November 2014
Open Comment addresses the problem of free text entry Automated formative assessment tool Free text entry for students Automated feedback and guidance Open questions, divergent assessment No marks awarded For use by Arts Faculty DMW Congress Colombia November 2014
Causal models of explanation First step: Identification of question types where students exhibit causal reasoning Looked for questions with: Salient variables Description of these variables Identification of trends Identification of relationship between the variables i.e. causality DMW Congress Colombia November 2014
DMW Congress Colombia November 2014
DMW Congress Colombia November 2014
DMW Congress Colombia November 2014
Praise for effort and not just ability Praise for ability per se can hinder learning (Mueller & Dweck, 1998) Praise = being clever Negative feedback now without ability Disempowering and demoralising DMW Congress Colombia November 2014
Praise and motivation: feedback and self regulation “Feeling good” feedback part of many guidelines Mueller & Dweck (1998) Praise for effort as well as performance Raven’s test given to USA school children First test feedback was praise Second test most difficult. Half praised for effort, half praised for ability Third test medium difficulty On the third test pupils praised effort increased score by 1. Praised for ability decreased 1 mark (scale 0-10) DMW Congress Colombia November 2014
How does feedback effect mindsets? Your intelligence is something very basic about you that you can’t change very much You can learn new things but you can’t really change how intelligent you are No matter how much intelligence you have you can always change it quite a bit You can always substantially change how intelligent you are DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 Mindsets (Dweck, 2006) Fixed mindset Super sensitive about being wrong Always trying to prove themselves Growth mindset Stretch themselves Confront obstacles as challenges Lack of tension when learning as they know they are novices and can improve DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 Stages of analysis by computer of students’ free text entry for Open Comment: Advice with respect to content (socio-emotional support stylised example) Stage 1a: Detect errors E.g. Incorrect dates, facts. (Incorrect inferences and causality is dealt with below) Instead of concentrating on X, think about Y in order to answer this question Recognise effort (Dweck) and encourage to have another go You have done well to start answering this question but perhaps you misunderstood it. Instead of thinking about X which did not…….. Consider Y DMW Congress Colombia November 2014
Computer analysis continued Stage 2a: Reveal first omission Consider the role of Z in your answer Praise what is correct and point out what is missing Good but now consider the role X plays in your answer Stage 2b: Reveal second omission Consider the role of P in your answer Praise what is correct and point out what is missing Yes but also consider P. Would it have produced the same result if P is neglected? DMW Congress Colombia November 2014
Final stages of analysis Stage 3:request clarification of key point 1 Stage 4: request further analysis of key point 1 – stages 3 and 4 repeated with all the key points Stage 5:request the inference from the analysis of key point 1 if it is missing Stage 6:request the inference from the analysis of key point 1 if it is not complete Stage 7:check the causality Stage 8:request all the causal factors are weighted DMW Congress Colombia November 2014
Open Comment drivers for reflection Students are able to find facts similar to X Know how X might be disputed Are able to make predictions about X Know how to use X in an argument Know how far X can be pushed Supported with tools and strategies for effort DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 SAFeSEA: Supportive Automated Feedback for Short Essay Answers An automated tool supporting online writing and assessment of essays providing accurate targeted feedback TODO: finding the right LOGO for Primma SAFeSEA Professor Denise Whitelock Professor John Richardson Professor Stephen Pulman DMW Congress Colombia November 2014 59
DMW Congress Colombia November 2014 About SAFeSEA No tutor support for drafts of first assignment Reduce dropout rate with automatic feedback? Effect of summaristion What are the beneficial factors? Correlate measures of learner activity and essay improvement http://www.open.ac.uk/researchprojects/safesea/ DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 Talk Back Checking understanding by ‘talk back’ Summaries in OpenEssayist Key words = key ideas www.open.ac.uk/researchprojects/safesea/ DMW Congress Colombia November 2014
OpenEssayist: What it tells you The system’s focus is to present summaries of your own work in different ways, to encourage you to reflect constructively on what you have written. In other words Open Essayist tells you from its analysis what are the most important or key points in your essay. You can then think about whether that was what you intended to emphasis in your essay. If not then you can make the appropriate changes. A very important aspect of the OpenEssayist system is that it will not tell you what to write, or how to rewrite sections of your essay, or even what is correct or incorrect in your essay. DMW Congress Colombia November 2014
GRAPHICS ANALYSIS ESSAY
OpenEssayist: How it gives feedback Three aspects of your essay are analysed by the system: the structure of the essay (which paragraphs constitute the introduction, the conclusion, the discussion sections, etc.), the key words and key phrases of your essay (which are the most important words and phrases, the ones that are most representative of your essay's overall meaning) the key sentences of your essay (which are whole sentences that are most representative of your essay's overall meaning). DMW Congress Colombia November 2014
DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 Sample key phrases dispersion plot DMW Congress Colombia November 2014
Visualisations as a Thinking Tool Visualising text (Bertin, 1981; Johnson et al, 1993) Free text visualisation still problematic Do users need training with visualisations? Convention vs. Instruction? DMW Congress Colombia November 2014
Short text for illustration of Rainbow Diagrams DMW Congress Colombia November 2014
Sentence graph of short text DMW Congress Colombia November 2014
Pretend essay: 10 identical paragraphs DMW Congress Colombia November 2014
Pretend essay: 50 identical sentences DMW Congress Colombia November 2014
Stanford University Boothe Prize essay DMW Congress Colombia November 2014
OU essay awarded high grade DMW Congress Colombia November 2014
OU essay awarded low grade DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 Good vs. bad? Good: densely connected red nodes (conclusion) central close links between violet (intro) and red nodes Bad: not densely connected red nodes (conclusion) not central few links between violet (intro) and red nodes DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 Findings “You don’t show me anybody’s text. You are not revealing anyone else’s essay. So students cannot plagiarize. But you are saying ‘Look hang on, this is the way this essay connects together.’ That’s what telling a good story is about this linking.” “One of the clues was talking about the way the color nodes, that the red ones are at the end. If they are a good connection with a good explanation for each paragraph. They should be connected together but the darker ones should be in the middle. But you see the color groups together so for me it automatically pulls my eye to this page because all the colors are closer together and more in the middle. So that would be the student course assignment essay with the highest mark.” DMW Congress Colombia November 2014
Rainbow diagrams related to mark awarded Multivariate analysis of variance on marks awarded to 45 students Submitted two essays Rainbow diagrams produced from these essays and rated as high, medium or low attainment Covariate showed a significant relationship with the marks F(1, 43) = 5.92, p = .01 using a directional test Essays rated as high would be expected to receive 8.56 percentage points more than essays rated as medium 17.2 percentage points higher than essays rated from rainbow diagrams as low DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 Is NLP the bridge between Computer Assisted Assessment and Learning Analytics? Automatic marking Recognising text Refining text DMW Congress Colombia November 2014
How about feedback first? Hints before writing? R.C.T. 2 essays F(1,41) = 3.23 p = 0.04 for hints DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 Creating teaching and learning dialogues: towards guided learning supported by technology Learning to judge Providing reassurance Providing a variety of signposted routes to achieve learning goals Provide socio-emotive support DMW Congress Colombia November 2014
Feedback: Advice for Action Students must decode feedback and then act on it Boud (2000) Students must have the opportunity to act on feedback Sadler (1989) Gauging efficacy through student action DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 Advice for Action Characteristics Descriptor Authentic Involving real-world knowledge and skills Personalised Tailored to the knowledge, skills and interests of each student Negotiated Agreed between the learner and the teacher Engaging Involving the personal interests of the students Recognise existing skills Willing to accredit the student’s existing work Deep Assessing deep knowledge – not memorization Problem oriented Original tasks requiring genuine problem solving skills Collaboratively produced Produced in partnership with fellow students Peer and self assessed Involving self reflection and peer review Tool supported Encouraging the use of ICT Elliott’s characteristics of Assessment 2.0 activities DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 The 4Ts Pyramid Transfer Learning Transformation Tasks Training of staff Tool Development DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 Training Question development Quality REAQ Plagiarism software Advice for Action Socio e-emotive content Maintain empathy with the Learner DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 e-Assessment Futures Research issues Adaptive testing Automatic marking Learning Analytics Data mining Web 2.0, 3.0 … DMW Congress Colombia November 2014
Feedback to prompt self reflection Analysis must prompt “Advice for Action” Self reflective discourse with computer feedback Visual representation of feedback can open a discourse between tutor and student Prompt peer to peer discourse DMW Congress Colombia November 2014
DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 References 1 Whitelock, D. (2010) Activating Assessment for Learning: are we on the way with Web 2.0? In M.J.W. Lee & C. McLoughlin (Eds.) Web 2.0-Based-E-Learning: Applying Social Informatics for Tertiary Teaching. IGI Global. pp. 319–342. Whitelock, D. (2008) ‘Accelerating the assessment agenda: thinking outside the black box’. In Scheuermann, F. & Pereira, A.G. (eds.) Towards a Research Agenda on Computer-Based Assessment: Challenges and Needs for European Educational Measurement, published by the European Commission, Joint Research Centre. pp15-21 ISSN. 1018-5593 DMW Congress Colombia November 2014
DMW Congress Colombia November 2014 References 2 Van Labeke, N., Whitelock, D., Field, D., Pulman, S. & Richardson, J. (2013) ‘OpenEssayist: Extractive Summarisation & Formative Assessment of Free-Text Essays’. Workshop on Discourse-Centric Learning Analytics, 3rd Conference on Learning Analytics and Knowledge (LAK 2013), Leuven Belgium Whitelock, D. (2011) Activating Assessment for Learning: are we on the way with Web 2.0? In M.J.W. Lee & C. McLoughlin (Eds.) Web 2.0-Based-E-Learning: Applying Social Informatics for Tertiary Teaching. IGI Global. pp. 319–342. Field, D., Pulman, S., Van Labeke, N., Whitelock, D. and Richardson, J.T.E. 2013. Did I really mean that? Applying automatic summarisation techniques to formative feedback. Proceedings of the 9th International Conference Recent Advances in Natural Language Processing (Hissar, Bulgaria, Sep. 2013). Van Labeke, N., Whitelock, D., Field, D., Pulman, S. and Richardson, J.T.E. 2013. What is my essay really saying? Using extractive summarization to motivate reflection and redrafting. Proceedings of the AIED Workshop on Formative Feedback in Interactive Learning Environments (Memphis, TN, Jul. 2013). Alden, B., Van Labeke, N., Field, D., Pulman, S., Richardson, J.T.E. and Whitelock, D. 2013. Using student experience to inform the design of an automated feedback system for essay answers. Proceedings of the 2013 International Computer Assisted Assessment Conference (Southampton, UK, Jul. 2013). Field, D., Richardson, J.T.E., Pulman, S., Van Labeke, N. and Whitelock, D. 2013. Reflections on characteristics of university students’ essays through experimentation with domain-independent natural language processing techniques. Proceedings of the 2013 International Computer Assisted Assessment Conference (Southampton, UK, Jul. 2013). Van Labeke, N., Whitelock, D., Field, D., Pulman, S. and Richardson, J.T.E. 2013. OpenEssayist: extractive summarisation and formative assessment of free-text essays. Proceedings of the 1st International Workshop on Discourse-Centric Learning Analytics (Leuven, Belgium, Apr. 2013). DMW Congress Colombia November 2014
Five assessment special issues Whitelock, D. (2012). Special Issue of International Journal of e-Assessment (IJEA) ‘Computer Assisted Assessment: Solutions and Challenges’. Vol.2 No. 1 Whitelock, D. and Warburton, W. (2011). Special Issue of International Journal of e-Assessment (IJEA) ‘Computer Assisted Assessment: Supporting Student Learning’ . Vol.1 No. 1 Brna, P. & Whitelock, D. (Eds.) (2010). Special Issue of International Journal of Continuing Engineering Education and Life-long Learning, ‘Focusing on electronic feedback: Feasible progress or just unfulfilled promises?’, Volume 2, No. 2 Whitelock, D. (Ed.) (2009). Special Issue on e-Assessment: Developing new dialogues for the digital age. Volume 40, No. 2 Whitelock, D. and Watt, S. (Eds.) (2008). Reframing e-assessment: adopting new media and adapting old frameworks. Learning, Media and Technology, Vol. 33, No. 3 DMW Congress Colombia November 2014