e-delivery of Formative Assessment and Feedback: How well does it work? Michael J Cox, Bradford School of Optometry & Vision Science An interactive poster presentation for: Bradfords Got TALENT The 9 th Annual Learning, Teaching and Assessment Conference, June 2010 Click to begin

e-delivery of Formative Assessment and Feedback: How well does it work? Michael J Cox, Bradford School of Optometry & Vision Choose an Option The Aims The Actions QMP – Pros and Cons Formatives in Action: Choose a Demonstration Evaluating Effectiveness: Facts and Figures Evaluating Effectiveness : User Comments Conclusions

e-delivery of Formative Assessment and Feedback: What are we trying to do? The Problem A large (110+) cohort of students A level 3, 20 credit module covering a broad and challenging area: Ocular & Systemic Disease Formative assessments existed but providing individual feedback to these was too resource hungry to be practical The Aims Develop and extend the breadth and depth of formative assessment available to each student Provide individual feedback to students Enhance the effectiveness of the formative assessment in assisting the learning of each student

The Module and the New Formative Assessment Structure AssessmentDateDeliveryFeedback Slides in classQMP through Blackboard 1Sem 1, Wk 570%30% (in class)Class session 2Sem 1, Wk 90%100%Class session and QMP 3Xmas Vacation0%100%QMP only 4Sem 2, Wk 50%100%Class session and QMP 5Sem 2, Wk 90%100%QMP only A 20 credit module, taught over 24 weeks and divided into 4-week blocks for the purposes of formative assessment An assessment covers material from the latest block and revises some earlier material An assessment takes 30 minutes, although students can take longer, if they wish as there is no limit to the number of attempts The attainment on the assessments is entirely for the use of the student – it does not influence summative assessment marking

What can QMP do? Present questions to students Provide a rich variety of question types Record their answers Mark many of their answers and archive results Provide feedback, dependent upon the students answer, to explain why a right answer is right and a wrong answer, wrong Provide model answers, with a marking scheme, for questions with paragraph length answers so that students can self-mark Provide a grading scheme and instructions for students so that they can determine their total mark and find out where this places them academically

Benefits when using QMP for Formative Assessment Good integration with Blackboard Flexible (well – bendable) in operation and availability QMP can mark single word or multiple word answers for you QMP will deliver feedback for you. It is immediate and can be answer (not just right/wrong) driven Simple presentation of images/video in questions and feedback

Benefits when using QMP for Formative Assessment Sequencing questions and Hotspot questions can be particularly powerful and automatically marked Archiving of students answers allows the quality control loop to be closed It makes you think hard about your assessments Watch the demo to find out more

Challenges in using QMP for Formative Assessment Not everyone is comfortable using a computer Developing your design skills takes time It can be an awkward tool to use Development time can be extensive Youll want to make things perfect, so spend more time UoB server version doesnt yet work with Win 7 It makes you think hard about your assessments Watch the demo to find out more

Find out more: Choose a Demonstration Video Full resolution versions available from the author on request. Best viewed in Full Screen Click me to return to the main navigation screen Click me to watch the YouTube video demonstration about Feedback (5 Minutes) Click me to watch the YouTube video demonstration about Accessing the Assessment (2 Minutes) Click me to watch the YouTube video demonstration about Using different Question Types (7 Minutes)

Initial Evaluation All planned delivery took place Uptake of the VLE delivered material was 80%+ (following the assessments with friends could mean this is even higher) Summative assessments, also delivered through QMP, showed a small, but definite, improvement in the average student performance

Course Unit Evaluation Written Comments In major strengths, 25 out of the 33 individuals who commented on major strengths identified the formative assessment In how to improve the unit three comments were made about formative assessment – More time to go through the formatives together in class – More formative assessments – Formatives not to appear online, but in classes, to force students to revise for them 53% return rate 4.15 score (1->5 point scale, 1 bad, 5 good) for Assessment of my work was helpful in my learning Previous scoring for this area was 3.06 (2008) and 3.41 (2005), despite equally well received overall satisfaction at ~4.3 Learning objectives were clear to me has also increased over this time, but less sharply Although not directed towards specifically evaluating this work, the CUEQs gave valuable information on the success of the work

User Survey Results Mean satisfaction scores from 23 students that completed the survey. All except two had completed all four QMP formative assessments. Two had completed three of the four assessments Error bars show 95% confidence interval 0 – Strongly Disagree, 4- Strongly Agree Survey Questions 1.Formative Assessments in OSD were useful in my learning 2.QMP was easy to use 3.The variety of question styles used was helpful 4.Feedback from the questions was helpful 5.Image quality in QMP was better than in paper based examinations 6.I wish to see QMP continue to be used in formative assessment in the future All students registered on the module were invited to complete an online survey about the use of QMP for formative assessment The Best The Worst

User Survey Comments (Likes) Instant feedback and explanations for the correct answers were very helpful. It was useful to have access to all the assessments so you can re-do the assessments at a later date and to recap all the information covered in the module. The variety of subjects was excellent. The questions were superb in the main, although one or two of the questions seemed a bit irrelevent to my actual understanding of OSD. The varied content of MCQ and short answer and photo recognition work very well. Mimicked exams taken in Jan very well. Good feedback points.

User Survey Comments (Likes) Formative assessments were extremely helpful in preparing for the osd exam. Was useful to get used to the layout and the feedback helped in letting us know how much detail to go into with the answers. An absolute brilliant way of revision, and it really helped me in preparation for the exams in way of the format of the exam, it also made the exam enjoyable if that can at all be possible.It is a pity that at the moment not all exams are possible in this format. immediate feedback aided revision and the chance to repeat tests also allowed me to go back and test myself again on certain areas where i struggled.

User Survey Comments (Likes) the feed back! the variety of questions, extremly useful in testing my knowledge/ revision, and refelctes how the summative assesment will be Very close to the exam format that we had in January and therefore prepares you well for what to expect. Liked variety of question styles rather than purely text based. Demonstrating knowledge through the identification of signs on the photos themselves ( remember a glaucoma one particularly) by placing marker over sign seemed useful and relevant! please make available every topic like this as class work conjunction with on line lecture. for every subj if possible

User Survey Comments (Dislikes) More feedback with wrong answers. Increase the number of available questions. I did not feel that getting together and going over the assessments was always a good use of my time. I think gaining the feedback via my PC good enough Not being the best speller (and from an in the first semester i think i know your views on this one)some lee way on spelling or perhaps auto correction would help so long as it is not miles out. ps have used spell check on this.

User Survey Comments (Dislikes) Allow feedback on unanswered questions! Otherwise, they were so useful the only way to improve them would be to offer more!!! The questions which involved scrolling up and down were not the easiest to follow, probably due to lack concentration more than anything. The automatic marking seemed a bit fussy, in that; if I had the correct answer but another word also it was marked as wrong. In every other way I think that the formatives were extremely useful. the quality of the images!, I dont like how our exams are done on computers, as takes alot longer to type and think using a p.c esp for people who do no not use a p.c for revision but a book instead however overall EXCELLENT WORK ! love it

Conclusions QMP was used with Blackboard to deliver re- designed and extended formative assessments for the module Ocular & Systemic Disease Assessments were successfully run and accessed throughout the academic year Question design encompasses a broad range of different tasks Automatic marking, self-marking and directed individual feedback were all used The student response to the new delivery was very positive