Presentation on theme: "So, just how much do your students think they know? Using online formative assessment to enhance student learning Dr Arlëne G. Hunter Centre for Open Learning."— Presentation transcript:
So, just how much do your students think they know? Using online formative assessment to enhance student learning Dr Arlëne G. Hunter Centre for Open Learning in Maths, Science, Computing and Technology (COLMSCT), The Open University in Ireland
Session overview definition and role of formative assessment overview of OU formative assessment project initial findings and implications for future use discussion threads: –formative versus summative assessment –impact of technology on learning –importance of student self-esteem and motivation to learn
Why formative assessment? What is formative assessment? –one line descriptor for students, staff, department etc… –what isnt formative assessment? How is it used? –example(s) from your course/ own learning experience Why is it used? –the intended benefits for each of the stakeholders involved
Formative assessment issues Common concerns and questions: students wont engage with it they wont know how to use it to help it may demotivate some learners its a lot of effort for no return (from the tutor and student perspective) students are already assessed too much apart from the grades, how does it differ to summative assessment how does it differ to self-assessment Should formative assessment be numerically graded or have a threshold?
Formative assessment benefits Potential benefits: safe environment to test the effectiveness of learning pre-empts difficulties before they count directs progression (scaffolding) promotes better awareness of strengths and weaknesses boosts confidence and self-esteem encourages reflection offers targeted support motivates learning (and the learner) What are the benefits for you as the tutor/instructor?
Background to project – the challenge new level 2 course in Earth System Science –conceptually challenging course –requires broad range of scientific skills and prior knowledge (Biology, Chemistry, Earth Sciences, Maths, Physics) –different approach to standard UKOU course (co-published text) Challenge devise mechanism to support and inform all stakeholders of academic progress (students, tutors, course teams…) SOFA: Student Online Formative Assessment
Original goals/objectives formative –safe learning environment to test depth of learning and application –enhance student awareness of personal strengths, weaknesses, achievements and progression adaptive –match questions to learner competence [logistically challenging] integrated –embed SOFAs into the whole course, using a learning outcomes approach –involve all stakeholders in feedback process on academic progression timely –offer just in time and self-directed study options
Book 1 seven SOFAs linked to Chpts 1-7 each SOFA – 10 questions, 2 levels Revision SOFA – random selection – specific outcomes 3-stage feedback end results learning outcomes based – overall result – per learning outcome mixed format (different learning styles/ skills) SOFA framework complexity of learning increases between SOFAs accessed from course websitecourse website
engage with course text apply learning integrate resources Questions worked example summary advice course references context in course importance of skills tested sources of help – online, AL Final Feedback listed end of question end summary competency demonstrated knowledge; cognitive; key skills Learning Outcomes tips how to progress references to text show by example Stepped Feedback flag errors
Initial challenges and problems student-course interaction an unknown entity –how would they engage with an online formative assessment –what academic/ time demands would affect engagement question setting –what represents the norm level and what to test –building in adaptivity: versions per question = question bank –technical issues/limitations dealing with non-standard characters coping with discursive questions and answers stepped learning –designing questions not activities time and resource for development and use of data
What are your goals/objectives? Why are you interested in formative assessment? How does/could it fit into your course structure? What would it test (and for whom)? Unidirectional or iterative exercise (for whom)?
Question Types Question typeMethod of interaction assign scientific termfree text; drag and drop MCQ (1 or more answers)radial buttons complete a tablefree text; drag and drop link geological processes/ interactionsfree text; drag and drop label diagramsdrag and drop simple/complex calculations simple/complex calculations (1-2 steps; sig. figs; sci. not, units…) numeric/text entry; use or find equation, graph, info… interpret graphs, diagrams, equationsvaried multi-formatcombination
e.g. Assign scientific term
e.g. Complete a table
e.g. Link geological processes
e.g. Label diagram
e.g. Two-step calculation
e.g. Interpret graphs
e.g. Multi-format question
Final formative feedback
Learning outcomes summary
Evaluation of SOFAs (2007, 2008) Metadata collected by system (all users) –number of users, timing and frequency of use per student –time spent per question/ per assessment –mode of interaction: click through –submitted answers (analyse common errors) inbuilt questionnaire at the end of each SOFA (all users) –quick feedback questions – perceived usefulness –free-text comment box targeted interviews (success case method analysis) –specific subsection of community –exemplar success and non-success cases
SOFA usage: who and when? ~60% of all students used at least once usage per SOFA decreased over time (53% - 20%) used by more female students (67%) than males (59%) high achievers x5 more likely to use than not use low achievers/fails were x2-3 less likely to use it at all predominantly used at first scheduled opportunity –if >5 questions attempted, likely to reuse for final revision –revision SOFA predominantly used mid-course number of unique users typically declined from Q1-10 –number of attempts per question constant at ~x2 number of users –implication for optimum number of questions?
How were the SOFAs used? click through showed most did questions in order –very few reviewed questions first –some repeat users targeted specific questions plausible answers generally submitted in early attempts –change in use by some users –spurious answers entered (x3) to get to final answer –repeat new version, submitting plausible attempts repeated attempts at questions in one sitting –correct answers: repeated positive engagement –thoroughly testing understanding –repeated wrong answers: deterioration in engagement –shift from plausible attempts to spurious answers
Perceived value to learning First attempt/ scheduled study period test understanding, useful for revision, measure progress made learning outcomes apparent, pace learning Revision period(s) useful for revision, test understanding, measure progress made learning outcomes apparent, pace learning Structured interviews –shift from extrinsic drivers (e.g. got the right answer)… –to intrinsic motivators (e.g. starting to recognise personal strengths and weaknesses, increased desire to see sustained improvement)
Student feedback Positive/neutral experiences reflective, focussed on personal progression, seeking dialogue/reassurance This is a great way to test what has been learnt and is a great revision tool!! I now know where I have gaps in my knowledge and also learnt some things which I didn't pick up when reading the book. You don't always appreciate what youre reading and its significance until you try and apply it. [2007 student] Negative experience typically emotive, focussed on perceived personal errors/difficulties Mathematical questions - very frustrating as I have the method right, but may get a significant figure wrong or miss a step - this cannot be seen, so [the quiz] just says wrong…[2008 student] Suggestions for improvement instructional, focussed on technical processes and/or perceived errors in questions Instead of saying one or more of your answers are incorrect it should tell you exactly which ones are wrong before you have your second attempt. [2008 student]
Structured interviews - positive What worked? – technical/system set-up access via course website Check your understanding... repeat function and changing questions instant feedback on effectiveness of their learning approach What results were achieved? – learning progression impartial insight into personal strengths, weaknesses and misunderstandings demonstrated ongoing progress instant pacing mechanism – review learning approach/materials What was its value? – perception of worth focussed learning and helped direct areas in need of extra work built confidence in personal abilities and progress provided real measure of personal achievement
Structured interviews - negative Barriers to use dont like working online – preference for paper, more flexible, more familiar time pressures – too much to do, not worth anything Barriers to learning academic level – too high, demoralising when wrong cant compare Q&A – preference for passive rather than active learning too many clicks to final answer – answer rather than process driven expected to get all questions right first time – unrealistic expectations Barriers to progression dont like being told wrong or try again – unwillingness to accept feedback should focus on what is known – unwillingness to test learning Big brother is watching – fear of reproach (all self-esteem issues)
Initial outcomes majority of students who expressed an opinion, liked the SOFAs and found them helpful (>97%) –but only reaching proportion of the student population (40% never tried) –more confident and able students more likely to use resource –academically able students make most use of resource majority of students who did the SOFAs, repeated them –quantitative data and qualitative perceptions indicate general improvement in application of learning –positive reinforcement for some; negative demotivator for others qualitative feedback demonstrates more confident learners –greater personal awareness of efficiency of learning approach –growing confidence in personal skills and sense of academic progress –awareness of personal strengths, weaknesses and learning needs –changing perception from reviewing feedback (i.e. what was wrong) to focussing on feeding forward (i.e. what needs to be developed) but, some still wanted hard copy version –issue of technology or due to familiarity?
Points for discussion Role of formative (versus summative) assessment –balancing extrinsic/intrinsic needs and expectations –mode and role of formative feedback –what type of student benefits/ who loses out (is gender really an issue) Impact of technology on formative assessment –use of technology to maximise potential effectiveness of formative assessment and/or feedback (benefit or barrier) –time constraints on staff and students –metadata – what should be collected and how should it be used Relevance of self-esteem on use and learning –using formative assessment to promote positive self-reflection rather than a negative barrier to progression –impact of self-esteem on ability to use both positive and negative feedback
Concluding points for reflection Used effectively, formative assessment can: provide a scaffolding for learning enable a greater sense of self-worth improve awareness of personal competency levels motivate individuals to aim higher but, for some students formative assessment can: act as a barrier to learning reinforce current deficits in knowledge and application negatively impact on self-esteem demotivate and prevent any attempts to improve Needs to be implemented carefully into the whole learning experience, with appropriate back-up resources to support all learners recognise the value to personal progression.
Dr Arlëne G. Hunter Teaching Fellow Centre for Open Learning in Maths, Science, Computing and Technology (COLMSCT) The Open University in Ireland 40 University Road Belfast Northern Ireland. BT7 1SU