Presentation is loading. Please wait.

Presentation is loading. Please wait.

1 Encouraging reflective learning through formative assessment: a learning outcomes approach Arlëne G. Hunter Centre for Open Learning in Maths, Science,

Similar presentations


Presentation on theme: "1 Encouraging reflective learning through formative assessment: a learning outcomes approach Arlëne G. Hunter Centre for Open Learning in Maths, Science,"— Presentation transcript:

1 1 Encouraging reflective learning through formative assessment: a learning outcomes approach Arlëne G. Hunter Centre for Open Learning in Maths, Science, Computing and Technology (COLMSCT), The Open University

2 2 Session overview brief overview of the project – development of an online formative assessment tool (SOFA) review perceptions of the value of the student online formative assessment as a reflective learning tool –methods of evaluation –outcomes and methods of engagement –preliminary conclusions

3 3 Background to project – the challenge new OU level 2 course on Earth System Science –academically course is conceptually challenging –students need to draw on broad range of skills and prior knowledge (Biology, Chemistry, Earth Sciences, Maths, Physics) –text co-published with external partner altering standard method of in-text distance learning support Challenge devise mechanism to support and inform all stakeholders of academic progress throughout course SOFA: Student Online Formative Assessment

4 4 Book 1 seven SOFAs linked to Chpts 1-7 each SOFA – 10 questions, 2 levels Revision SOFA – random selection – specific outcomes 3-stage feedback end results learning outcomes based – overall result – per learning outcome mixed format (different learning styles/ skills) SOFA framework complexity of learning increases between SOFAs accessed from course websitecourse website

5 5

6 6 Original goals/objectives formative –safe learning environment to test depth of learning and application –enhance student awareness of personal strengths, weaknesses, achievements and progression adaptive –match questions to learner competence [logistically challenging] integrated –embed SOFAs into assessment, course and programme strategies via learning outcomes approach –involve all stakeholders in feedback process on academic progression timely –offer just in time and self-directed study options

7 7 Initial challenges and problems student-course interaction an unknown entity –how would they engage with an online formative assessment –what academic/ time demands would affect engagement question setting –what represents the norm –adaptivity = × question = question bank –dealing with non-standard characters –coping with discursive questions and answers stepped learning –designing questions not activities time and resource for development

8 8 Question Types Question typeMethod of interaction assign scientific termfree text; drag and drop MCQ (1 or more answers)radial buttons complete a tablefree text; drag and drop link geological processes/ interactionsfree text; drag and drop label diagramsdrag and drop simple/complex calculations simple/complex calculations (1-2 steps; sig. figs; sci. not, units…) numeric/text entry; use or find equation, graph, info… interpret graphs, diagrams, equationsvaried multi-formatcombination

9 9 e.g. Assign scientific term

10 10 e.g. MCQ

11 11 e.g. Complete a table

12 12 e.g. Link geological processes

13 13 e.g. Label diagram

14 14 e.g. Two-step calculation

15 15 e.g. Interpret graphs

16 16 e.g. Multi-format question

17 17 Final formative feedback

18 18 Learning outcomes summary

19 19 Evaluation of SOFAs Metadata collected by system (all users) –number of users (62% in total; range 15-52%; ave 27%), timing and frequency of use per student –time spent per question/ per assessment (ave ~70 mins) –all answers (analyse common errors ; difference in scores in chapters vs revision SOFAs) Question 11 at end of each SOFA (all users) –quick feedback questions – perceived usefulness: test understanding, measure progress, revision aid –free-text comments box generic questionnaire on Book 1 usage (whole community) –which (if any) SOFAs used –how did they help learning Success Case Method targeted interviews –specific subsection of community –exemplar success and non-success cases

20 20 Success Case Method (SCM) evaluation offers quick means of gaining quantitative and qualitative insight develop models of absolute success and absolute failure meaning of success will vary between stakeholders failure does not equal absence of absolute success collect extreme robust (provable) stories to identify drivers behind experiences focus on what, so what, now what? how to make improvements

21 21 SCM process students and tutors invited to complete generic questionnaire gain two perspectives on perceived value of SOFAs 12 respondents selected for detailed interviews ~5% of all SOFA users interviewed* 9 potential success cases 3 potential non-success cases (*based on total of 270 users who accessed at least 1 SOFA between February – May 2007; ideally, 10% is preferred representative quota)

22 22 SCM Interview buckets Success interview buckets what was used that worked? what results were achieved? what good did it do (what was its value)? what helped? suggestions? Non success interview buckets barriers? suggestions?

23 23 SCM – success case summary What was used that worked? access via course website Check your understanding... repeat function and changing questions instant feedback on effectiveness of their learning approach What results were achieved? obtained impartial insight into their strengths, weaknesses and misunderstandings demonstrated ongoing progress instant pacing mechanism – review learning approach/materials What was its value? focussed learning and helped direct areas in need of extra work built confidence in personal abilities and progress provided real measure of personal achievement formative, integrated, adaptive adaptive, timely, formative formative, timely, integrated

24 24 SCM – success case summary cont/. What helped? flexible, formative structure – instantly adaptable to own needs targeted feedback on specific errors and omission positive affirmation of good work putting all learning (content and skills) into context Suggestions printable version of questions and final explanation for revision BUT recognition that targeted feedback needs SOFA to be online General benefits consolidated and clarified learning identified gaps, misunderstandings, unknown unknowns before it was too late and offered advice on how to improve shift from seeing reflective feedback to using focussed feedback formative, integrated formative, adaptive, integrated, timely formative, adaptable

25 25 SCM – non success case summary Barriers – general issues working online – prefer paper, more flexible, more familiar academic level – too high, so demoralising when wrong time pressures – too much to do in course; going online is time consuming accessing answers – prefer immediate access to read alongside questions Barriers to value expected to get all questions right first time – SAQ experience did not like being told wrong or to try again too many clicks to get to the final answer Suggestions focus on prior background skills and knowledge – gap filling downloadable/ print-based – question and answer immediately accessible colourful, fun and tactile formative, adaptive

26 26 Preliminary outcomes majority of students who tried them, liked the SOFAs and found them helpful –but only reached on average 27% (total usage 62%); 15% did all chapter SOFAs –how could remainder be encouraged to engage and continue to use individual students who did the SOFAs were repeating them –quantitative data indicated general improvement in application of learning (content and skills) on repeats qualitative feedback demonstrates more confident learners –greater personal awareness of efficiency of learning approach –growing confidence in personal skills and sense of academic progress –awareness of personal strengths, weaknesses and learning needs –changing perception from reviewing feedback (i.e. what was wrong) to focussing on feeding forward (i.e. what needs to be developed) majority (success and non-success cases) still wanted hard copy version for revision –outcome of formative versus summative usage? –familiarity with paper over PC? formative, timely formative, adaptive, integrated, timely ? formative, adaptive formative, adaptive

27 Dr Arlëne G. Hunter Teaching Fellow Centre for Open Learning in Maths, Science, Computing and Technology (COLMSCT) The Open University Walton Hall Milton Keynes England. MK7 6AA

28 28 Links – open access to SOFAs Chapter specific SOFAs – Book 1 https://students.open.ac.uk/openmark/s book1chapter1world/ https://students.open.ac.uk/openmark/s book1chapter2world/ https://students.open.ac.uk/openmark/s book1chapter3world/ https://students.open.ac.uk/openmark/s book1chapter4world/ https://students.open.ac.uk/openmark/s book1chapter5world/ https://students.open.ac.uk/openmark/s book1chapter6world/ https://students.open.ac.uk/openmark/s book1chapter7world/ Revision SOFAs – random selection of questions https://students.open.ac.uk/openmark/s book1chapter8ran1world/ https://students.open.ac.uk/openmark/s book1chapter8ran2world/ https://students.open.ac.uk/openmark/s book1chapter8ran3world/ https://students.open.ac.uk/openmark/s book1chapter8ran4world/

29 29 Working with diagrams https://students.open.ac.uk/openmark/s book1chapter8diagmodworld/ https://students.open.ac.uk/openmark/s book1chapter8diagtaxingworld/ Understanding geological processes https://students.open.ac.uk/openmark/s book1chapter8geoprocessmodworld/ https://students.open.ac.uk/openmark/s book1chapter8geoprocesstaxingworld/ https://students.open.ac.uk/openmark/s book1chapter8geotermsworld/ Using geological terms https://students.open.ac.uk/openmark/s book1chapter8mathmodworld/ https://students.open.ac.uk/openmark/s book1chapter8mathtaxingworld/ Summarising information with tables https://students.open.ac.uk/openmark/s book1chapter8tablesworld/ Links – open access to SOFAs


Download ppt "1 Encouraging reflective learning through formative assessment: a learning outcomes approach Arlëne G. Hunter Centre for Open Learning in Maths, Science,"

Similar presentations


Ads by Google