Presentation is loading. Please wait.

Presentation is loading. Please wait.

The eAssessment process: focusing on providing useful and detailed formative feedback presentation to the ANU Assessment Working Party Peter Evans

Similar presentations


Presentation on theme: "The eAssessment process: focusing on providing useful and detailed formative feedback presentation to the ANU Assessment Working Party Peter Evans"— Presentation transcript:

1 The eAssessment process: focusing on providing useful and detailed formative feedback presentation to the ANU Assessment Working Party Peter Evans

2 I am speaking as Peter Evans from eGrading Solutions and not Peter Evans from DoI or USQ eMarking Assistant was developed outside ANU or USQ I have an agreement with USQ and with ANU that it is totally independent of my university work the university takes no responsibility for the software or its support I am currently selling & supporting eMarking Assistant via

3 I plan to talk about why feedback on assessment is so important good feedback enables feedforward to future learning do digital tools support all steps in eAssessment? types of eMarking and eFeedback systems 10 minute tour of eMarking Assistant effective feedback principles and possible implementations using eMarking Assistant questions & comments

4 “It is impossible to overstate the role of effective comments on students’ progress in any discussion of effective teaching and assessment” (Ramsden, 2003, p.187)

5 "[good quality feedback is] the most powerful single moderator that enhances achievement." (Hattie 1999)

6 “Where courses are delivered by distance education (DE) or primarily on-line, the role of the marker becomes pivotal... markers are effectively the gatekeepers for university quality.... little attention appears to have been paid to the role of the marker. Perhaps... universities are unwilling to open a Pandora’s box.” (Smith& Coombe, 2006)

7 We want: strong feedback which results in strong feedforward into future learning student does assignment 1 Student does assignment 2 Marker produces detailed and timely feedback student uses feedback to feedforward into future learning

8 But often we have a (very) vicious cycle eroding the value of formative assessment feedback is not as useful or as detailed as it should be students are not taught how to use feedback teachers & students become disillusioned students do not fully use feedback teachers are not supported to provide effective feedback

9 We know lots about effective feedback: Seven Principle of Good Feedback Practice 1.Facilitates the development of self-assessment (reflection) in learning. 2.Encourages teacher and peer dialogue around learning. 3.Helps clarify what good performance is (goals, criteria, and expected standards). 4.Provides opportunities to close the gap between current and desired performance. 5.Delivers high quality information to students about their learning. 6.Encourages positive motivational beliefs and self-esteem. 7.Provides information to teachers that can be used to help shape the teaching. (Nicol et al. 1994)

10 Seven Principle of Good Feedback Practice (Nicol et al. 1994) Facilitates the development of self-assessment (reflection) in learning Encourages teacher and peer dialogue around learning Helps clarify what good performance is (goals, criteria, and expected standards) Provides opportunities to close the gap between current and desired performance Delivers high quality information to students about their learning Encourages positive motivational beliefs and self-esteem Provides information to teachers that can be used to help shape the teaching But how do we apply these principles without burning brightly for 6 months then burning out

11 Also there are gaps between “policy & practise”, “rhetoric & reality”, fact & fantasy” most institutions have policies that emphasise the importance of summative and timely formative assessment But Bailey and Garner claim that institutional policies have resulted in a steady reduction in effective formative assessment for several reasons: – modularisation to create flexibility in curricula – assessment has been end-loaded – class sizes and marking loads have inevitably increased – more diverse student populations – fewer interactions between tutors and their students

12 We can totally rethink assessment approaches from the traditional 2000 word paper but … teacher marked assessment is very common revolutionary changes or incremental changes many people who “do assessment” are often early career, casual or outsourced and have little impact on long term planning incremental (and measurable) improvements are good first steps we need good usable eMarking tools

13 Student perceptions of the usefulness of comments Walker looked at student responses to 125 comments written on assignments and of these comments and analyses them into15 response themes. By far the most common being that the student could not understand the comment and needed more explanation (27.2%) over 51% of the “content” comments were unusable by students e.g. they did not understand it or they did not how or why it was relevant to their work

14 Thus providing feedback on 100s of 2000 word electronic assignments (within a 10 day turnaround period) is problematic

15 eAssessment is an end to end process which is as strong as the weakest link we know students prefer electronically marked and returned assessment many staff are resistant to electronic marking – often because of lack of training or inappropriate tools and once we print the assignment the eAssessment process is broken … it is as strong as the weakest link

16 the eAssessment process peer review of the assignment submit the assignment engage student with feedback record mark & return assignment support feedforward into future assignments moderate (team) marking mark & grade the assignment train & manage contract markers prepare the assignment

17 Do digital tools support each step in the eAssessment process? peer review of the assignment submit the assignment engage student with feedback record mark & return assignment support feedforward into future assignments moderate (team) marking mark & grade the assignment train & manage contract markers prepare the assignment not well supported well supported Click the drag the item to the left or the right depending on how well it is supported. Press CRTL to drop it

18

19 Students prefer electronic submission, marking & return advantages faster turnaround more convenient & anytime & anywhere submission receipt & assignment tracking ease of storage disadvantages large file sizes and uploading problems cost to print

20 Staff and administrators (can) prefer electronic marking often quicker (if trained & supported) reduction in turnaround time, cost, lost assignments, friction more systematic processes for – tracking and monitoring – identifying plagiarism and collusion

21 Types of eMarking solutions 1.automated marking systems – used but not likely for me (at least not in a year or three) 2.inking systems over a either a PDF or over word – good for some domains and media but not relevant to me 3.typing comments – embedded, track changes, end of assignment 4.audio and video comments – embedded or linked

22 What eMarking systems are available? eMarking Assistant addin for Microsoft Word Markin software from Creative Electronic Feedback Software GradeMark, Peer Review and WriteCycle from Turnitin Semi Automatic Grader Annotate for Word from Markers Assistant LightWork: managing marking effectively ReMarks Electronic Feedback TA Toolbar Essay Grading Software reviewed in ant.com and look at the community ant.com

23 examples of emarking systems 20 or so different eMarking systems noted at There are several strategies and tools … try them and use the ones that work for you, your markers, your examiner and your students

24 a 10 minute demo of eMarking Assistant open the toolbar (ALT F8) insert and customise reusable comments insert an audio comment and save it save comments for reuse use the eRubric and inserting comments create eRubrics highlight a phrase, do a Google search comment banks, load, export, harvest

25 A 10 minute video of eMarking Assistant you can see the 10 minute version at: view the other videos at:

26 Do a 10 minute guided tour go to select Download free > eMarking Assistant 30 day trial allow the automatic download or click the link save the file to your computer Open the file??

27 the eMarking Assistant interface

28 The basic idea of reusable comments markers often make similar comments easily reused comments they can be made more detailed, useful, and consistent detailed comments containing rich media can go beyond identifying problems to explaining the problem and then prompting for a solution a bank of high frequency reusable comments means that students can receive more personalised feedback comments banks can be shared

29 We will now look at: a principle of effective formative assessment and feedback (a goal) and ideas for implementing this principle, without causing too much additional work (an implementation)

30 Goal: leverage the power of peer discussion “the student’s peer group is the single most potent source of influence on growth and development during the undergraduate years” (Astin, p. 398). Educators should formalize the peer feedback process to help students develop a habit of seeking feedback from others.... Peer to peer feedback is too important to leave to serendipity.

31 Implementation: supported peer review and assessment use the LMS to support peer review & assessment provide students with detailed eRubrics for setting standards and helping them evaluate work provide students with reusable comment banks to help them provide feedback

32 Establish an eAssessment workflow

33 Implementation: Naming, file type & number conventions Assig 1 files Assig 1.doc Final assig 1.doc Assig 1.doc Assig 1 files Assig 1.doc Final assig 1.doc Assig 1.doc

34 Using it yourself download the 30 day trial open the document enable macros look at the guided tour click the button to install it restart Word and show the toolbar using ALT F8 or – 2007 or later: Addins menu > Show Toolbar – 2003 or earlier: View > Toolbars > Show Toolbar

35 Goal: clearly communicate criteria and standards many advantages for students, markers & examiners but the mechanics often overshadow the advantages – same number of standards for each criteria – using "nice" numbers so it adds up – not everything can be expressed as a one-dimensional criterion – all criterion are weighted the same somethings can’t be quantified another thing to return

36 Implementation: eRubrics easier to create – the eRubric is a Word table with some specific columns easier to use – select, vary and calculate weighted marks – total the marks – converts totals to percentage, rescaled mark, & grade flexible – grid or list type eRubrics – marking sheets professional result

37 Findings: Rubrics improve feedback (Anglin & Anglin, 2006) criteria & standards are clearer (helps teacher, examiner, & student) assessment is more transparent to the student Computer based rubrics are more efficient – 200% faster than hand grading without a rubric – 300% faster than hand grading with a rubric – 350% faster than typing comments

38 Look at eRubric Assistant Demo it

39 Assessment needs to be valid, reliable, & consistent and perceived this way increasingly important with multiple markers or outsourced markers (CQU project) comment banks to guide markers – level, content and style of comments eRubrics to more clearly communicate standards authority (or perhaps novelty) of the printed word or electronic assessment

40 Implementation ideas: harvest comments and revisions from a folder of assignments eyeball type, quality and quantity of comments from different markers compare comments to marks student feedback on the usefulness of assessment feedback

41 Goal: Formative feedback is most effective if it is part of an ongoing dialogue for improvement between students and lecturer between students 1 student looking at feedback over several assignments and courses help students to understand and use feedback

42 Implementation: embed a link in the feedback to a feedback web site in the course perhaps in a database, forum, wiki or glossary students can discuss and clarify issues and even rate the feedback marker can see that the feedback is being acted on – perhaps for extra credit in this or future assignments markers can see if comments are useful show this??

43 Managing comment banks Active comments in Word Insert Load Send to others Save Harvest Folder of marked assignments assignments you are marking Reuse Comment bank document Send to others Quality processes

44 Principle: promote opportunities for revision without an opportunity for revision there is little point giving feedback encourage students to engage with feedback feedback and revision needs to be timely markers will be motivated by knowing that students are using feedback (markers need an audience not just writers)

45 Implementation: of effective resubmission – Use grouped forums for peer feedback – Use a Moodle 1 to 1 forum for traceable (and reusable) communication between you and the student – Ask people to resubmit fragments with the original and the new version and with self assessment – Ask students to indicate which aspects should be commented on

46 Goal: examine quality of formative feedback proactively monitor the quality and quantity of feedback and marking look at consistency between markers and consistency between comments and marks ensure that people are marking similar aspectrs of the assignment e.g. some on grammar and some on the content

47 Implementation ideas: harvest comments from marked assignments harvesting comments, revisions and marks from a folder (of folders) of marked assignments eyeballing these for quality distribute to markers for “quality assurance and training purposes”

48 Goal: using assessment to help improve course development and teaching with increasing use of “casualised lecturers” there is sometimes a lack of information passed on to future teachers – previous comments will help the markers – patterns in last year’s comments might trigger redesign – encourage students to revisit comments before submitting the next years assignment

49 Moodle implementation reporting is kept with the course for benefit of future teachers shared comments glossary benefit the student and the markers and the teaching staff

50 Other eMarking Assistant tools easier access to: – showing spelling or grammar mistakes – revision marks and versions – the document within Word select a word/phrase in the assignment then: – highlight it throughout (with variations) – Google it (web, scholar, groups, news etc) include eMarking information for students at end

51 Summary eassessment is an end to end process which is as strong as the weakest link – formative assessment provision by teachers – use of formative assessment by students formative assessment is critical important we can start to use electronic tools to assist us and students Moodle provides an environment to use these tools

52 eMarking Assistant or eRubric Assistant download from eMarking Assistant – 30 day trial – reusable comment banks, eRubrics & other tools – $20 for a yearly license and site licenses – Word for Windows (all versions) eRubric Assistant (free) – free – eRubric only – Word for Windows or Macintosh (apart from Office 2008 for Mac)


Download ppt "The eAssessment process: focusing on providing useful and detailed formative feedback presentation to the ANU Assessment Working Party Peter Evans"

Similar presentations


Ads by Google