Presentation on theme: "Standards, criteria and assessment of the Geography Dissertation in the UK Dr Margaret E. Harrison University of Gloucestershire Dr Dawn T. Nicholson Manchester."— Presentation transcript:
Standards, criteria and assessment of the Geography Dissertation in the UK Dr Margaret E. Harrison University of Gloucestershire Dr Dawn T. Nicholson Manchester Metropolitan University Professor W. Brian Whalley Queens University of Belfast Assessment for Learning Conference, June 22 nd 2009
Geography dissertations 'Dissertations have had a long history in geographical higher education, being widely regarded as the pinnacle of an individual's undergraduate studies and the prime source of autonomous learning' (Gold et al., 1991) Often comprise a large proportion of the overall weighting for undergraduate degrees Performance may be used to adjudicate degree award in borderline cases Demonstration of independent working therefore grade awarded of interest to employers
Wider background Marking reliability, marking to the full range Maintenance of standards in HE (grade inflation) Accountability (litigation) Increasing evidence of new assessment culture emerging (Rust 2007) Assessment criteria (e.g. Harrison and Whalley 2008) Grade descriptors Formative assessment Feedback But still poor practice that go unchallenged (Rust 2007)
Inconsistency, ambiguity etc Inconsistent approaches to use of criteria (including weightings and standards) for dissertation marking (Hand and Clewes 2000) Inconsistent use of terms in feedback (Pepper et al 2001) Confusion among teachers over concepts of criteria and standards (Sadler 2005) Considerable ambiguity concerning the use, meaning and application of assessment criteria (Webster et al 2000)
Students and criteria Students dont automatically understand the meaning of criteria (Penny and Grover 1996, Rust et al 2003, Woolf 2004) Criteria need to be explicit and understood by students to effectively support learning (Gibbs and Simpson 2005) An important principle is that students and markers are aware of and understand the assessment criteria and/or schemes that will be used to mark each assessment task. (QAA 2006, p16-17).
Some key research questions What assessment criteria are used, how are they established and approved? How are students assisted to interpret assessment criteria? What is the role of supervision in this? What grade descriptors and marking schemes are used? What are the procedures for double marking, anonymous marking, blind marking? What happens in the event of a disagreement between first and second markers? Identify good practice and showcasing of innovative practice
Methods Questionnaire survey sent to all GEES Departments – a mixture of closed and open questions Follow-up telephone interviews with selected respondents Number of responses: 22 This represents 20 HEIs - 2 separate responses from 2 HEIs. Responses include institutions throughout the UK with an even split in terms of pre-1992 and post-1992 HEIs.
Credit-rating and word length Credits (CATS) 1520243040 No. responses 13198
Form of dissertation Responses Printed document21 E-document6 Printed document & field notebook 1 Printed document & attachments (audio &/or visual) 4 Other1 Study period and product format Most dissertations are submitted as printed documents, but the handbook provides students with the freedom to be creative about the form of the exercise.... the student can undertake a practical.... or a performance project.
Elements assessed Responses Final document (100% of marks) 14 Final document (70-90%) 8 Written progress report (10-15%) 4 Seminar/oral presentation on progress (0- 30%) 4 Final oral presentation (15%) 1 Other*2 * PowerPoint poster on progress and literature review Elements that are assessed
Assessment criteria Clarity of aims, topic identification Theoretical understanding Awareness of literature Research design and methodology Quality of database Data presentation, analysis and interpretation Conclusions justified Originality and perceptiveness Sustained argument Use of referencing conventions Integration of illustrative materials Standard of presentation Use of English language Use of complex academic terminology Compliance with requirements Risks and ethical considerations Engagement with administrative processes Cognitive, practical and personal skills Reflective ability Personal initiative and responsibility Ability to work independently Conduct and competence during practical work Use of ICT applications Critical ability Flair, innovationThink creatively
Assessment criteria We used to have weightings for each bit but we now use a more general mark sheet. Every time we get a new external examiner they have different ideas and we tend to change our marking strategy fairly regularly. (post 1992 HEI) Established by tradition … some of our descriptors are rather woolly. … The bottom line criteria for our dissertations is that it must be original. (pre 1992 HEI
Assessment criteria Assessment scheme developed by a departmental working party … and then approved by a team meeting (equal weighting of 6 aspects including research skills, analysis/discussion, and presentation) (post 1992 HEI) Im not sure what youre asking here – do you want to know the universitys quality assurance system? (pre 1992 HEI) It was laid down at the start of the Modular Course in 1974; Im not aware that it has ever changed. Indeed it is the only feature of the course that has never changed. (post 1992 HEI)
Grade descriptors and marking schemes See attached. These are departmental descriptors (pre-1992 HEI). Attached – this is institutional (post-1992 HEI). The grade structure and broad descriptors are university wide; the detailed interpretations were drawn up by the department (pre-1992 HEI). We dont have this by level. The University keeps promising it! (post-1992 HEI).
Marking All departmental marking is anonymous, but clearly supervisors are likely to recognise the dissertations of their supervisees (post-1992 HEI) 95% marked by the supervisor ALL are double marked (64% blind and 36% seen) Six are marked anonymously…….?! One uses viva (supervisor and second marker) to inform mark Qualifications of the second marker – an issue?
Disagreement generally >5% or different class Markers encouraged to discuss in first instance Two markers would meet and discuss the reasons for their grading, which may reveal something not noticed or unknown to one of them….. If no resolution, introduce third marker Use median mark from three Discuss and agree Discuss at internal exam board If no resolution (still) Module leader decides or external examiner adjudicates Disagreements about marks
Supervision Supervisory contact Responses 1-to-1 meetings 22 Small group sessions 8 Whole class sessions 11 Other5 Examples of other activities: peer led work, presentations, field and laboratory training
Feedback Only three (14%) supervisors discuss content but dont actually read, edit or comment on specific written material. Some specify no. chapters or pages, % of work, specific chapters (eg lit. review) or specific date. Feedback on complete draft rare - as work not available! Feedback on drafts Responses No comment on any drafts 3 Limited number of draft chapters 8 Complete draft 11 [Feedback] comments should not compromise the independence of the project by modifying its principal organising ideas or otherwise adjusting its conceptual structure.
Supporting students There are [also] group sessions…… and formal supervisory meetings where it is discussed (post- 1992 HEI) We top and tail the research design course with sessions on What is research and Writing up, both of which include a discussion of expectations / features of a good dissertation / assessment criteria / grade descriptors (pre-1992 HEI) Done as part of the supervision process (post-1992 HEI) Scheme leader and tutors brief students on what is expected (pre-1992 HEI).
Supporting students There is a lecture at the start of the final year, but this mainly comes through tutors. (pre-1992 HEI) It is part of the tutorial support (post-1992 HEI) Dissertation Guide – explanation of criteria Students peer-mark past dissertations using criteria Criteria discussed during Research design module Whole cohort lecture given to explain criteria Examples of good practice
More good practice...introduction of the preliminary literature review as part of the Research Design [level 5] course. Students.... deliver oral presentations to a panel of supervisors and their peers. Dissertation topics are not prescribed. After a period in which dissertations were downgraded to an option, I think there is every chance that they might.... return to being compulsory.
Conclusions Requirements for dissertations vary but there is broad consensus across the sector Consistent range of themes embedded in criteria, eg: Employability: Skills, conduct, reflection Research: Design, methods, data analysis Higher skills: Criticality, flair, innovation, originality Lot of variation in the way that assessment criteria are established: Passive: Tradition, evolution, shared experience Reactive: Compliance with University generic criteria Proactive: Designed collaboratively and reviewed
Conclusions There is some confusion in use of the terminology Evidence of rigor and transparency in marking Disagreements are dealt with in a variety of ways and the role of the external examiner varies considerably Supervision time is consistent but effective learning can be achieved through other means A range of measures are in place to support students interpret assessment criteria How do we know learning achieved? Feedback on drafts is variable – implications for maintenance of standards and student independence
Implications Care needed in the formulation of criteria that are explicit HEI staff need to adhere closely to criteria to achieve equivalence and consistency Given the confusion over terms and the widely differing practices, this has implications for staff development needs All of this is irrelevant if students are not party to understanding criteria and expectations
Bettany-Saltikov, J., Kilinc, S. and Stow, K. (2008). Bones, boys, bombs and booze: an exploratory study of the reliability of marking dissertations across disciplines. Assessment and Evaluation in Higher Education 1, 1-19. Gibbs, G. and Simpson, C. (2005). Conditions under which assessment supports students learning. Learning and Teaching in Higher Education 1, 3–31. Gold, J. R., Jenkins, A., Lee, R., Monk, J., Riley, J., Shepard, I., and Unwin, D. (1991). Teaching Geography in Higher Education. Basil Blackwell, Oxford. Hand, L. and Clewes, D. (2000). Marking the Difference: an investigation of the criteria used for assessing undergraduate dissertations in a business school. Assessment and Evaluation in Higher Education 25(1), 5-21. Harrison, M. E. and Whalley, W. B. (2008). Undertaking a dissertation from start to finish: the process and product. Journal of Geography in Higher Education 32(3), 401-418. Partington, J. (1994). Double-marking students work. Assessment and Evaluation in Higher Education 19(1), 57–60. Penny. A. J. and Grover, C. (1996). An analysis of student grade expectations and marker consistency. Assessment and Evaluation in Higher Education 21(2), 173-184. Pepper, P., Webster, F. and Jenkins, A. (2001). Benchmarking in Geography: some implications for assessing dissertations in the undergraduate curriculum. Journal of Geography in Higher Education 25(1), 23-35. Quality Assurance Agency (2006). Code of Practice for the Assurance of Academic Quality and Standards in HE: Section 6 - Assessment of Students. Available online at: http://www.qaa.ac.uk/academicinfrastructure/codeOfPractice/section6/COP_AOS.pdf. [Accessed 4th March 2009]. http://www.qaa.ac.uk/academicinfrastructure/codeOfPractice/section6/COP_AOS.pdf Rust, C. (2007). Towards a scholarship of assessment. Assessment and Evaluation in Higher Education 32, 229-237. Rust, C., Price, M., and O'Donovan, B. (2003). Improving students learning by developing their understanding of assessment criteria and processes. Assessment and Evaluation in Higher Education 28, 146-164. Sadler, D. R. (2005). Interpretations of criteria-based assessment and grading in higher education. Assessment and Evaluation in Higher Education 30, 175-194. Webster, F., Pepper, D. and Jenkins, A. (2000). Assessing the undergraduate dissertation. Assessment and Evaluation in Higher Education 25(1), 71-80. Woolf, H. (2004). Assessment criteria: reflections on current practices. Assessment and Evaluation in Higher Education 29(4), 479-493. References
Thank for your attention Any questions? This project is being supported through the GEES Subject Centre Small-Scale Project Funding scheme We would like to acknowledge Karen Logan, our Research Assistant from Queens University Belfast, for her assistance with this project.
Survey Questions 1 Do all your students undertake a Dissertation as part of their Geography Honours degree programme? How many calendar months do your students have to work on their Dissertation? What is the credit rating (CATS or ETCS credits) for a Dissertation in your Department? Does any element of a Dissertation contribute to credit at less than final year?
Survey Questions 2 Does your Department specify the number of hours of supervisory contact permitted per student? What form does the supervisory contact take and who initiates supervisory sessions? What is your Department position concerning feedback on drafts? What form may a Dissertation take? Please indicate sub-elements of the Dissertation and their relative weightings.
Survey Questions 3 Please provide a copy of your Department marking schemes and/or assessment criteria. Please indicate how this scheme / criteria is (a) established and (b) approved. Please comment on any assistance given to students in interpreting the assessment criteria provided. Please provide a copy of the grade descriptors or 'standards' scheme for the Dissertation.
Survey Questions 4 Which marking procedures are adopted for Dissertations (e.g. double marking; blind marking; anonymous marking). What procedures are followed in the event of a disagreement between markers. Please comment on whether external examiners are asked to adjudicate. Please provide an example of good practice in Dissertation assessment.