1. What are examiners thinking when they examine a thesis? Margaret Kiley CEDAM, The Australian National University Margaret.kiley@anu.edu.au

2. Limits of this workshop… This is ‘just’ the theory To get the practice in your own university you need to contact your Graduate Studies authority This workshop does not specifically focus on doctorates in art, design and performance where an exhibition or equivalent plus exegesis is the norm

3. In pairs/small groups discuss One thing I don’t really understand about the examination process is… One of my biggest fears about the examination process is… Something I am hoping to learn from today’s session is… Something I hope to contribute to today’s session is…

4. Preparing for Examination What are your University’s expectations of a PhD or Masters thesis? What is the difference? Who selects examiners? Does the candidate have a say? Can a candidate include published work? If so, how? Is there a word limit? What happens re multi-disciplinary research? How strategic can supervisors and candidates be with the selection of examiners e.g. is it a good idea to appoint an examiner who might be a future employer? What are the University’s rules on conflict of interest between candidate, examiner, and supervisor?

5. Overview Research re the selection of examiners How examiners examine theses Discussion

6. Selection of Examination (Kiley, 2009) Recent research suggests that the selection of examiners varies depending on the perceived quality of the candidate and the dissertation. Experienced supervisors report that there are two sorts of criteria that they use: Professional / academic Personality attributes. The aim is to find examiners who are knowledgeable in the area and who will give a fair and balanced opinion.

7. Professional / academic considerations Topic/methodology fit Understanding of the system: country, type of doctorate (professional doctorate, exegesis) Examiners outside the academy Experience versus inexperience (avoid retirees with too much time on their hands) Availability

8. Personality Issues High standards but fair (avoid the pedants) Intellectual courtesy and respect (avoid the Smart Alecs) Reliability (avoid the ‘gunnas’)

9. Experienced Examiners report that they… (Mullins & Kiley, 2002) Expect the student to pass as they open the thesis Are very reluctant to fail a student with most experiencing considerable distress if they do so Come to a decision about the quality of a PhD by about the end of Chapter 2 Have a formative rather than summative view of thesis examination Believe that there is a risk attached to sending theses to inexperienced examiners

10. Experienced Examiners appear to… Be fiercely independent in their views Hold varying views about the purpose of the PhD. (Is it the thesis or the student being examined?) Consider professional duty as the main reason for examining, followed by the fact that they are going to be needing examiners for their own students! Devote considerable time to examining each thesis Have surprisingly inclusive approaches to methodology/ paradigm Demonstrate few discipline differences in their responses, other than regarding publications

11. Inexperienced Examiners (Kiley & Mullins, 2004) Report a high level of confidence in their ability to examine (which is not always reflected in what they say in response to other questions) Suggest experience comes from supervising & examining Honours students and theses and reviewing manuscripts Adopt a similar approach to the process of examining as more experienced colleagues, although they are more likely to focus on the components of a PhD rather than the whole

12. Inexperienced Examiners … At a surprisingly high rate, wanted to fail their first thesis or said it was ‘awful’ Follow institutional criteria more than experienced colleagues Felt (some of them) they were being examined too Suggest a main difficulty is inability to benchmark See role as maintaining standards and performing their summative assessment role ‘correctly’

13. Is there a difference? From work of Trafford (2003) from 130 vivas it was possible to determine that: Experienced examiners tended to ask questions that can be defined as ‘Defending doctorateness, contributing to knowledge, critique of research, synthesizing concept’ Inexperienced examiners tended to ask more ‘technical’ questions

14. Trafford’s Categorisation Innovation and Development HIGH C. Questions generally related to issues such as research question, choice of topics, location of study D. Defending doctorateness, contributing to knowledge, critique of research, synthesizing concept A. Types of questions include resolving research problems, content, structure B. Implications, awareness of, and familiarity with wider literature Scholarship & Interpretation LOW LOW HIGH

15. Strategies for examining There is no one way, but most: Begin by reading the Abstract, Acknowledgements, Introduction & Conclusion to gauge the scope of the work and whether what candidates say they are going to do is actually done Look at the references to see what sources have been used and whether they need to follow up on any of them, and to see if they have been referenced Read from cover to cover taking detailed notes, finally go back over the thesis to check whether their questions have been answered or whether their criticisms are justified Tip

16. Strategies for examining There is no one way, but most: Begin by reading the Abstract, Acknowledgements, Introduction & Conclusion to gauge the scope of the work and whether what candidates say they are going to do is actually done Look at the references to see what sources have been used and whether they need to follow up on any of them, and to see if they have been referenced Read from cover to cover taking detailed notes, finally go back over the thesis to check whether their questions have been answered or whether their criticisms are justified

17. Analysis of reports demonstrates that… (Kiley, 2004) A ‘good’ thesis has: Critical analysis & argument Confidence & a rigorous, self- critical approach A contribution to knowledge Originality, creativity & a degree of risk taking Comprehensiveness & scholarly approach Sound presentation & structure Sound methodology A ‘less than ideal’ thesis has: Too much detail with lack of analysis Lack of confidence, energy & engagement by the candidate Lack of argument and rigour Shoddy presentation (typos etc) Lack of critique of own analysis/ sweeping generalisations based on opinion rather than analysis Inadequate or poorly expressed methodology & scope

18. A very experienced supervisor (Lees, 2008)  Failing a PhD is an awful experience….It can take months, and many re-readings, to finally come to that decision.  On the other hand… “It’s a delight to deal with a thesis which has a clear, concise, research question; a simple structure; a research design which was decided upon before the data was gathered; a literature review which relates to the topic; and a logical progression from ‘Results’ through ‘Discussion’ to ‘Conclusion’. These get marked briskly!”

19. Discussion and Questions In small groups list the main qualities of a ‘good’ doctoral thesis Through discussion then decide which are the top three qualities and why QUESTIONS?

20. Resources Kiley, M. (2009). You don't want a smart alec: Selecting examiners of doctoral dissertations Studies in Higher Education 34(8) Kiley, M. (2004). What examiners' comments can tell us about the postgraduate learning environment. In C. Rust (Ed.), Improving student learning: Theory, research and scholarship (pp. 213-222). Hinckley, Leicestershire: The Oxford Centre for Staff and Learning Development. Kiley, M., & Mullins, G. (2004). Examining the examiners: How inexperienced examiners approach the assessment of research theses. International Journal of Educational Research, 41(2), 121-135 Lees, B. (2007). Theses I have marked. Presentation at the Summer ACT Inter- University research workshop, ANU. Mullins, G. & Kiley, M. (2002) ‘It’s a PhD, not a Nobel Prize’: How experienced examiners assess research theses. Studies in Higher Education 27(4) pp. 370-386 Research Supervision@ANU http://researchsuper.cedam.anu.edu.au SORTI web site at the University of Newcastle has information on examining theses, especially in the performing/visual arts http://www.newcastle.edu.au/centre/sorti/publications.html http://www.newcastle.edu.au/centre/sorti/publications.html Trafford, V. (2003) Questions in doctoral vivas: Views from the inside, Quality Assurance in Education 11(2) pp 114-122