1. Structured Oral Examination as an assessment tool
Dr Samira Rahat Afroze
Internal Medicine Dept
IMC & BIRDEM General Hospital

2. What will be covered Introduction to oral examination
Structured oral examination (SOE)
Oral examinations : Bangladesh perspective
The process of conducting SOE
Do’s & Don’ts

3. The oral examination / viva A traditional form of assessment
Face-to-face discussion
One or more examiners ask the candidate questions
Allows assessment of
intellectual abilities
professional attitude
communication skill
appearance
confidence
Cox KR, Ewan CE. The Medical teacher. Edinburgh : New York:
Churchill Livingstone; 1982.

4. Where applied in medical field?
Undergraduate medical education
Postgraduate medical education
Job interview
Today…. I shall be focusing on undergraduate oral examinations

5. Blooms Taxonomy
3 Domains of Education

6. BLOOM’S TAXONOMY
Oral examination is aimed at assessing mainly the higher degree of cognitive and problem solving skills of the candidates

7. Educational Handbook for Health Personnel (WHO)
Advantages
Direct personal contact with candidates
Allow candidate to formulate his own replies without
cues
Opportunity to take mitigating circumstances into
account
Candidate can be asked how he arrived at an answer
depth of knowledge
Flexibility in moving to & fro a candidate’s strong
points to weak points
Simultaneous assessment by ≥ 2 examiners
Cox, 1982 ; Gibbs et al., 1993 ;Gibbs et al., 1988; Wakeford et al., 1995; Jolly & Grant, 1997; Wass et al., 2003

8. Lack of standardization; objectivity & reproducibility of results
Disadvantage (WHO)
Lack of standardization;
objectivity & reproducibility of results
Shortage of trained examiners
Costly in terms of professional time
Difficulty in sampling from curriculum
Costly in terms of professional time in relation to the limited value of information yielded.
better tested in written examinations.
Largely test factual knowledge
Permit favoritism & biasness
Suffer from undue influence of irrelevant factors

9. Traditional oral exam (TOE)
Subjective test
Fails to assess the cognitive and problem solving skills properly
The atmosphere of the examination is often threatening
Independent marking by a pair of examiners is rarely done

10. Poor Reliability ? Validity
Different assessment of different content areas
Different difficulty levels of the questions asked
Varying levels of prompting or help provided
? Validity

11. Structured oral examination
And… This is where stuctured oral examination comes in

12. “If the decision is taken to retain the orals in the examination system, there are several suggestions regarding best practice that should be followed.”

13. Structure the oral on clinical scenarios
with well-defined learning outcomes
2) Use a number of orals
“reliability when using a number of orals is better
than the reliability of a single oral examination”
Use a number of examiners
Careful selection of examiner and paring a new examiner with a more experienced person.

14. 4) Ask all candidates the same questions
improve sampling of the syllabus
avoid overlap between the orals and other
components of the assessment
Care should be taken to

15. Use descriptors, rubrics or criteria for answers
Pre-planned blue print
provide clear written guidelines about
asking questions
what is & is not an acceptable answer
Rating scale must be standardized

16. Test Matrix CVS Resp Neuro Nephro HBS GIT Endo Psy Derm Knowledge
(Define) x
Comprehension
(Interpretation)
Analysis
(Differentiate)
Synthesis
(problem solving)
Evaluation
(Criticize / defend)
Define heart failure
Interpret : Chest pain & SOB, ECG & CXR findings
Analysis : How will you clinically differentiate from Acute severe asthma?
Synthesis: What steps will you take to manage this patient?
Evaluation: SOB is not responding to treatment, how will you evaluate?

17. 6) Train the examiners All examiners must be familiar with
the rating scale.

18. Bangladesh perspective

19. Written examination format SAQ & MCQ
Undergraduate medical curriculum of extensive modification of the assessment system
Written examination format
SAQ & MCQ
with 10% mark added by formative assessment.
Traditional practical & oral assessment
OSPE/OSCE and SOE
modified to

20. MBBS professional examinations: SOE implemented
Pre-clinical : Jan, 2005
Para-clinical :Jan, 2007
Clinical : July, 2009

21. SOE (curriculum recommendations) Pre-examination meeting of examiners Question of varying degrees of difficulty recall 50-60% interpretative 20% problem solving questions 10-20% Correct answer & rating scale Decisions made jointly Questions should be typed on a card
help to neutralize the assessor effect to some extent

22. Procedure Cordial atmosphere Two boards : 2+2 = 4 examiners
Equal time : 15 minutes
Timer or stop watch
Candidate pick up the cards randomly from a box
/ boxes of defined domain
After answering the questions the examiner place a tick in an appropriate box on a prepared rating scale
candidates feel secured and tension free environment

23. How IS it PRACTISED ? Research Ibrahim Med. Coll. J. 2015; 9(1): 1-10
To evaluate critically the current status of
structured oral examination (SOE) format as practiced in the professional MBBS examination
Views of the faculties regarding the concept of SOE as an assessment tool.

24. SOE as assessment tool
in undergraduate medical curriculum
is not achieved
&
it is not appropriately implemented

25. How to improve? Policy makers : regular and intensive training program
Continuous studies
determine the reason(s)
its suitability in our medical curriculum
redesigning the SOE as an assessment tool
For faculty development

26. The
other
Do’s

27. Steps of Oral Examination
Phrase question clearly. Allow sec…… Encourage student to respond. Acknowledge correct answer. Ask lower to higher cognitive order questions.

28. If still fails to answer …. Shift to the next question
A student often cannot understand the question.
Is the question clear to you?
Do you want me to rephrase the question?
Which part you did not understand?
Is it too difficult?
Studies have shown that even modest increase of wait time, the correctness improves.
Allow time
If still fails to answer …. Shift to the next question

29. Feedback
Always an important aspect of upgrading validity & reliability of any assessment tool.
Student : better if given individually
Provides opportunity to find out the gaps in learning & teaching

30. What examiners
should avoid
during oral exams

31. interrupt
provide answer
criticize

32. Conclusion
Structured oral examinations can be a useful component of assessment
It has a special place in medical education because…….
it assess the communication skill of the candidate
Competency of a doctor is also judged by way he/she talks

35. Give an example

37. Weingarten MA, Polliack MR, Tabenkin H, et al. 2000
Weingarten MA, Polliack MR, Tabenkin H, et al Variations among examiners in family medicine residency board oral examinations. Med Educ. 34:13–17
best practice-to-date oral examinations have been criticized on the basis that inter-observer agreement does not necessarily imply reliability because two examiners sitting together during an examination inevitably affect each others assessment leading to a tendency towards agreement

38. Spike N, Jolly B. 2003. Are orals worth talking about
Spike N, Jolly B Are orals worth talking about? Medical Education. 37: 92–93.
1. there was potential for adverse educational impact on the candidates; 2. there was no allowance for compensation and; 3. it could be unsuitable for borderline candidates.

39. Is there a need of
oral examinations?

40. If we select MCQs to assess knowledge (knows) Extended matching items or varieties of short-answer questions to assess application of knowledge (knows how) OSCE to assess ‘‘shows how’’ direct observation of practice to assess ‘‘does’’
Davis MH, Karunathilake I The place of the oral examination in today’s assessment systems. Medical Teacher. 27(4):

41. Giving up such a tradition is not easy And
It is likely that the oral examination will continue to have a place in assessment systems for some time to come.
It has been decided that…..
If structured properly, Oral examinations can be a useful assessment

42. One example I found online ………. was on Post-graduate oral exam

43. Verbal scores were converted to numerical scores,
Outstanding = 9
Excellent = 8
Good = 7
Satisfactory = 6
Bare pass = 5
Not adequate = 4
Unsatisfactory = 3
Poor = 2
Dangerous = 1

44. Descriptive study, 2007, 9 centers
36 examiners in 18 SOE boards, 26 interviewed
SOE boards were observed with a checklist
Analysis of the questions
97% recall type
3% interpretation & problem solving types

45. 38% had no clear idea regarding learning objectives
Examiners response
38% had no clear idea regarding learning objectives
100% regarding test blueprint
Model answer of SOE questions was not prepared
Observation
56% arrived late
22% absent for prolonged period examiners
3% marking the written scripts
14% showed abusive behaviors
44% rating scale not used
94% boards scored prompted answer
Equal time: not maintained
80% examiners agreed there should be prior preparation
rather scored in traditional way at the end