1. A Program of Assessment for a Program of Study
Implementing an Integrated System for Programmatic Assessment of the MBBS Program
Dr Tracey Papinczak

2. Quality assurance in education
“the totality of systems, resources and information devoted to maintaining and improving the quality and standards of teaching, scholarship and research, and of students learning experience”
Quality Assurance Agency in UK Higher Education

3. Quality assurance system
Does the assessment
Align with best practice?
Have the following been undertaken?
Self-reflection
Examination of student feedback
Peer review
External review

4. 1. Best practice in assessment: Individual instruments
EBP
Reliable
Valid
Objective
Acceptable
Constructively aligned
Equitable and fair
Cost-effective
Time-efficient

5. 1. Principles: Best practice in assessment programs
Clarify what are expected standards (criteria)
Facilitate deep learning
Deliver high quality feedback
Positively motivate students
Support development of learning communities
Involve students in decision making
Deliver authentic assessment
Encourage reflection & self-assessment
Be paced to support continuous learning
Have strong validity and reliability for all individual assessment pieces

6. 2. Self-reflection Taking time Reflecting in-action and on-action
What is working?
What is not working?
How can we improve?
Where can we access support?

7. 3. Student feedback Student feedback to the Evaluation Unit
Concerns about subjectivity of consultant assessment years 3 & 4
Lack of time to complete some written examinations
Excessive assessment in some rotations
In yrs 1 & 2, too little assessment
Course evaluations administered by TEDI
2010 SET-C
greater feedback on assessment needs to be provided
iCEVAL
LO to assessment match needs greater clarity
Assessment requirements and marking criteria need to be provided at the start of the course
Assessment needs to be spread across the semester not concentrated at the end

8. 4. Peer review

9. 4. Peer review
Examples:
Peer review of examination papers in years 1 and 2
Peer review of PBL tutors in year 1
Peer review of lecturers in Discipline of General Practice

10. 5. External Review Australian Medical Council Review
Lack of centralised control over and management of assessment
Variable quality of assessment
Redundancies and gaps
Lack of examiner training

11. Planning for quality assurance
Quality Assurance of the Program of Assessment
Purposeful plan
Mapped to aims and goals
Gaps identified and addressed
Redundancies removed
Quality of each piece of assessment
EBP in assessment
Best choice for KSA

12. Threshold learning outcomes
ALTC – Learning & Teaching Academic Standards project Health, Medicine and Vet Science Six TL outcomes revised through stakeholder consultation Implications for assessment within all programs once TEQSA is underway (new Federal Government auditing agency within DEEWR)

13. Upon completion of their program of study, health graduates at professional entry level will be able to:
Demonstrate professional behaviours
Assess individual and population health status, and where necessary, formulate and implement management plans in consultation with patients/clients/carers
Promote and optimise the health and welfare of patients/clients and populations
Retrieve, critically evaluate, and apply evidence in the performance of health care activities
Deliver safe and effective health care in collaboration with other health care professionals
Reflect on current skills, knowledge and attitudes, and plan ongoing professional development activities.

14. Assessment in the MBBS Program
Threshold learning outcomes
(once TEQSA established)
Aims, goals and learning outcomes
UQ Graduate attributes

15. Graduate attributes In-depth knowledge & skills in the field of study
A comprehensive and well-founded knowledge in the field of study.
An understanding of how other disciplines relate to the field of study.
An international perspective on the field of study.
Effective Communication
The ability to collect, analyse and organise information and ideas and to convey those ideas clearly and fluently, in both written and spoken forms.
The ability to interact effectively with others in order to work towards a common outcome.
The ability to select and use the appropriate level, style and means of communication.
The ability to engage effectively and appropriately with information and communication technologies.

16. Graduate attributes Independence and Creativity Critical Judgement
The ability to work and learn independently.
The ability to generate ideas and adapt innovatively to changing environments.
The ability to identify problems, create solutions, innovate and improve current practices
Critical Judgement
The ability to define and analyse problems
The ability to apply critical reasoning to issues through independent thought and informed judgement
The ability to evaluate opinions, make decisions and to reflect critically on the justifications for decisions
Ethical and Social Understanding
An understanding of social and civic responsibility
An appreciation of the philosophical and social contexts of a discipline
A knowledge and respect of ethics and ethical standards in relation to a major area of study
A knowledge of other cultures and times and an appreciation of cultural diversity.

17. 3

18. 7

19. 35

20. Programmatic assessment
Focussed on the whole of a program of study
Collaborative & consultative
Charts students progress over time
Involves multiple perspectives
Measures knowledge, skills & affective domains
Has a strong workplace focus
Based on principles of best practice in education
Measures previously underserved areas of assessment
Avoids overlap & over-assessment of commonly measured KSAs

21. From the perspective that a program of study is more than the sum of each individual course, a program of assessment must be more than each individual measure.
The whole is greater than the sum of the parts

22. Research aim
To develop & to implement a comprehensive, integrated assessment program which explicitly maps to the three aims, seven goals & 35 learning outcomes of the MBBS Program and the six threshold learning outcomes of the TEQSA

23. Educational approach Dijkstra, Van der Vleuten & Schuwirth (2009)
Six dimension framework
Guides programmatic assessment
Holistic focus
Explicit mapping to program goals BUT
Establishing a set of goals for an assessment program in medical education

24. Method Dijkstra dimension Dimension elements Details of Implementation
Goals
Establishing a clear goal for the assessment program
Mapping of existing assessment items to aims, goals & learning outcomes;
Identification of gaps, redundancies & assessment practices & instruments inconsistent with best practice;
Consultation with stakeholders -assessment across all four years of the Program, potential barriers & other issues;
Establishment of a shared goal for the assessment program
Supporting the program
Technical support for quality assessment & involvement of stakeholders
Professional development and support;
Continued stakeholder input
Documenting the program
Assessment rules, blueprinting & planning;
Plotting the assessment processes for clarity & accountability
Literature review to identify best evidence for assessment in higher education & medical education;
Analysis of strategic plans & priorities at School, Faculty & University level;
Development of a holistic plan for assessment
Program in action
Collecting data, combining the data, reporting & decision-making
Implementation of the integrated assessment program
Improving the program
Constant monitoring & application of relevant research
Evaluation & action research
Accounting for the program
Internal & external review; Accountability to stakeholders
UQ & AMC Review;
Stakeholder feedback

25. Likely outcomes
Creation of comprehensive map: learning goals, aims & outcomes matched to current assessment processes
Establishment of holistic assessment program: sampling knowledge, skills & other key aspects of competence both summatively & formatively
Development of spectrum of complementary assessment strategies: sampling student achievement in an iterative manner but with increasing sophistication
Synthesis of well-informed, logical & highly defensible plan of assessment.
Enhanced collaboration & understanding between all stakeholders

26. Conclusion
The opportunity exists to produce, over the next three years, an innovative program of assessment that best serves the needs of all stakeholders & sets an example of the high quality which can be achieved by one of Australia’s premier medical schools.

27. References
Dijkstra, J., Van der Vleuten, C. & Schuwirth, L. (2009). A new framework for designing programmes of assessment. Advances in Health Sciences Education, Advance online publication. Doi: /s z.