1. Webinar Series Competency based assessment 2 Aug 2017: 07:00AM UTC/GMT
Assoc. Prof. Claire Palermo
Monash University
Just to let you know:
By participating in the webinar you acknowledge and agree that:
The session may be recorded, including voice and text chat communications (a recording indicator is shown inside the webinar room when this is the case).
We may release recordings freely to the public which become part of the public record.
We may use session recordings for quality improvement, or as part of further research and publications.
Webinar Hosts
Professor Geoff Crisp, PVC Education, University of New South Wales
g.crisp[at]unsw.edu.au
Dr Mathew Hillier, Office for Learning & Teaching, Monash University
mathew.hillier[at]monash.edu
e-Assessment SIG

2. Competency based assessment Taking a programmatic approach to assessment to ensure graduate employability
Assoc. Prof. Claire Palermo
Deputy Director, Monash Centre for Scholarship in Health Education
&
Department of Nutrition, Dietetics and Food
Faculty of Medicine, Nursing and Health Sciences
A bit about me
Fellowship – best practice CBA in medical education was programmatic

3. What comes to mind when you hear programmatic approach to assessment?

4. Competency Based Education – Graduate Employability
Consumer needs
Workplace system
Graduate
Capabilities
Curriculum
Assessment
Competency based education framework provide the foundation from which to develop the workforce.
Competencies – outcomes to be achieved by the profession
Should be informed by population health needs and health system needs
Competencies inform – curricula and assessment
This framework applies for preparation for the workforce but also to support the qualified workforce
Dietitics we were one of the first professional to develop national competency standards (in 1992).
However, this approach has not necessarily been fully transferred to the preparation of and continuing education of dietitians to be outcomes focussed.
Competency based education represents a fundamental shift in educational philosophy and perspective.
A number or factors that have impeded full implementation.
If we are to truly have an impact on health and nutrition we need to strengthen our approaches.
Today in addition to exploring the new competency standards I hope to present to you accepted “truths” that resonate in the profession
I aim to dispel these truths by presenting evidence to the contrary.
By letting go of these accepted practises and embracing change we may truly be able to transform the profession of the future.
Frenk J. Health professionals for a new century: transforming education to strengthen health systems in an interdependent world. Lancet. 2010

5. Dreyfus & Dreyfus Development Model
…the routine and considered use of communication, knowledge, technical skills, clinical reasoning, emotions, values and reflection that are applied to a practice problem with the aim of improving outcomes.
Expert/
Master
Proficient
Competent
Advanced Beginner
The Dreyfus model of skills acquisition describes that development of competence.
Competence is the routine and considered use of communication, knowledge, technical skills, clinical reasoning, emotions, values and reflection that are applied to a practice problem with the aim of improving (in the case of dietetics) nutrition outcomes.
In the development and maintenance of competence we must recognize that skills develop over time, practice, experience and training.
We expect different levels of competence for different skills and our journey to the development of skills is different.
The reality is that competency development is linear at all as suggested by this model but rather looks something like this… or this…
However, it is a point on the spectrum of performance. We should be striving towards excellence – competence isn’t enough!
What is competence?
“the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values and reflection”
(Epstein and Hundert 2002)
“is a point on the spectrum of improving performance” (Khan, 2012)
What is performance?
“a composite of (clinical) cognitive, psychomotor and affective abilities (attitudes) of the individual alongside their non-clinical skills like team-work, situational awareness… personality traits.”
(Khan, 2012)
Novice
Time, Practice, Experience
Dreyfus SE and Dreyfus HL. 1980
Carraccio CL et al. Acad Med 2008;83:761-7
Acknowledgements: Eric Holombe

6. What is a programmatic approach to assessment?
Focus is on outcomes
Assessment looks at performance and application of knowledge and skills that are needed for employment
Every student is expected to achieve explicit standards which are criterion referenced
Differences in trajectory of progress during training are expected (developmental)
Formative feedback to help students improve performance is a priority
Learner responsibility and accountability are stressed

7. Programmatic Assessment - Conceptually
Van der Vleuten et al (2012)

8. Any questions?

9. Programmatic Assessment - Practically
Focus is on outcomes
Define outcomes e.g. Competency standards, graduate attributes
Assess a broad range of capabilities
Assessment to focus on outcome rather than process

10. Programmatic Assessment - Practically
Assessment looks at performance and application of knowledge and skills that are needed for employment

11. Programmatic Assessment - Practically
Every student is expected to achieve explicit standards which are criterion referenced

12. Programmatic Assessment - Practically
Differences in trajectory of progress during training are expected (developmental)

13. Programmatic Assessment - Practically
Formative feedback to help students improve performance is a priority

14. Programmatic Assessment - Practically
Learner responsibility and accountability are stressed

15. Twelve Tips for programmatic assessment
Develop a master plan for assessment
Develop examination regulations that promote feedback orientation
Adopt a robust system for collecting information
Assure that every low-stakes assessment provides meaningful feedback for learning
Provide mentoring to learners
Ensure trustworthy decision-making
Organise intermediate decision-making assessments
Encourage and facilitate personalised remediation
Monitor and evaluate the learning effect of the programme and adapt
Use the assessment process information for curriculum evaluation
Promote continuous interaction between the stakeholders
Develop a strategy for implementation
VAN DER VLEUTEN et. al. Twelve Tips for programmatic assessment. Med Teach. 2015, 37: 641–646.

16. What is required for:
A health professional to manage a patient with chronic disease?
A teacher to deliver a multiplication maths lesson to year 2 students?
An engineer to design structural steel for a new home?

17. Provide effective care to a person with chronic disease
Example of designing a program of assessment What does it take for a health care professional to:
Provide effective care to a person with chronic disease

18. Example of designing a program of assessment
Work-place assessors feedback
Work-place assessors feedback
Work-place assessors feedback
Case study report on patient with diabetes
Oral communication exam
Learning experiences

19. Challenges in Implementation of Programmatic Approaches
Stakeholders
Assessment policy and procedures
Unit/Course structures
▪ Resistance to integrating curriculum – Silos and resistance to new approach to thinking
▪ A desire to use A single-best instrument
▪ Institutional requirements of assessment – Accrediting body requirements – Formative vs. Summative assessment requirements
▪ Resource and curriculum renewal requirements (time and resources) – Less is more (do a few things well) ▪ Student resistance to change (buy in essential)

20. Programmatic approaches to assessment supports
Summary
Programmatic approaches to assessment supports
A focus on outcomes/authentic assessment
Holistic judgement on criteria linked to capability
Longitudinal, multiple pieces of assessment evidence
Feedback central
Student directed/led

21. Assoc. Prof. Claire Palermo

22. Webinar Series Webinar Session feedback Recording available
With thanks from your hosts
Professor Geoff Crisp, PVC Education, University of New South Wales
g.crisp[at]unsw.edu.au
Dr Mathew Hillier, Office of Learning & Teaching Monash University
mathew.hillier[at]monash.edu
Recording available
e-Assessment SIG