1. The Competency Assessment Toolkit for Professional Psychology
Catherine L. Grus, Ph.D.,
Associate Executive Director for
Professional Education and Training
Education Directorate
American Psychological Association

2. THANK YOU Many thanks to Nadya Fouad, University of Wisconsin
Nadine Kaslow, Emory University
Benchmarks Workgroups
Assessment “Toolkit” Workgroup

3. Overview Considerations in the assessment of competence
Development of the Toolkit
Putting the toolkit into practice

4. Competence …
“ … the habitual and judicious use of communication, knowledge, technical skills, clinical reasoning, emotions, values, and reflection in daily practice for the benefit of the individual and community being served.”
From Epstein and Hundert (2002)

5. Competence …
… is developmental depending on stage of professional functioning
… context dependent
… executed according to ethical principles, guidelines, standards, and values of the profession and
… requires public verification

6. Competencies … are elements of competence that are …. Observable
Measurable
Containable
Practical
Derived by experts
And flexible
are elements of competence that are ….

7. Assessment of Competence - Considerations
Use of both “formative” and “summative” evaluation methods
Authentic evaluations
Real world tasks, meaningful activities
[High] Fidelity: Degree to which
the assessment represents
actual performance

8. Employ a range of perspectives (multi-informant) and methods
Challenge of evaluating attitudes and values
Psychometric issues
Feasibility/costs
Lifelong self-assessment

9. What methods are commonly used to assess students/ trainees?
In class examinations and course “grades”
Faculty/supervisor completes rating scale
Letters of recommendation
EPPP, state jurisprudence exams

10. Developing Faculty/Supervisor Competencies
Training in “difficult conversations”
How to “critique constructively”
Develop awareness of potential dual roles in evaluation and how that could influence results
Benefits of having other faculty evaluate students’ performance
Explicit obligations to make evaluations accurate
Grade inflation
Letters of recommendation to “market” vs. evaluate
Knowledge of a range of assessment tools, access to resources “toolkits”

11. Competency Assessment for “Toolkit” for Professional Psychology
Charge from APA Board of Educational Affairs: Develop a “Toolkit” for professional psychology
Purpose: Promote broader implementation of competence assessment and provide information about application of assessment methods to the assessment of competence
Coordinated with Benchmarks Work Group

12. Benchmarks Core Competencies
Foundational Competencies:
Professionalism
Reflective practice
Scientific knowledge and methods
Relationships
Individual and cultural diversity
Ethical and legal standards and policy
Interdisciplinary systems
Functional Competencies:
Assessment
Intervention
Consultation
Research and evaluation
Supervision
Teaching
Administration
Advocacy

13. How are the Benchmarks Organized?
Core Foundational and Functional Competencies
Essential Component: what are the critical elements of/what knowledge/skills/attitudes that make up this competency?
Behavioral Anchor: what would it look like if you saw it (essential component)?

14. Components of Toolkit Background and Introduction
Assessment Method Fact Sheets
Grid of Assessment Methods and Competencies/Essential Components Best used for

15. Fact Sheet: Portfolio Review
Description
A portfolio is a collection of products, gathered by the person being assessed …
Application
Competency
Applies to1
Predominant Use of2
Developmental Level3
Professionalism E
F,S
2,3,4
Reflective Practice
1,2,3,4
Implementation
Deciding on form (e.g., web-based or hard copy)
Psychometric Properties
Reliability has not been well established due to the variable content included
in a portfolio
Strengths
Provides a low cost assessment strategy
Challenges
Requires intense commitment of time and labor intensive for the assessors
and the person being assessed
Future directions
Create and implement training modules for assessors and people being
assessed in the portfolio assessment strategy

16. Grid of Possible Assessment Methods (Toolkit)
Annual review
Case reviews
Process/outcome data
Rating Forms
Consumer surveys
OSCE
Portfolio reviews
Ratings Performance
Record reviews
Self-assessment
Simulations/role plays
Stand. patient interview
Stand. oral exams
Written exams
360 Evaluations
Intervention
Knowledge interventions 2 3 1
Intervention planning
Skills
Implementation
Progress evaluation

17. Assessment Methods in Toolkit
360 evaluation
Portfolio
OSCE
Structured Written & Oral Exams
Case Presentation
Simulation/Role Play
Competence Evaluation Rating Form
Self-Assessment
Ratings of live or recorded performance
Standardized Client Interview
Client/Patient Process/Outcome Measure
Consumer Satisfaction Survey
End of Rotation Performance Review

18. Putting it all Together …
Benchmarks
Toolkit
Using the Toolkit and Benchmarks to Guide the Assessment of Student Learning Outcomes

19. Identify expected student learning
outcomes using competency model (e.g., Benchmarks)
Expected level of competence attainment and training plan developed with trainee
Select assessment method(s)
Student demonstrates problems with competencies acquisition
Remediation plan developed
Formative assessment
Student demonstrates emerging competencies
Training plan continues
Summative evaluation to assess readiness for next level of training

20. Identify expected student learning
outcomes using competency model (e.g., Benchmarks)

21. Program Outcome: Students will demonstrate the following intervention (functional) competencies prior to going on internship.
Knowledge of scientific, theoretical, empirical and contextual bases of intervention, including theory, research, and practice
Formulates and conceptualizes cases and plan interventions utilizing at least one consistent theoretical orientation
Clinical skills
Implements evidence-based interventions that take into account empirical support, clinical judgment, and client diversity (e.g., client characteristics, values, and context)
Evaluate treatment progress and modify treatment planning as indicated, utilizing established outcome measures

22. Identify expected student learning
outcomes using competency model (e.g., Benchmarks)
Select assessment method(s)

23. Grid of Possible Assessment Methods (Toolkit)
Annual review
Case reviews
Process/outcome data
Rating Forms
Consumer surveys
OSCE
Portfolio reviews
Ratings Performance
Record reviews
Self-assessment
Simulations/role plays
Stand. patient interview
Stand. oral exams
Written exams
360 Evaluations
Intervention
Knowledge interventions 2 3 1
Intervention planning
Skills
Implementation
Progress evaluation

24. Identify expected student learning
outcomes using competency model (e.g., Benchmarks)
Expected level of competence attainment and training plan developed with trainee
Select assessment method(s)

25. A. Knowledge of Interventions Readiness for Practicum
Developmental Level
A. Knowledge of Interventions
Readiness for Practicum
Readiness for Internship
Essential Component:
Basic knowledge of scientific, theoretical, and contextual bases of intervention and basic knowledge of the value of evidence-based practice and its role in scientific psychology
Behavioral anchor:
Articulates the relationship of EBP to the science of psychology
Identifies basic strengths and weaknesses of intervention approaches for different problems and populations
Knowledge of scientific, theoretical, empirical and contextual bases of intervention, including theory, research, and practice
Behavioral Anchor:
Demonstrates knowledge of interventions and explanations for their use based on EBP
Demonstrates the ability to select interventions for different problems and populations related to the practice setting
Investigates existing literature related to problems and client issues
Writes a statement of one’s own theoretical perspective regarding intervention strategies

26. Identify expected student learning
outcomes using competency model (e.g., Benchmarks)
Expected level of competence attainment and training plan developed with trainee
Select assessment method(s)
Student demonstrates problems with competencies acquisition
Remediation plan developed
Formative assessment
Student demonstrates emerging competencies
Training plan continues

27. Remediation Plan Competency Domain/ Essential Components Problem
Behaviors
Expectations for Acceptable Performance
Trainee’s Responsibilities/
Actions
Supervisors’/
Faculty Responsibilities/
Timeframe for
Acceptable
Performance
Assessment
Methods
Dates of
Evaluation
Consequences
for Unsuccessful Remediation

28. Identify expected student learning
outcomes using competency model (e.g., Benchmarks)
Expected level of competence attainment and training plan developed with trainee
Select assessment method(s)
Student demonstrates problems with competencies acquisition
Remediation plan developed
Formative assessment
Student demonstrates emerging competencies
Training plan continues
Summative evaluation to assess readiness for next level of training

29. For more information:
Coming soon:
Kaslow, N. J., Grus, C. L., Campbell, L. C., Fouad, N. A., Hatcher, R. L., & Rodolfa, E. R. (in press) Competency Assessment Toolkit for Professional Psychology. Training and Education in Professional Psychology.