1. Development of a Rating Scale to Assess Geropsychology Practice Competencies
Michele J. Karel, PhD
VA Boston Healthcare System
Harvard Medical School
For the CoPGTP Task Force on Geropsychology Competency Assessment:
Michele Karel, Jeannette Berman, Jeremy Doughan, Erin E. Emery, Victor Molinari, Sarah Stoner, Yvette N. Tazeau, Susan K. Whitbourne, Janet Yang, Richard Zweig
APPIC Conference, April 18, 2009

2. Objectives
Increase awareness of efforts to define and evaluate competencies for geropsychology practice
Consider challenges for designing and implementing a competency rating scale
Utility in internship and postdoc settings
Address ongoing challenges and directions for work in this area

3. Importance of Professional Geropsychology Training and Evaluation
Aging population
Workforce shortages for geriatric health care, across disciplines
Relatively few psychologists have formal training in geropsychology
Increasing numbers will be working with older adults
How to evaluate competence, and define training needs, for psychological practice with older adults
How to help training programs, and post-licensure psychologists, expand opportunities for geropsychology training

4. Growth of Professional Geropsychology
Growing number of grad, intern, and fellowship programs offering geropsychology training
Growth of research, journals, professional organizations
Several recent milestones:
APA Office on Aging, established 1998
APA Guidelines for Psychological Practice with Older Adults (2004)
CRSPPP recognition of gero as proficiency area, 1998 and 2005
Pikes Peak Model for Geropsychology Training, National Training Conference
Pikes Peak Training Model* defined Attitude, Knowledge, Skill competencies for geropsychology practice
How to evaluate professional geropsychology competence?
*Knight, B.G., Karel, M.J, Hinrichsen, G.A., Qualls, S.H., & Duffy, M. (in press). Pikes Peak model for training in professional geropsychology. American Psychologist.

5. Project Initiation
Council of Professional Geropsychology Training Programs (CoPGTP)
Formed after 2006 Pikes Peak conference
For training programs at all levels
CoPGTP undertook to explore methods to evaluate Pikes Peak competencies
CoPGTP established a Task Force
Agreed to start with goal of developing a competency rating scale

6. Project Goals Develop an evaluation tool that would:
Address the Pikes Peak geropsychology attitude, knowledge and skill competencies
Be useful for supervisor evaluation of trainees
Formative and summative evaluation
Be useful for self-evaluation by learners
Be useful across levels of training
Graduate practicum, internship, fellowship, post-licensure
Be useful to help define ongoing training goals

7. Pikes Peak Geropsychology Practice Competencies
Competency definitions informed by:
APA Guidelines for Psychological Practice with Older Adults (2004)
Cube Model for Competency Development (Rodolfa et al., 2005)
Psychology foundational and functional competencies develop over course of training career
2006 Pikes Peak Conference process:
Working groups address 6 competency domains
(separate working groups address 4 levels of training)
Iterations of competencies lists during conference, and afterwards via listserv discussion

8. Pikes Peak Competencies
Are aspirational
Aim to facilitate as many psychologists as possible to work effectively with older adults, not to pose barriers to joining the field
Competencies are aimed at level of practice of a newly licensed psychologist who has finished one year of geropsychology postdoctoral training
Competencies not geared to level of an “expert”, but to someone who practices with wide range of older adults, families, systems

9. Pikes Peak Competencies: Attitudes
Work within one’s scope of competence
Recognize how one’s beliefs about aging and older adults may affect practice
Appreciate diversity among older adults, and interactions between age/cohort and other aspects of individual diversity
Seek continuing education, supervision, consultation

10. Pikes Peak Competencies: Knowledge Base
General knowledge about adult development, aging, and the older adult population
Foundations of professional geropsychology practice
Assessment
Intervention
Consultation

11. Pikes Peak Competencies: Skills
Professional Geropsychology Functioning
i.e., foundational skills applied to geropsychology practice
Assessment
Intervention
Consultation
Note: Skills for functional domains of research, supervision, and management are important for growth of geropsychology, but were not included in Pikes Peak core competencies for practice with older adults

12. Pikes Peak Competencies: Settings of Care
Geropsychologists should be able to deliver services effectively in range of (at least 2) settings, such as:
Outpatient mental health services
Outpatient primary care/medical settings
Inpatient medical service
Inpatient psychiatric service
Long-term care settings including nursing homes, assisted living facilities, home care, day programs
Rehabilitation settings
Hospice
Others

13. CoPGTP Competency Evaluation Task Force Methodology
~Monthly conference calls, 9/07 to 7/08
Reviews of relevant, informing materials
Small working groups define “behavioral anchors” for each competency domain
Iterative reviews of all scale items by larger group
Group discussion, selection, editing of:
Rating scale/anchors and vignette
Introduction to the evaluation tool
Task Force and other CoPGTP members try out the scale to evaluate: a student, oneself, or both

14. Informing Materials: Psychology Competencies Assessment
Guiding principles and recommendations for the assessment of competence
(Kaslow et al., 2007 and other publications of APA Task Force on Assessment of Competence)
Assessment of Competency Benchmarks Work Group
A developmental model for defining and measuring competence in professional psychology
Joint effort of APA BEA and CCTC
The Practicum Competencies Outline
Hatcher & Lassiter (2007), workgroups of ADPTC and CCTC

15. Informing Materials: Geropsych Competency Assessment
Example of evaluation tools used in various geropsychology practicum, internship, and postdoctoral programs
Also, examples of competencies and evaluation methods from:
Medicine – ACGME Outcome Project, 2000
Geriatrics, Palliative Care, Geriatric Social Work, Geriatric Psychiatry

16. Delineation of Geropsychology Knowledge and Skill Competencies
Each Pikes Peak competency domain (e.g., foundational geropsychology skills) is defined by 3-8 specific competencies (e.g., ethical practice, appreciation of diversity, self-reflection…)
We aimed to further specify each of these competencies with behavioral anchors
How would you know it if you saw it (in oneself or another)?
e.g., what does it really mean to “Relate effectively and empathically with older adults clients, families, and other stakeholders in a range of professional roles and settings?”
This approach leads to a very long evaluation tool!

17. Example: Delineating one of the foundational skill competencies

18. Example: Delineating one of the assessment skill competencies

19. Evaluation of Attitudes
Pikes Peak attitudes for competent geropsychology practice are embedded in the foundational skills
E.g., self-reflection re: biases/discomforts, awareness of limits of one’s competence, seeking consultation, seeking continued education
Attitudes are specified in tool’s introduction
For now, attitudes not evaluated separately
Perhaps they should be? How?

20. Rating Scale Developmental rating scale
Novice, Intermediate, Advanced, Proficient, Expert
Scale anchors conceptually complex
Adapted definitions from Hatcher and Lassiter (2007)
Included in anchor definitions how much supervision/consultation likely needed
Provided vignette and approach to that case to illustrate each level of competence
Instructions to rate each Pikes Peak competency, not each behavioral specifier (but can if desired)

21. Rating Scale Abbreviated scale anchors, printed on each page:
N = Novice: Possesses entry-level skills; needs intensive supervision
I = Intermediate: Has a background of some exposure and experience; ongoing supervision is needed
A = Advanced: Has solid experience, handles typical situations well; requires supervision for unusual or complex situations
P = Proficient: Functions autonomously, knows limits of ability; seeks supervision or consultation as needed
E = Expert: Serves as resource consultant to others, is recognized as having expertise

22. Rating Scale
Expected ratings vary considerably through levels of training
Practicum: Novice through Advanced
Internship: Novice through Proficient
Fellowship and post-licensure: Intermediate through Expert
Ideally, usable to evaluate a trainee, AND to evaluate oneself

23. Introduction to the Evaluation Tool
Reviews:
Purpose of tool
Pikes Peak competencies as aspirational
Application across range of geropsychology practice and training settings
Instructions for use
Including that evaluation should include the learner's perspective (self-assessment), observation of the learner's work, and regular supervision involving case discussion
Psychologists and trainees conducting self-assessments can evaluate their training and supervision needs in each area
Definition of rating scale anchors, and illustration of its use through a case vignette

24. Piloting the Evaluation Tool
Asked CoPGTP members, and ourselves, to complete the tool to evaluate a student or oneself
13 responses
6 CoPGTP members (other than us)
4 Task Force members
3 from Task Force members’ students/staff
Used tool to:
4 to evaluate a student
5 to evaluate oneself
2 evaluated oneself and a student
2 provided general feedback without specifying

25. Asked for Feedback: Asked folks to answer these questions:
How did you use the tool (to evaluate a student, or yourself?)
What was your general impression of using the tool?
Was the introduction to the tool clear…?
Was the rating scale (Novice, Intermediate, Advanced, Proficient, Expert) understandable…?
Was the length of the tool a problem? Did it adequately address breadth/depth of geropsychology competencies?
What do you think the best use of this tool will be?
Do you expect to use this tool? In what way?
Any other feedback?

26. Feedback
Most felt tool is very long, but worth it given attention to breadth and depth
A few felt it was too long for practical purposes
(Challenge – how to cut?)
Most felt introduction was clear, but needed clarification not to rate each specifier
Most liked the NIAPE rating scale

27. Feedback Concerns raised about the scale included:
Does it apply as well to knowledge as skills?
Does it work as well for practicum level training where students may be novices in most areas?
Is Expert rating too “loaded” (people not comfortable using that term?)
Can scale capture development over time?
If/how best to evaluate attitudes
Suggestion to have summary page
Strengths
Areas for growth
Ongoing training goals

28. Use of the Scale to Date
Disseminated to CoPGTP training programs and APA 12-2 and 20, PLTC listservs
Programs (grad, intern, and postdoc) are starting to use the tool for:
Initial training needs assessment with students
Mid-year/final supervisory evaluations
Student self-evaluation (encouraging “metacompetence”)

29. Use of the Scale to Date
Tool being used by some for program self-study and development
Defining training program objectives
Evaluating, in discussion with students and supervisors, how well program is addressing each competency domain
Helping to evaluate competence of new geropsychology supervisors and define their training goals
Being used in community/CE education programs on geropsychology

30. Uses for Intern/Postdoc Training
Along with APA standards, can help define rotation/track training objectives
Gero evaluation scale is rotation/track specific, and adjunctive to training program’s more general evaluation tools
Extremely helpful for initial training needs assessment because students come in with widely varying previous gero experience
Clinical, academic, research
Can tailor initial training plan and review over course of the training year/rotation
It is additional work at evaluation time for supervisor and student – not clear how strong a barrier this may pose

31. Study-in-the-Works
Aims: Examine utility and validity of the rating scale, initially for self-evaluation purposes
Which competencies rated as most vs least well-developed?
Do self-rated competencies relate to years of training or practice?
To settings and major activities of practice?
What feedback is provided after completing the scale?
Planning on-line survey
Of psychologists who work with older adults and geropsychology graduate, intern, posdoc students
Eventually, plan longitudinal study of supervisory evaluations of developing geropsychologists

32. Concerns/Challenges
Will clinicians see such a tool as helpful or somehow restrictive to practice?
Will supervisors/learners use such a long evaluation tool (even if can pick and choose domains of relevance)?
Can competencies be reduced/streamlined?
Can we boil down further “core” competencies? Are some more essential than others? (??)
May vary across settings/populations of older adults
Will tool be useful as self-evaluation, self-study aid?
Can competency ratings be linked to training resources?

33. Opportunities/Future Directions
Helpful for initial training needs assessment with a student, to guide training plan
May help to develop on-line version, then select only domains that are relevant to training program, rotation
Useful as guide to developers of CE programs?
Develop organized, sequential CE offerings?
Ultimately, plan to link competencies to training resources and opportunities
Working to keep such list updated at CoPGTP website
Expand toolbox of gero competence evaluation tools
Largest challenge - how to expand interest in, and opportunities for, training in professional geropsychology