1. Assessment of Trainee Competencies: Preliminary Psychometric Study
David J. Martin, Ph.D.
Harbor-UCLA Medical Center
Geffen School of Medicine at UCLA

2. Models of Competency
Cube Model of Competency Development (Rodolfa, Bent, Eisman, Nelson, Rehm & Ritchie, 2005)
Functional Competency
Foundational Competency
Stages of Professional Development

3. Models of Competency Cube Model (Rodolfa et al., 2005):
Functional Competency
Assessment-Diagnosis-Case Conceptualization
Intervention
Consultation
Research Evaluation
Supervision-Teaching
Management-Administration

4. Models of Competency Cube Model (Rodolfa et al., 2005):
Foundational Competency
Reflective Practice-Self Assessment
Scientific Knowledge-Methods
Relationships
Ethical-Legal Standards—policy
Individual-Cultural Diversity
Interdisciplinary Systems

5. Models of Competency Cube Model (Rodolfa et al):
Stages of Professional Development
Doctoral Education
Doctoral Internship/Residency
Post-doctoral Supervision
Residency/Fellowship
Continuing Competency

6. Models of Competency

7. Models of Competency Practicum Competencies Outline
Association of Directors of Psychology Training Clinics (ADPTC) Practicum Competencies Workgroup (February 27, 2004)
Council of Chairs of Training Councils (CCTC) Practicum Competencies Workgroup (March 25, 2004)
Final CCTC comments, November, 2004
Summarized by Hatcher & Lassiter (2006)

8. Models of Competency Practicum Competencies Outline
Levels of Competence
Novice
Intermediate
Advanced
Proficient
Expert

9. Models of Competency Practicum Competencies Outline
Areas of Competence
Relationship/Interpersonal Skills
With Patients/Clients/Families
With Colleagues
With Supervisors
With Support Staff
With Teams at Clinic
With Community Professionals
For the Practicum Site Itself

10. Models of Competency Practicum Competencies Outline
Areas of Competence
Skills in Application of Research
Intervention Skills
Psychological Assessment Skills
Consultation Skills/Interdisciplinary
Collaborations
Diversity (Individual & Cultural Differences)

11. Models of Competency Practicum Competencies Outline
Areas of Competence
Ethics
Professional Development
Practical Skills to Maintain Effective Clinical Practice
Professional Development Competencies
Metaknowledge/Metacompetencies
Development of Leadership Skills
Supervisory Skills

12. Harbor-UCLA Assessment
Psychological assessment skills
Extern
Fellow
Ability to select and implement multiple methods and means of evaluation in ways that are responsive to and respectful of diverse individuals, couples, families, and groups I P
Ability to utilize systematic approaches to gathering data to inform clinical decision making
Knowledge of psychometric issues and bases of assessment methods A
Knowledge of issues related to integration of different data sources

13. Harbor-UCLA Assessment
Competency Level
Unsatisfactory
Needs Improvement
Satisfactory
Strong
Outstanding
N/A 1.
Ability to select and implement multiple methods and means of evaluation in ways that are responsive to and respectful of diverse individual, couples, families and groups. P 1 2 3 4 5 6 2.
Ability to utilize systematic approaches to gathering data to inform clinical decision making. 3.
Knowledge of psychometric issues and bases of assessment methods. 4.
Knowledge of issues related to integration of different data sources.

14. Warning !
Attempts at complete review of the following slides may result in serious information overload!

15. Internal Consistency Fellows Externs Relationship/Interpersonal Skills
Items N α
Relationship/Interpersonal Skills
With Patients/Clients/Families 4 90
.938
203
.918
With Colleagues 3 85
.923
186
.902
With Supervisors 6 92
.958
206
.948
With Support Staff
N/A
With Teams at Clinic 2 77
.907
184
.900
With Community Professionals
For the Practicum Site Itself 74
.935
178
.928

16. Internal Consistency Fellows Externs Skills in Application of Research
Items N α
Skills in Application of Research 2 87
.910 --
Intervention Skills 8 70
.954 98
.959
Psychological Assessment Skills 7 55
.960 92
.950
Consultation Skills/Interdisciplinary Collaborations 60
.961
110
.933
Diversity (Individual and Cultural Differences) 3 91
.955
200
.939
Ethics 6 86
.972
188
.975
Professional Development
Practical Skills to Maintain Effective Clinical Practice 75
.938
161
.943
Professional Development Competencies 11 73
.964
136
Metaknowledge/Metacompetencies 5
.914
190
Development of Leadership Skills 24
.966
Supervisory Skills 56
.901

17. Descriptive Statistics
Fellows N
Min
Max
Mean
S.D.
Relationship/Interpersonal Skills
With Patients/Clients/Families 90
3.00
5.00
4.55
.609
With Colleagues 85
.619
With Supervisors 92
.622
With Support Staff 86
4.67
.583
With Teams at Clinic 77
.650
With Community Professionals 76
4.47
.739
For the Practicum Site Itself 74
4.53
.682

18. Descriptive Statistics
Fellows N
Min
Max
Mean
S.D.
Skills in Application of Research 87
3.00
5.00
4.30
.688
Intervention Skills 70
4.47
.612
Psychological Assessment Skills 55
4.32
.669
Consultation Skills/Interdisciplinary Collaborations 60
4.42
.654
Diversity (Individual and Cultural Differences) 91
4.49
.639
Ethics 86
4.52
.640
Professional Development
Practical Skills to Maintain Effective Clinical Practice 75
2.17
4.38
.700
Professional Development Competencies 73
4.50
.590
Metaknowledge/Metacompetencies
4.48
.580
Development of Leadership Skills 24
3.96
.637
Supervisory Skills 56
4.16
.570

19. Descriptive Statistics
Externs N
Min
Max
Mean
S.D.
Relationship/Interpersonal Skills
With Patients/Clients/Families
203
2.75
5.00
4.20
.656
With Colleagues
186
3.00
4.28
.639
With Supervisors
206
1.50
4.23
.711
With Support Staff --
With Teams at Clinic
184
4.17
.761
With Community Professionals
153
2.00
4. 14
.776
For the Practicum Site Itself
178
2.67
4.14
.732

20. Descriptive Statistics
Externs N
Min
Max
Mean
S.D.
Skills in Application of Research --
Intervention Skills 98
2.13
5.00
3.82
.731
Psychological Assessment Skills 92
3.00
4.21
.760
Consultation Skills/Interdisciplinary Collaborations
110
.597
Diversity (Individual and Cultural Differences)
200
2.00
4.13
.740
Ethics
188
1.50
4.25
.739
Professional Development
Practical Skills to Maintain Effective Clinical Practice
161
2.17
4.02
.748
Professional Development Competencies
136
2.18
4.24
.675
Metaknowledge/Metacompetencies
190
4.15
.682
Development of Leadership Skills
Supervisory Skills

21. Inter-Scale Correlations
Fellows’ Inter-Scale Correlations Ranged From .39 (Relationship Skills with Support Staff – Leadership Skills) (Assessment Skills – Consultation Skills)
Externs’ Inter-Scale Correlations Ranged From .63 (Relationship Skills with Team – Ethics) (Intervention Skills – Consultation Skills)

22. Pre – Post Comparisons Fellows Relationship/Interpersonal Skills
Evaluation Period t p 1 2 N M
S.D.
Relationship/Interpersonal Skills
With Patients/Clients/Families 42
4.37
.723 48
4.71
.438
2.77
.007
With Colleagues 40
4.33
.693 45
4.74
.471
3.26
.002
With Supervisors 44
4.39
.698
4.69
.511
2.34
.02
With Support Staff
4.52
.707
4.82
.390
2.41
.018
With Teams at Clinic 38
4.41
.761 39
4.68
.493
1.86
.066
With Community Professionals 35
4.14
.879 41
4.76
.435
3.94
.000
For the Practicum Site Itself
4.32
.803 36
4.75
.439
2.80
.006

23. Pre – Post Comparisons Fellows Skills in Application of Research 41
Evaluation Period t p 1 2 N M
S.D.
Skills in Application of Research 41
4.13
.775 46
4.46
.566
2.23
.028
Intervention Skills 32
4.20
.687 38
4.70
.431
3.72
.000
Psychological Assessment Skills 25
.410
.803 30
4.50
.474
2.31
.025
Consultation Skills/Interdisciplinary Collaborations
4.18
.798 34
4.64
.411
2.93
.005
Diversity (Individual and Cultural Differences) 44
4.32
.753 47
4.65
.463
2.51
.014
Ethics 40
4.33
.727
4.68
.479
2.68
.009
Professional Development
Practical Skills to Maintain Effective Clinical Practice
4.08
.754 43
4.59
.574
3.33
.001
Professional Development Competencies
4.23
.691
4.69
.421
3.55
Metaknowledge/Metacompetencies
.656 45
4.62
.476
2.37
.020
Development of Leadership Skills 13
3.65
.545 11
.560
2.94
.008
Supervisory Skills 28
3.92
.602
4.41
.420
3.50

24. Pre – Post Comparisons Externs Relationship/Interpersonal Skills
Evaluation Period t p 1 2 N M
S.D.
Relationship/Interpersonal Skills
With Patients/Clients/Families 99
4.01
.674
104
4.38
.587
4.20
.000
With Colleagues 90
4.13
.591 96
4.41
.656
3.05
.003
With Supervisors
101
4.07
.710
105
.683
3.14
.002
With Support Staff 88
4.18
.736 93
4.51
.653
3.13
With Teams at Clinic 89
3.99
.748 95
4.34
.738
3.17
With Community Professionals 66
4.00
.744 87
4.26
.784
2.11
.036
For the Practicum Site Itself
3.97
.757
4.31
.665
3.26
.001

25. Pre – Post Comparisons Externs Skills in Application of Research --
Evaluation Period t p 1 2 N M
S.D.
Skills in Application of Research --
Intervention Skills 49
3.57
.665
4.06
.715
3.56
.001
Psychological Assessment Skills 44
4.56
.811 48
4.92
.674
2.27
.026
Consultation Skills/Interdisciplinary Collaborations 51
4.08
.645 59
4.32
.532
2.15
.034
Diversity (Individual and Cultural Differences) 97
3.98
.771
103
4.28
.683
2.87
.004
Ethics 91
.763
4.42
.680
3.20
.002
Professional Development
Practical Skills to Maintain Effective Clinical Practice 77
3.86
.721 84
4.17
.743
2.74
.007
Professional Development Competencies 68
4.11
.676
4.376
.651
2.34
.021
Metaknowledge/Metacompetencies
4.01 99
4.271
2.64
.009
Development of Leadership Skills
Supervisory Skills

26. Discussion
Preliminary data suggest that competencies can be measured reliably
Inter-scale correlations raise important questions concerning competency constructs

27. Discussion Future Directions
Need for additional data to inform competency evaluation process
More controlled study of competency scales (e.g., mid-term and final ratings)
Factor-analytic approaches may address questions of inter-scale shared variance
Need for validation strategies