Enhancing Competency Based Training Through the Screening, Selection, and Performance Evaluation of Psychology Interns Paul Robins, Ph.D 1,2., Thomas Power, Ph.D 1,2., Mary Rourke,Ph.D 1., Melissa Alderfer, Ph.D 1,2., Cathi Grus, Ph.D 3., & Marley Watkins, Ph.D 4 1 The Childrens Hospital of Philadelphia, 2 University of Pennsylvania, 3 APA, 4 Arizona State University

Competency Based Training Forms a basis for defining and measuring trainee learning outcomes Competencies conceptualized as elements or components of competence, and consist of discrete knowledge, skills, and attitudes (Kaslow et al., 2004). Cube model: twelve core competencies conceptualized as either foundational or functional competencies (Rodolfa, et. al, 2005) l

Todays Presentation 1. Describe process of developing a blueprint of foundational and functional competencies 2. Describe competency based intern evaluation instrument 3. Describe process of developing an intern screening tool that reflects core mission priorities 4. Present reliability and validity data of the intern screening tool 5. Future directions/learnings

Developing Core Competencies Foundational documents –Hospital mission –Behavioral Health Center mission –Internship program mission –Funding priorities: GPE and LEND missions Translate documents to core training domains –Iterative process –Responsive to multiple missions and our understanding of training psychologists as leaders in the field

Foundational Documents: Hospital Mission The Childrens Hospital of Philadelphia, the oldest hospital in the United States dedicated exclusively to pediatrics, strives to be the world leader in the advancement of health care for children by integrating excellent patient care, innovative research and quality professional education into all of its programs.

Behavioral Health Center Mission The Childrens Hospital of Philadelphias Behavioral Health Center (BHC) strives to be among the nations top leaders in childrens behavioral health. The BHC provides an inter-disciplinary approach to improving the behavioral health and development of children and their families and advancing knowledge through excellent clinical care, research, training and advocacy. The BHC promotes innovative research and the integration of science and practice, focusing on prevention and intervention. We partner with others within the hospital, regionally, and nationally to increase the availability of evidence-based behavioral services for all patients at CHOP. The BHC is committed to outstanding scientist- practitioner training and to supporting the competence of caregivers and staff. Our endeavors integrate disciplines and systems of care in childrens health.

Internship Program Mission The Internship Program aims to prepare advanced doctoral-level students to take leadership roles in developing, providing, evaluating, and disseminating effective psychological services for the diverse population of children and families with health and/or mental health needs. The Program is committed to preparing psychologists as leaders in the field, engaged in clinical practice, advocacy, interdisciplinary collaboration, and scientific investigation.

Funding Agencies HRSA: BHP (GPE) & MCHB (LEND) Vision The Health Resources and Services Administration (HRSA) envisions optimal health for all, supported by a health care system that assures access to comprehensive, culturally competent, quality care. Mission HRSA provides national leadership, program resources and services needed to improve access to culturally competent, quality health care.

Development of Core Competencies Functional Competency Domains (6) 1.Assessment 2.Intervention/prevention/advocacy 3.Consultation 4.Research 5.Professional development 6.Diversity effectiveness

Identify Unique Strengths of Programthe Big Five Foundational Competency Domains 1.Prevention/advocacy (BHP/GPE, future) 2.Ecological/systems underpinning (our unique history, PCGC) 3.Leadership training (Hospital, MCHB/LEND) 4.Interdisciplinary collaboration (Hospital family centered care, systems) 5.Community based, medically underserved populations (BHP/GPE)

Content Validity--Blueprint 6 Functional Competency Domains X 5 Foundational Competency Domains55 items Provided blueprint of core skills and outcomes (content validity) Formed basis of required intern skills and outcomes

Assessing Intern Competencies Core competencies rated by rotation supervisors along a 3 point scale (scaling being revised to reflect APA CoA standards) Based upon end of the internship performance expectations, rate this intern on each item using the following scale: n/a = not applicable 1 = does not yet meet expectations 2 = meets expectations 3 = exceeds expectations (second semester only)

Assessing Intern Competencies Item Examples Assessment –Demonstrates effective clinical interviewing skills with children and families –Demonstrates ability to effectively collect and integrate data from multiple sources and/or disciplines

Assessing Intern Competencies Research –Effectively searches for and applies most relevant empirical findings to inform clinical practice –Demonstrates skills in developing research designs

Intern Selection Develop screening method/instrument which assists us in identifying interns who are a strong match based on our mission and values –Across functional and foundational competency domains, our blueprint

Description of Screening Instrument Iterative process, revisions, & training 13 items total, anchored Likert scale (1-4) Breath of clinical training (5 items), across: –Systems of care (e.g., clinic, hospital, community, school) –Treatment modalities (e.g., individual, group, family) –Age/demographics (e.g., infants, toddlers, school age, adolescents) –Presenting diagnoses (e.g., mental health, medical) –Clinical activities (e.g., assessment, intervention, consultation)

Description of Screening Tool Depth of clinical training (3 items) –Amount/type of clinical supervision –Direct treatment hours (adults and children) –Number of comprehensive assessment reports (adults and children)

Description of Screening Tool Big Five items (Foundational) –Developmental-ecological model –Diversity effectiveness –Interdisciplinary collaboration –Research potential –Leadership potential

The Childrens Hospital of Philadelphia Psychology Internship Training Program Pre-Doctoral Internship Selection – Screening Form (Class of ) 1. Rate breadth of training…across multiple systems of care (e.g.,mental health, medical, school, and community systems) 4 1 or more training experiences within 4 systems of care 3 1 or more training experiences within 3 systems of care 2 1 or more training experiences 2 systems of care 1 1 training experience within 1 system of care

with regard to various treatment modalities (e.g., individual therapy, family therapy, multiple family therapy, parent training, child group intervention {i.e. classroom-based} and group therapy) 4 Training experience in 4 or more treatment modalities 3Training experience in 3 different treatment modalities 2Training experience in 2 different treatment modalities 1Training experience in only 1 treatment modality

6. Rate level of ethnic, linguistic, and cultural diversity awareness and effectiveness, taking into consideration experience in and commitment to learn in this area as demonstrated through the following domains: research, coursework, teaching experiences, seeking practicum placements, case conceptualization, and committee membership. 4 Diversity effectiveness is a clear, defining priority, guiding graduate training, as evidenced in more than 3 of the above domains 3Diversity effectiveness informs graduate training and is important as demonstrated in at least 3 of the above domains 2 Diversity effectiveness is a part of graduate training as demonstrated by awareness and exposure in at least 2 of the above domains 1Diversity effectiveness is reflected in only 1 domain or is not evident in training

8. Rate research potential in child-oriented psychology. 4 Extramural pre-doctoral grant and 1+ first author peer- reviewed journal publication (excluding submitted for publication) 2 1 first author publication (excluding submitted for publication) or a pre-doctoral grant; plus at least 2 national poster or paper presentations as a first author 21 or more publications (including in press) on which applicant is not the first author; plus 1 or more national poster or paper presentations 1No publications; may have some national poster or paper presentations

Screening Process Administrative screening: ~210 applications/year Each of ~190 applications reviewed by 2 psychology staff using screening instrument (early Nov-mid Dec) Total score obtained Preliminary rank order determined Hand reviewed by TD 35 on -site interviews

Psychometrics: Construct Validity Factor AnalysisReviewer 1 –N = 382 –2 factor solution –Factor 1 alpha =.69 (acceptable) –Factor 2 alpha =.70 (acceptable) Factor AnalysisReplication (Reviewer 2) –N = 382 –2 factor solution –Factor alphas.69 and.72, respectively Factor 1 & 2 congruence coefficients =.94 &.98 (good to excellent factor similarity between sets of raters) F1 and F2 r =.44 (moderate)

Pattern Matrix Item12 Interdisciplinary collaboration Developmental ecological model Diversity effectiveness Leadership effectiveness Research potential Practicum hours Supervision quality Number assessment reports Number intervention cases Breath across age groups Breath across treatment modalities Breath of clinical activities Systems of care training.22.24

Reliability Alphas: –Factor 1 (Reviewers 1 & 2) =.69 &.72 –Factor 2 (Reviewers 1 & 2) =.70 &.70 –acceptable Inter-rater reliability, factor level –F1 =.62 –F2 =.61 –marginal

Results suggest: Two factors makes sensecapture breath of clinical experience and foundational items (but still only capture 27.4% of variance) Two factors makes sensecapture breath of clinical experience and foundational items (but still only capture 27.4% of variance) Use inter-rater item correlations Use inter-rater item correlations

Foundational or Functional?

Conclusions Attempt to operationalize core competencies step in the right direction Reflects our mission, our unique identity Ongoing process, iterative Screening tool: Clarify items: better anchor response choices Increase training (e.g., develop scoring manual with examples)