Using Participatory Action Research to Develop and Validate the Core Competency Measure (CCM) Stephen S. Leff, Ph.D., Nathan Blum, M.D., Abbas Jawad, Ph.D., Judith Silver, Ph.D., Symme Trachtenberg, MSW, Karen Hudson, MSW, & Karen Tate, Family Faculty The Children’s Hospital of Philadelphia & University of Pennsylvania School of Medicine Poster presented at the Association of University Centers on Disabilities Annual Meeting and Conference (Washington, DC, November, 2007) Requests for reprints: Stephen S. Leff; Children’s Hospital of Philadelphia, Dept. of Psychology, Rm at 3535 Market, 3405 Civic Center Blvd.; Philadelphia, PA 19104;

Purpose and Overview The LEND Program Evaluation Committee at CHOP has developed specific measurement tools to assess the impact of LEND fellowship training We will highlight the Core Competency Measure (CCM) –The CCM assesses fellows’ perception of their knowledge and skills across core competency domains of LEND We will discuss how the CCM was designed through a participatory action research model We will examine an initial validation sample for the past five cohorts of LEND fellows, and discuss potential implications and uses of this new measure

The LEND Program Evaluation Committee at CHOP Interdisciplinary Team including: –Psychologists –Biostatistician –Pediatrician –Social Worker –LEND fellows from a variety of disciplines Primary Goals –Develop a core set of outcome measures –Modify LEND curriculum based upon outcome evaluation (translating research to practice) –Provide mentoring to fellows in conducting program evaluations through a Participatory Research model

Participatory Action Research (PAR) Framework Combining theoretical principles, past empirical research, and key stakeholder input into design of measures (Leff et al., 2004) Partnership drives the project Researchers & participants form non- hierarchical trusting relationships Partnership ensures meaningfulness Links Research to Practice

Description of Measures Core Competency Measure (CCM) –Modeled after the core dimensions of LEND training –Originally piloted with two cohorts of LEND fellows –Addresses LEND’s 6 Major Objectives Clinical Skills (8 items) Interdisciplinary Collaboration (5 items) Family-Centered & Culturally-Sensitive Services (7 items) Community-Based Services (9 items) Research Methods (9 items) Knowledge of Disability Policies & Regulations (6 items) –Likert response scale 1=Disagree, 2=Somewhat Agree, 3=Agree; 4=Strongly Agree

Subscales of the CCM Sample Items Clinical Skills –I have a good understanding of normal development –I am able to identify common procedures for early screening & diagnosis of developmental delay or disability –I am able to discuss the impact of disability on the child & family Interdisciplinary Collaboration –I have knowledge of the different characteristics of health care professions, including assessment tools, developmental domains, an intervention strategies addressed by the respective disciplines –I understand ways to foster strong collaborations with health care professionals from different disciplines Family-Centered & Culturally-Sensitive Services –I am knowledgeable about the key elements of family-centered care & its impact on care delivery outcomes –I can describe a typical daily routine & activities families may encounter when living with a special needs child –I am knowledgeable about how culture & traditional/non-traditional beliefs about health care impact families’ experiences with health care delivery

Subscales of the CCM Sample Items Community-Based Services –I am able to refer and coordinate community-based care services for children with special needs –I can collaborate well with colleagues from community-based agencies and families to develop transition plans –I have a good understanding of how to influence practice in a broad way via institutional and systems change Evaluate and Contribute to Research –I am able to easily develop research questions –I understand the role of partnership in designing, implementing, and evaluating research projects –I have an understanding of the role of the Institutional Review Board and how to prepare an IRB proposal Knowledge of Regulations/Entitlements that Impact Youth with Disabilities –I am able to effectively advocate to support child & family entitlements –I can critically examine examples of discrimination towards people with disabilities or special health care needs

Using PAR to Design the CCM The CCM was designed by combining what was considered the core dimensions of LEND with literature reviews on each primary domain combined with feedback from our Program Evaluation Committee A draft version of the CCM was then piloted with two cohorts of LEND fellows at the beginning and end of their respective training years The Program Evaluation Committee and LEND fellows provided additional feedback on subscales, wording, and likert-response scales The CCM measure was then finalized

Psychometric Analyses Subscales Internal Consistency Test-Retest Reliability (n = 73) (n = 21) Clinical  =.82 r =.82 Interdisciplinary  =.78 r =.62 Family & Culture  =.88 r =.72 Community- Based Services  =.91 r =.85 Research  =.94 r =.94 Regulations & Entitlements  =.87 r =.80

Intercorrelations Between Subscales (n=73) Clinical Interdisc. Family/Cult. Community Research Regulations Clinical 1r=.54**r=.28*r=.16r=.22r=.17 Interdisc. 1r=.59**r=.51**r=.20r=.42** Family/ Culture 1r=.60**r=.19r=.51** Community 1r=.19r=.63** Research 1r=.08 Regulations 1

Core Competency Measure (CCM) Pre- and Post-LEND Year Results

Summary –Use of a Participatory Action Research model helps to ensure the usefulness of measures, and has allowed us to modify our curriculum to be responsive to fellows’ feedback –The CCM has strong internal consistency and adequate test-retest reliability across the six subscales –The Interdisciplinary, Family-Culture, Community, & Regulations/Entitlements subscales are highly intercorrelated, while the Research subscale is not significantly correlated with any other subscales –The CCM appears to be sensitive to treatment effects. Overall, LEND fellows clearly report significant knowledge/expertise advances in all areas over the course of the LEND year (e.g., clinical, interdisciplinary, family/cultural, community, research, and disability regulations/entitlements)

Future Directions –The CCM could be used to help other LEND and UCEDD Programs evaluate whether their fellows make progress in these six core competency areas over the course of their training –We are currently assessing how well the CCM correlates with other program evaluation measures we have designed and validated related to Research Competency and Interdisciplinary Collaborations