Presentation is loading. Please wait.

Presentation is loading. Please wait.

Reliability and Validity of the Disruptive Behavior Instrument for Physicians with Disruptive Behavior Philip Hemphill, Ph.D. Professional Enhancement.

Similar presentations


Presentation on theme: "Reliability and Validity of the Disruptive Behavior Instrument for Physicians with Disruptive Behavior Philip Hemphill, Ph.D. Professional Enhancement."— Presentation transcript:

1 Reliability and Validity of the Disruptive Behavior Instrument for Physicians with Disruptive Behavior Philip Hemphill, Ph.D. Professional Enhancement Program (PEP) Pine Grove Behavioral Health and Addictive Services

2 FSPHP Presenter Disclosure Slide “Reliability and Validity of the Disruptive Behavior Instrument for Physicians with Disruptive Behavior” Philip Hemphill, PhD is a paid consultant of Pine Grove Behavioral Health and Addiction Services FSPHP Annual Conference and Meeting Fort Worth, Texas April 23-26, 2012

3 Why use a MSF/360? 1. Facilitate insight of strengths and weaknesses for an individual. 2. Enhance organizational cultural change. 3. Provide summative assessment of performance. 4. Evaluate potential. 5. Improve team effectiveness. 6. Identify training & coaching needs. 7. Make organizational values explicit. 8. Measure progress after training & coaching.

4 MSF/360 Essentials Proven validity & reliability. Useful in assessing “humanistic” skills. Weak or no associations between self-rated assessment and external assessment. “…methods such as multisource feedback (360) evaluations may be a necessary next step, particularly when interpersonal, communication skills, or professionalism needs to be evaluated (p. 1101).” Source: Davis et al. (2006). Accuracy of physician self-assessment compared with observed Measures of competence.JAMA, 296 (9),

5 Key Players in MSF/360 Target Individual Raters Boss FacilitatorVendor

6 MSF/360 Internal Process Feedback Is it consistent with self-perception? Emotional Response Reflection and internal processing of emotion and content. Decision to accept and act, or not Action for learning, change Source: Sargeant et al. (2009) Advances in Health Sc Ed.

7 Relevant Literature Review of Multi-Source Feedback Measures The Fundamental Interpersonal Relations Orientation–Behavior™ (FIRO-B®)  In the late 1950s William Schutz, PhD. developed the FIRO-B theory to aid in the understanding and predicting of how high- performance military teams would work together.  In developing the FIRO-B theory, Schutz began with the premise that "people need people." He used the term interpersonal to indicate any interaction, real or imagined, occurring between people. He used the term need to describe a psychological condition that, if not satisfied, leads to a state of discomfort or anxiety.  He posited that interpersonal needs could be grouped into three categories: Inclusion, Control, and Affection. The FIRO-B model describes the interaction of these three categories of interpersonal need along two dimensions: expressed and wanted.  Awareness Scales and were designed primarily to help individuals with their self-awareness and of understanding their relation to other people (Schutz, 1978).

8 Relevant Literature Review of Multi-Source Feedback Measures 1. Developing insight into strengths and weaknesses, for the individual (Farh & Dobbins, 1989; Hazucha et al., 1993; Martocchio & Judge, 1997; Fletcher, 1999; Keeping et al., 1999; London et al., 1999; McCarthy & Garavan, 1999) and the organization (London & Beatty, 1993; Lepsinger & Lucia, 1998; Steensma et al., 1998; Tornow & London,1998) to act upon. 2. Enhancing culture change (Cardy & Dobbins, 1994; O’Reilly, 1994; Crossley & Taylor, 1995; Church & Bracken, 1997; Bracken et al., 1998; DeSimone, 1998). 3. Summative assessment of performance (Newble et al., 1999). 4. Evaluating the potential of individuals, for instance to use in career advice or selection (Tornow, 1993; Lindsey et al.,1997; Facteau et al., 1998). 5. Enhancing team effectiveness (by allowing opportunity to comment) (Garavan et al., 1997; Towers-Perrin, 1998). 6. Identifying training needs for the system (as opposed to the individual) (Towers-Perrin, 1998).

9 Development of Disruptive Behavior Instrument  2004 – Developed list of disruptive behaviors and boundary violation behaviors. Approximately 70 items…  5 people monitored over 12 months – Instrument too long; complaints from participants; descriptive data….  Cut instrument down to 50 items(1 Fctr)  Cut down to current version  Been using current version for past 4 years…

10 Definition of Disruptive Behavior Disruptive behavior is any inappropriate behavior, confrontation, or conflict, ranging from verbal abuse to physical or sexual harassment. Disruptive behavior causes strong psychological and emotional feelings, which can adversely affect patient care. Rosenstein A, O’Daniel M. (2008). Managing disruptive physician behavior: Impact on staff relationships. Neurology, 70,

11 Disruptive Behavior Items  Avoids taking responsibility for actions  Inappropriately blames others  Places other interests above the safety and welfare of community members  Suspects, without sufficient basis, that others are exploiting, harming, deceiving, or out to “get” him  Seems to have a preoccupation with unjustified doubts about the loyalty or trustworthiness of others  Reads hidden, demeaning, or threatening meanings into benign remarks or events  Perceives attacks on his character/reputation not apparent to others and is quick to react angrily/counterattack  Shows rigidity and stubbornness  Displays an unpredictable mood  Responds to constructive criticism in an immature/irresponsible fashion  Takes too long to requests from others  Arrives late  Fails to complete paperwork in a timely manner  Responds to requests in an avoidant, unreliable, and/or uncooperative manner  Displays a sense of entitlement  Shows perfectionism that interferes with task completion Behaves in a sexually inappropriate manner (verbally or physically)  Lacks empathy Intimidates others or attempts to intimidate others  Exhibits physically aggressive or assaultive behaviors  Exhibits arrogant behaviors and/or has an arrogant attitude  Demonstrates an inflated sense of self-importance  Takes advantage of others in order to get what he wants  Is reluctant to delegate tasks or to work with others unless they submit to exactly his way of doing things  Expresses anger/hostility inappropriately  Exhibits passive-aggressive behavior  Uses abusive, belittling, condescending, demeaning, and/or threatening language

12 Client signs release allowing for electronic correspondence, and client given monitoring instructions and materials with due date. Client identifies MC, gives MC instructions and rating materials, and with MC identifies approx. 15 raters.. Ratee addresses Collection of surveys online. Rating process is repeated at predetermined intervals until four uneventful reports Client, MC, and other relevant stakeholders review the feedback report Significant impairment is present and PHP/LAP intervenes CLIENT SIGNS/ UPDATES MONITORING CONTRACT WITH PHP/LAP AND/OR No significant impairment is present Client attends treatment and rating process is temporarily suspended PHP/LAP recommends interventions Client participates in interventions while continuing rating process PHP/LAP recommends intensive/ residential treatment Client completes treatment Report produced and ed to persons indicated on release. MONITORING RATING FLOWCHART

13 33 questions  28 Likert-type questions regarding ratee’s behavior  1 question concerning how many hours rater works with ratee  1 question about the rater’s title  3 open-ended questions that solicit comments

14

15

16 Survey Dimensions  Appropriate Documentation  Arrogance  Citizenship  Compliance  Empathy  Exploitation of Others  Expression of Anger  Intimidation  Job Attitudes  Multicultural Sensitivity  Openness to Others  Perfectionism  Respect for Others  Response to Criticism  Responsibility for Actions  Rigidity  Satisfaction with Ratee  Sexual Appropriateness  Stability of Mood  Stress Management  Supervision  Suspiciousness  Tardiness  Team Impact  Threatening Behavior  Verbal Aggression  Working with Others

17 Disruptive Behavior Instrument  Descriptive Characteristics of Sample:  N = 22 ratees (498 surveys)  Average Age = 38 years  Male = 19 ; Female = 3  Survey Breakdown by rater:  Self-Report = 53  Staff = 328  Supervisor = 37  Peer = 79  Number of Times each patient was assessed:  Range – 1-8 times

18 Disruptive Behavior Instrument Aggression Aggression: Non-verbal Aggression: Physical Aggression: Verbal Aggression: Written Aggression Aggression: Non-verbal Aggression: Physical Aggression: Verbal Aggression: Written Interpersonal Difficulties Intimidation Lack of Empathy Lack of Multicultural Sensitivity Lack of Openness to Others Problematic Perfectionism Response to Criticism Rigidity Suspiciousness Interpersonal Difficulties Intimidation Lack of Empathy Lack of Multicultural Sensitivity Lack of Openness to Others Problematic Perfectionism Response to Criticism Rigidity Suspiciousness Sexually inappropriate behavior Sexual Inappropriateness Sexual/Inappropriate Remarks Sexually inappropriate behavior Sexual Inappropriateness Sexual/Inappropriate Remarks Self Regulation Mood Instability Poor Stress Management Self Regulation Mood Instability Poor Stress Management Disruptive Behavior Arrogance Blaming Others Disobeying Rules Disrespect for Staff/Colleagues Ease of Working with Ratee Employee Turnover Intentions Exploitation of Others Poor Citizenship Tardiness Disruptive Behavior Arrogance Blaming Others Disobeying Rules Disrespect for Staff/Colleagues Ease of Working with Ratee Employee Turnover Intentions Exploitation of Others Poor Citizenship Tardiness Originally hypothesized to be based on 5 factors

19 Disruptive Behavior Instrument  Results of hypothesized 5-factor loadings; 59% cumulative total variance explained Q Q Q Q Q Q Q Q Q Q Q Q Q Factor Q Q Q Q Q Q Q Q Factor Q Q Q Factor Q Q Factor Q Factor 5

20 Raw Data Eigenvalues, Mean & Percentile Random Data Eigenvalues Ncases 498; Nvars 27; ; Ndatsets 1000; Percent 95 Root Raw Data Means Percentile

21 Disruptive Behavior Instrument  Recent analyses reveal that the items are loaded on three distinct factors 123 Q Q Q Q Q Q Q Q Q Q Q Q Q Factor Q Q Q Q Q Q Q Q Q Factor Q Q Q Q Factor 3 * Excluded Item 26

22 1. Aggression: Non-VerbalInterpersonal Aggression 2. Aggression: PhysicalInterpersonal Aggression 3. Aggression: VerbalInterpersonal Aggression 4. Aggression: WrittenInterpersonal Aggression 5. ArroganceInterpersonal Aggression 6. Blaming OthersInterpersonal Aggression 7. Disobeying RulesInterpersonal Aggression 8. Disrespect for Staff/Colleagues Professionalism 9. Ease of Working With RateeInterpersonal Aggression 10. Employee Turnover Intentions Critical Item 11. Exploitation of OthersInterpersonal Aggression 12. IntimidationInterpersonal Aggression 13. Lack of EmpathyProfessionalism 14. Lack of Multicultural Sensitivity Interpersonal Aggression 15. Lack of Openness to OthersProfessionalism 16. Mood InstabilityProfessionalism 17. Poor CitizenshipInterpersonal Aggression 18. Poor Stress ManagementProfessionalism 19. Problematic PerfectionismProfessionalism 20. Response to CriticismProfessionalism 21. RigidityProfessionalism 22. Sexual InappropriatenessCritical Item 23. SuspiciousnessInterpersonal Aggression 24. TardinessProfessionalism 25. Sexual and inappropriate Remarks Critical Item 26. Substance AbuseProfessionalism 27. Sexual InappropriatenessCritical Item 28. Prescribing medicationsCritical Item Disruptive Behavior Instrument Factor 1: Interpersonal Aggression Items: 1, 2, 3, 4, 5, 6, 7, 9, 11, 12, 14, 17, 23 Factor 2: Professionalism Items: 8, 13, 15, 16, 18, 19, 20, 21, 24, 26 Critical Items: Items: 10, 22, 25, 28

23 Disruptive Behavior Instrument Total Variance Explained FactorExtraction Sums of Squared LoadingsRotation Sums of Squared Loadings Total% of VarianceCumulative %Total Extraction Method: Principal Component Analysis. a. When components are correlated, sums of squared loadings cannot be added to obtain a total variance.

24 Disruptive Behavior Instrument Component Correlation Matrix Factor Extraction Method: Principal Component Analysis. Rotation Method: Promax with Kaiser Normalization.

25 Disruptive Behavior Instrument Cronbach’s Alpha ( α ) N of Items Factor Factor Factor Reliability for Three Factors Poor reliability for items on Factor 3 suggests that a 2 Factor structure may be a better fit.

26 Disruptive Behavior Instrument 2 Factors Interpersonal Aggression Aggression: Non-verbal Aggression: Physical Aggression: Verbal Aggression: Written Arrogance Blaming Others Disobeying Rules Exploitation of Others Intimidation Lack of Multicultural Sensitivity Poor Citizenship Suspiciousness Interpersonal Aggression Aggression: Non-verbal Aggression: Physical Aggression: Verbal Aggression: Written Arrogance Blaming Others Disobeying Rules Exploitation of Others Intimidation Lack of Multicultural Sensitivity Poor Citizenship Suspiciousness Professionalism Disrespect for Staff/Colleagues Lack of Empathy Lack of Openness to Others Mood Instability Poor Stress Management Problematic Perfectionism Response to Criticism Rigidity Tardiness Professionalism Disrespect for Staff/Colleagues Lack of Empathy Lack of Openness to Others Mood Instability Poor Stress Management Problematic Perfectionism Response to Criticism Rigidity Tardiness

27 Disruptive Behavior Instrument  Instead of a third factor, items were named “critical items”.  5 Critical Items:  “I think about quitting or transferring to a different job because of the ratee.”  “Touches or looks at one or more patients/coworkers in an inappropriate/sexually suggestive manner.”  “Makes sexual remarks in the workplace that are inappropriate or tells jokes that makes others uncomfortable.”  “Performs unnecessary exams on female and/or male patients.”  “Demonstrates lax boundaries when prescribing controlled substances, giving out samples, or storing samples of controlled substances.”

28 Disruptive Behavior Instrument  Conclusions/Limitations:  We have 2 main factors and 5 critical items.  Two factors:  Interpersonal Aggression  Professionalism

29 Future Directions:  Measuring and refining Inter-rater Reliability  Measuring and refining Test-Retest Reliability  How is this data utilized?  What are cutoff scores?  Can we track disruptive behavior without treatment?  Can instrument be utilized outside of medicine?

30 We do not deal much in fact when we are contemplating ourselves. Mark Twain


Download ppt "Reliability and Validity of the Disruptive Behavior Instrument for Physicians with Disruptive Behavior Philip Hemphill, Ph.D. Professional Enhancement."

Similar presentations


Ads by Google