Presentation is loading. Please wait.

Presentation is loading. Please wait.

Using observational methods to measure treatment integrity in psychosocial intervention research Kimberly S. Van Haitsma, Ph.D. Director, Polisher Research.

Similar presentations


Presentation on theme: "Using observational methods to measure treatment integrity in psychosocial intervention research Kimberly S. Van Haitsma, Ph.D. Director, Polisher Research."— Presentation transcript:

1 Using observational methods to measure treatment integrity in psychosocial intervention research Kimberly S. Van Haitsma, Ph.D. Director, Polisher Research Institute Madlyn & Leonard Abramson Center for Jewish Life (formerly Philadelphia Geriatric Center) 1425 Horsham Road North Wales, PA Presented at the Gerontological Society of America 64 th Annual Scientific Meeting Pre-Conference Workshop Current and Future Challenges in Designing Behavioral Interventions: From Randomized Trials to Community Implementation November 18 th :30 to 4;30 1

2 K2A: Knowledge to Action Framework for Public Health EVALUATION Translation Supporting Structures Research Supporting Structures Institution- alization Supporting Structures DISCOVERY STUDIES EFFICACY STUDIES EFFECTIVENESS AND IMPLEMENTATION STUDIES TRANSLATION PHASE RESEARCH PHASE INSTITUTION- ALIZATION PHASE KNOWLEDGE INTO PRODUCTS DISSEMINATION PRACTICE INSTITUTION- ALIZATION DECISION TO TRANSLATE ENGAGEMENT DECISION TO ADOPT DIFFUSION Practice-based Discovery Practice-based Evidence Wilson, K., Brady, T. &Lesesne, C. (2011). An organizing framework for Translation in public health: The knowledge to action framework. Preventing chronic disease Public health research, practice and policy, 8(2),1-7.

3 Overview of presentation Illustration of how real-time observations can be used to enhance fidelity in psychosocial intervention studies – What behavior observation measures are available? – What aspects of fidelity measurement can be optimized by real time observational techniques? 3

4 An illustrative example: The Individualized Positive Psychosocial Intervention Study Funded by To examine the impact of an individualized recreational intervention on quality of life outcomes for persons with dementia residing in nursing homes. – Focus on CNA as the interventionist of choice (with the assistance of RT) – Intervention focus was on enhancing CNA communication skills and individualizing content of intervention – Outcomes focused on enhancing resident positive emotion and behavior and diminishing negative emotion and behavior. – Methodology focused on real-time observation of dyadic interactions.

5 Intervention Study Design Randomized Controlled Trial Residents randomly assigned to one of two intervention conditions (IPPI or Attention Control) or Usual Care CNAs were assigned to provide intervention based on their existing permanent assignment to a given resident and were blind to intervention condition Residents (n=180) CNAs (n=84) Number of real-time observations (n=2,638 occasions)

6 Resident Group Assignments Individualized Positive Psychosocial Intervention (IPPI) (n=44 residents) Content of activity selected based on resident preferences Attention Control Intervention (N= 43 residents) Standardized 1-1 interaction (e.g. reading a magazine aloud) Both Interventions – CNA intervention delivery facilitated by coaching from Recreational Therapist – 10 minute sessions, 3x/week, for 3 weeks – Half of CNAs received additional Communication training; half did not. Usual Care (n=93 residents) No prescribed interactions. Captured normative behaviors.

7 Intervention delivery focused on Communication Skills of CNAs 7

8 Definitions of Observed CNA Behavior Variables Van Haitsma, K., Lawton, M.P., Kleban, M., Klapper, J.A. & Corn, J.A. (1997). Methodological aspects of the study of streams of behavior in dementing illness. Alzheimer Disease and Associated Disorder, 11(4), PMID: Prescribed Positive Communication Behaviors GreetingSaying hello, goodbye, handshake ExplanationExplains what activity is or will be occurring CourtesySaying excuse me, please, thank you, you’re welcome Offers choiceOffers choice by explicitly identifying 2 alternatives Asks preferenceSolicits resident preferences for past, present, or future activities Praise/complimentExpresses praise, compliment or approval ReassuranceProvides reassurance or validates feelings Positive touchGently touching resident, stroking arm or back, holding hands, hugging, kissing Positive/neutral promptVerbal prompt to initiate an activity without negative tone Nonverbal promptPoints, shows something to orient toward object Task engagementVerbal interaction related to the task at hand; Physical manipulation of objects related to task at hand, assisting resident in manipulation of objects Proscribed Negative Communication Behaviors Ignores residentDoes not look at resident or respond to resident verbal or nonverbal behavior Talks to otherTalks to third party while interacting with resident Rebuke/disapproveReprimand, admonish, express disapproval Negative touchHandles resident roughly; grabs resident; moves resident without warning

9 Intervention Outcomes Focused on Enhancing Resident Affective and Behavioral Outcomes related to Quality of Life 9

10 Definitions of Observed Resident Behavior Variables Positive Behavior Task Engagement engages in conversation, manually manipulates or gestures toward an object Positive Verbalcoherent conversation, responding to question Very Positive Verbalcomplimenting, joking, singing Positive Touch physically receptive to another person, gently touching, stroking, hugging, kissing Negative behavior Aggressionhitting, kicking, throwing things, grabbing, spitting Uncooperativepulling away, saying “no”, turning head or body away Very Negative Verbalswearing, screaming, mocking, making strange noises Negative Verbalincoherent, repetitious statements, muttering General Restlessnesspacing, fidgeting, disrobing, repetitive movement Eyes ClosedSits with eyes closed Van Haitsma, K., Lawton, M.P., Kleban, M., Klapper, J.A. & Corn, J.A. (1997). Methodological aspects of the study of streams of behavior in dementing illness. Alzheimer Disease and Associated Disorder, 11(4), PMID:

11 Resident Affective Responses 11 Lawton, M. P., Van Haitsma, K., Perkinson, M., & Ruckdeschel, K. (1999). Observed affect and quality of life in dementia: Further affirmations and problems. Journal of Mental Health and Aging, 5,

12 Detailed information about measurement properties of observational measures of outcomes 12 Curyto, K., Van Haitsma, K., Vriesman, D. (2008). Direct Observation of Individual Behavior: A Review of Current Methods and Measures for Use with Older Adults with Dementia. Research in Gerontological Nursing 1(1), Updates pertinant to this review since 2008: Burgio, L. D., Park, N. S., Hardin, J. M., & Sun, F. (2007). A longitudinal examination of agitation and resident characteristics in the nursing home. The Gerontologist, 47(5), Cohen-Mansfield, J., Thein, K., Dakheel-Ali, M., & Marx, M. S. (2010). Engaging nursing home residents with dementia in activities: The effects of modeling, presentation order, time of day, and setting characteristics. Aging & Mental Health, 14(4), doi: / Ersek, M., Polissar, N., & Neradilek, M. B. (2011). Development of a composite pain measure for persons with advanced dementia: Exploratory analyses in self-reporting nursing home residents. Journal of Pain and Symptom Management, 41(3), doi: /j.jpainsymman Horgas, A. L., Elliott, A. F., & Marsiske, M. (2009). Pain assessment in persons with dementia: Relationship between self-report and behavioral observation. Journal of the American Geriatrics Society, 57(1), doi: /j x Husebo, B. S., Strand, L. I., Moe-Nilssen, R., Husebo, S. B., & Ljunggren, A. E. (2009). Pain behaviour and pain intensity in older persons with severe dementia: Reliability of the MOBID pain scale by video uptake. Scandinavian Journal of Caring Sciences, 23(1), doi: /j Pulsford, D., Duxbury, J. A., & Hadi, M. (2011). A survey of staff attitudes and responses to people with dementia who are aggressive in residential care settings. J Psychiatr Ment Health Nurs, 18(2), doi: /j x. Razani, J., Bayan, S., Funes, C., Mahmoud, N., Torrence, N., Wong, J., Josephson, K. (2011). Patterns of deficits in daily functioning and cognitive performance of patients with Alzheimer disease. Journal of Geriatric Psychiatry and Neurology, 24(1), doi: /

13 Observational Method used in the IPPI study: “The Observer”

14 Measuring Behavior with The Observer Customized coding scheme Precisely detail behaviors & affect Time- stamped event log  Annotate behavior via simple key press  Coding can be done live or via videorecording  Record who does what, where, when and how/to whom  Behaviors can be coded with modifiers, e.g., intensity  Maintain time-stamped event log  Synchronize with multimodal data

15 Pocket Observer: be free to move  Code on a handheld device  Mobile observations, outdoor use  Add behaviors/modifiers on the handheld  Supports Smart phones, rugged handhelds, PDAs (Windows Mobile 6.x) Gather & score behavior data live using Pocket Observer

16 Data collection: Continuous Coding via Video

17 The ObserverXT: Analyzing the data Descriptive statistics  Frequency: How many times did the resident strike out?  Duration: How long did the resident express pleasure?  Latency, rate per minute, percent observation, inter-rater reliability, etc.

18 Bellg, A, Borrelli, B., Resnick, B. et al., (2005). Enhancing treatment fidelity in health behavior change studies: Best practices and recommendations form the NIH Behavior Change Consortium, Health Psychology, 23, What is fidelity in psychosocial intervention research?

19 What aspects of fidelity are optimally measured by real- time observational methods? 19

20 Fidelity of Treatment Delivery 20 Do CNA’s adhere to communication skills (prescribed and proscribed) in intervention protocols while delivering a one to one intervention to persons with dementia?

21 Protocol fidelity for adherence to proscribed communication skills for CNAs (N=84) trained to deliver psychosocial interventions to nursing home residents with dementia: Percent who used communication behaviors CNA Communication Behaviors Total N=84 (%) Positive Greeting 86 Explanation 89 Courtesy 79 Offers choice 80 Asks preference 98 Praise/compliment 79 Reassure 60 Positive announcement 89 Positive touch 96 Positive verbal prompt 92 Non-verbal prompt 89 Task engagement 71 Negative Negative announcement 1 Ignores resident 6 Talks to another 87 Rebuke/disapprove 13 Negative touch 0

22 Benefits of real- time observation in measuring treatment delivery integrity 22 Interventionist self reports of adherence are heavily influenced by demand characteristics or need for social approval 1 Differences between observed and interventionist reported adherence can be substantial (e.g., 100% for self report, 44% for observed) 2 Precision in measuring adherence can prevent premature abandonment of potentially effective interventions that are simply not delivered effectively. 1. Perepletchikova, F. & Kazdin, A. (2005). Treatment integrity and therapeutic change: Issues and research recommendations. Clinic P al Psychology: Science and Practice, 12(4), Hardeman, W., Michie, S., Fanshawe, T., et al. (2008). Fidelity of delivery of a physical activity intervention: Predictors and consequences. Psychology and Health, 23 (1),

23 Fidelity and Study Design: How Observational Methods Can Help 23 Can enhance study power by reducing random or unintended variability resulting in a more modest “n” in future studies 2 Facilitate identification of crucial, active ingredients and their relationship to immediate (during intervention sessions) and longer term outcomes 1  Observational methods can enhance the ability to empirically validate which intervention ingredients are most or least crucial to outcomes  Observed intervention components should be theory-based and tied to outcomes of choice 1. National Advisory Mental Health Council Workgroup Report (2010). From discovery to cure: Accelerating the development of new and personalized interventions for mental illnesses. boards-and-groups/namhc/reports/fromdiscoverytocure.pdfhttp://www.nimh.nih.gov/about/advisory- boards-and-groups/namhc/reports/fromdiscoverytocure.pdf 2. Horner, S., Rew, L., & Torres, R (2006). Enhancing intervention fidelity: A means of strengthening study impact. JSPN, 11(2), 80-89

24 Lawton, M. P., & Nahemow, L. (1973). Ecology and the aging process. The psychology of adult development and aging, Theoretical basis for focusing on verbal and nonverbal communication behaviors of CNAs in the context of intervention delivery to persons with dementia

25 Pearson Correlations (N=2638 real-time observation occasions) Resident Negative BehaviorResident Positive Behavior Very neg verbal Neg verbal Gen restless Eyes closed Uncoop- erative Aggres- sion Pos verbal Very pos verbal Task en- gagement Pos touch Negative CNA communication behaviors Negative prompt. Negative announcement Ignores resident Talks to another Rebuke/disapprove Negative touch Positive CNA communication behaviors Greeting Explanation Courtesy Offers choices Asks preference Praise/compliment Reassurance Positive verbal prompt Positive announcement Positive touch Non-verbal prompt General conversation Verbally engaged in task Physically engaged in task **. Correlation is significant at the level (2-tailed). Very neg verbal Neg verbal Gen restless Eyes closed Uncoop- erative Aggres- sion Pos verbal Very pos verbal Task en- gagement Pos touch Negative prompt.12 Negative announcement Ignores resident Talks to another Rebuke/disapprove Negative touch Greeting Explanation Courtesy Offers choices Asks preference Praise/compliment Reassurance Positive verbal prompt Positive announcement Positive touch Non-verbal prompt General conversation Verbally engaged in task Physically engaged in task Very neg verbal Neg verbal Gen restless Eyes closed Uncoop- erative Aggres- sion Pos verbal Very pos verbal Task en- gagement Pos touch Negative prompt Negative announcement.11 Ignores resident Talks to another Rebuke/disapprove.10 Negative touch Greeting Explanation Courtesy -.14 Offers choices Asks preference Praise/compliment Reassurance Positive verbal prompt Positive announcement Positive touch Non-verbal prompt General conversation Verbally engaged in task Physically engaged in task..15 Very neg verbal Neg verbal Gen restless Eyes closed Uncoop- erative Aggres- sion Pos verbal Very pos verbal Task en- gagement Pos touch Negative prompt.12 Negative announcement.11 Ignores resident Talks to another Rebuke/disapprove Negative touch Greeting Explanation Courtesy Offers choices Asks preference Praise/compliment Reassurance Positive verbal prompt Positive announcement Positive touch Non-verbal prompt General conversation Verbally engaged in task Physically engaged in task

26 Pearson Correlations (N=2638 real-time observation occasions) Resident Negative BehaviorResident Positive Behavior Very neg verbal Neg verbal Gen restless Eyes closed Uncoop- erative Aggres- sion Pos verbal Very pos verbal Task en- gagement Pos touch Negative CNA communication behaviors Negative prompt. Negative announcement Ignores resident Talks to another Rebuke/disapprove Negative touch Positive CNA communication behaviors Greeting Explanation Courtesy Offers choices Asks preference Praise/compliment Reassurance Positive verbal prompt Positive announcement Positive touch Non-verbal prompt General conversation Verbally engaged in task Physically engaged in task **. Correlation is significant at the level (2-tailed). Very neg verbal Neg verbal Gen restless Eyes closed Uncoop- erative Aggres- sion Pos verbal Very pos verbal Task en- gagement Pos touch Negative prompt.12 Negative announcement Ignores resident Talks to another Rebuke/disapprove Negative touch Greeting Explanation Courtesy Offers choices Asks preference Praise/compliment Reassurance Positive verbal prompt Positive announcement Positive touch Non-verbal prompt General conversation Verbally engaged in task Physically engaged in task Very neg verbal Neg verbal Gen restless Eyes closed Uncoop- erative Aggres- sion Pos verbal Very pos verbal Task en- gagement Pos touch Negative prompt Negative announcement.11 Ignores resident Talks to another Rebuke/disapprove.10 Negative touch Greeting Explanation Courtesy -.14 Offers choices Asks preference Praise/compliment Reassurance Positive verbal prompt Positive announcement Positive touch Non-verbal prompt General conversation Verbally engaged in task Physically engaged in task..15 Very neg verbal Neg verbal Gen restless Eyes closed Uncoop- erative Aggres- sion Pos verbal Very pos verbal Task en- gagement Pos touch Negative prompt Negative announcement Ignores resident Talks to another Rebuke/disapprove Negative touch Greeting Explanation Courtesy Offers choices.42 Asks preference Praise/compliment Reassurance Positive verbal prompt.48 Positive announcement.37 Positive touch.40 Non-verbal prompt General conversation.66 Engaged in task Physically engaged in task

27 Why focus on positive emotion as an outcome of choice? 1. Fredrickson, B. (2004). The broaden-and-build theory of positive emotions. Phil Trans R Soc Lond B Biol Sci. September 29; 359(1449): 1367– doi: /rstb /rstb Fredrickson, B.L., & Losada, M.F. (2005). Positive affect and complex dynamics of human flourishing. American Psychologist, 60, doi: / X doi: / X Meeks, S., VanHaitsma, K., Kostiwa, I. & Murrell, S. (in press). Positivity and Well-Being among Community-Residing Elders and Nursing Home Residents: What is the Optimal Affect Balance? Journals of Gerontology: Psychological Sciences. Broaden-and-Build Theory of Positive Emotion (Fredrickson, 2004) 1 Positive emotions are a primary means to improve psychological and physical well-being over time. Positive emotions broaden people’s momentary thought-action repertoires (enhance attention, flexibility, openness to new experiences) and build their enduring personal resources (social attachments, resiliency, creativity, enhanced cardiovascular recovery). Positive emotions serve to “undo” or “correct” the after effects of negative emotional experiences. In order for an individual to flourish, the ratio of positive to negative emotional experiences needs to be at least 3 to 1. 2,3

28 Pearson Correlations (N=2638 real-time observation occasions) Resident Positive AffectResident Negative Affect PleasureInterest Anger Anxiety Sadness Positive CNA communication behaviors Greeting Explanation Courtesy Offers choice Asks preference Praise/ compliment Reassure Positive announcement Positive touch Positive verbal prompt Non-verbal prompt Verbal task engagement. Physical task engagement Negative CNA communication behaviors Negative announcement Ignores resident Talks to another Rebuke/ disapprove Negative touch **. Correlation is significant at the level (2-tailed). PleasureInterest Anger Anxiety Sadness Greeting Explanation Courtesy Offers choice Asks preference Praise/ compliment Reassure.12 Positive announcement Positive touch Positive verbal prompt Non-verbal prompt Verbal task engagement Physical task engagement Negative announcement Ignores resident Talks to another Rebuke/ disapprove.07 Negative touch PleasureInterest Anger Anxiety Sadness Greeting.15 Explanation Courtesy Offers choice.12 Asks preference Praise/ compliment Reassure Positive announcement.09 Positive touch.13 Positive verbal prompt.14 Non-verbal prompt Verbal task engagement.07 Physical task engagement.17 Negative announcement Ignores resident Talks to another Rebuke/ disapprove Negative touch PleasureInterest Anger Anxiety Sadness Greeting Explanation Courtesy Offers choice Asks preference Praise/ compliment Reassure Positive announcement Positive touch Positive verbal prompt Non-verbal prompt Verbal task engagement Physical task engagement Negative announcement Ignores resident Talks to another Rebuke/ disapprove.07 Negative touch

29 Pearson Correlations (N=2638 real-time observation occasions) Resident Positive AffectResident Negative Affect PleasureInterest Anger Anxiety Sadness Positive CNA communication behaviors Greeting Explanation Courtesy Offers choice Asks preference Praise/ compliment Reassure Positive announcement Positive touch Positive verbal prompt Non-verbal prompt Verbal task engagement. Physical task engagement Negative CNA communication behaviors Negative announcement Ignores resident Talks to another Rebuke/ disapprove Negative touch **. Correlation is significant at the level (2-tailed). PleasureInterest Anger Anxiety Sadness Greeting Explanation Courtesy Offers choice Asks preference Praise/ compliment Reassure.12 Positive announcement Positive touch Positive verbal prompt Non-verbal prompt Verbal task engagement Physical task engagement Negative announcement Ignores resident Talks to another Rebuke/ disapprove.07 Negative touch PleasureInterest Anger Anxiety Sadness Greeting.15 Explanation Courtesy Offers choice.12 Asks preference Praise/ compliment Reassure Positive announcement.09 Positive touch.13 Positive verbal prompt.14 Non-verbal prompt Verbal task engagement.07 Physical task engagement.17 Negative announcement Ignores resident Talks to another Rebuke/ disapprove Negative touch PleasureInterest Anger Anxiety Sadness Greeting Explanation Courtesy.38 Offers choice Asks preference Praise/ compliment.42 Reassure Positive announcement Positive touch Positive verbal prompt Non-verbal prompt.37 Verbal task engagement.51 Physical task engagement Negative announcement Ignores resident Talks to another Rebuke/ disapprove Negative touch

30 Summary: Pros and Cons afforded by real time observation of treatment fidelity compared to other methods 30 Cons: Expensive and time intensive to train researchers. Is not efficient for use in “real world” of clinical practice Pros: Provide a more accurate and objective account of treatment delivery Enhanced understanding what treatment ingredients give you the biggest “bang for your buck” in outcomes Knowledge of potent treatment ingredients can lead to 1) refinements of treatment protocols 2) more targeted treatment training materials 3) refined platform for building fidelity measures used for quality improvement purposes in clinical settings.

31 Thank you! 31


Download ppt "Using observational methods to measure treatment integrity in psychosocial intervention research Kimberly S. Van Haitsma, Ph.D. Director, Polisher Research."

Similar presentations


Ads by Google