1. You Can Observe a Lot by Just Watching: Methods for Assessing Parenting You Can Observe a Lot by Just Watching: Methods for Assessing Parenting Carla Matias Stephen B.C. Scott Thomas G. OConnor Acknowledgements: This work was made possible with funding from the ESRC Methods Programme grant, the Joseph Rowntree Foundation, The Jacobs Foundation, and the Psychiatry Research Trust. Additional funding ( CM PhD research) from Ministry of Science & Technology, Lisbon, Portugal

2. Aims & Outlook 1.To review the context in which the measures component of the ESRC programme grant was conceived 2.Describe some current (and probably accurate) thinking on the measurement of parent-child relationships 3.Describe in some detail a study that used an intervention design to tackle a major conceptual measurement question

3. Research Questions for Parenting/Family Studies 1. What are the evidence-based assessments, and is there a methodological best practice? 2. How good is the fit between constructs and their measurement? 3. Is there convergence among theories re: measurement? 4. Are there exemplar measures within each measurement approach?

4. Research Questions for Parenting/Family Studies, 1 Evidence-based assessments and best practice 1. There is robust evidence that observational, questionnaire, and interview methods all have a place at the research table 2. A Multi-method approach is a central take- home message 3. Valid & reliable measures exist, so there is no need to re-invent the wheel (although tweaking is normal)

5. Research Questions for Parenting/Family Studies, 2 Are measures good enough at capturing central constructs 1. There is general consensus re: the dimensions of the parent-child relationship that require attention 2. There is notably modest application of research measures in clinical contexts 3. There is a trade-off between detail/depth and time/money 4. There is no single best approach; neither is there a satisfactory quick & dirty way to assess the parent- child relationship

6. Research Questions for Parenting/Family Studies, 3 Theory unto measurement 1. Different theories emphasize different measures and settings, to some extent 2. There are theory-based differences in level of analysis (cf. parental commands v. mutuality) 3. A single observational paradigm might serve several models and measurement approaches

7. Research Questions for Parenting/Family Studies, 4 Exemplars and examples of measurement 1. Social learning theory: observations of coercive (harsh, negative control) interactions patterns are central 2. Attachment theory: observations of sensitive parenting, secure base behavior 3. Parenting typologies: qnaire or observations of authoritative parenting

8. Inevitable Challenges for Parenting/Family Studies Measures selection is a more confounded process than theory selection Measures selection is a more confounded process than theory selection Considerable time & money are needed to meet the best-practice standard of multi-method assessment Considerable time & money are needed to meet the best-practice standard of multi-method assessment Researcher prerogative to fiddle with and make up measures goes under- challenged Researcher prerogative to fiddle with and make up measures goes under- challenged

9. A Case Study Primary Age Learning Skills (PALS Project) Scott, OConnor & Futh, 2005 Primary Age Learning Skills (PALS Project) (Scott, OConnor & Futh, 2005

10. Primary Age Learning Skills (PALS Project) Scott, OConnor & Futh, 2005) Primary Age Learning Skills (PALS Project) (Scott, OConnor & Futh, 2005) In the context of an intervention study using a cognitive-behavioral approach, how should we assess the parent-child relationship? Should we stick to those measures used in standard assessments of this sort of intervention (i.e., behavioral counts) ? Should we stick to those measures used in standard assessments of this sort of intervention (i.e., behavioral counts) ? Which sorts of measures of relationship quality would be most sensitive to change induced by this intervention? Which sorts of measures of relationship quality would be most sensitive to change induced by this intervention?

11. Two Major Classes of Measurement of Parent-Child Relationship RESPONDENT-Based Measures - Subjective Questionnaires (parent, child) Questionnaires (parent, child) Interviews (parent, child) Interviews (parent, child) DIRECT OBSERVATION - Objective Coding of interactions by independent raters Coding of interactions by independent raters

12. Usefulness of Report Methods Advantages Lower Cost Accessibility & Flexibility - Quick Results Breadth of Assessment: 1) behaviour over longer time span & in different contexts; 2) Information on behaviour, attitudes, and other aspects of family life Disadvantages Informant Bias (e.g. distorted recall; social- desirability; interpretation issues, etc) Method Error (e.g. item- wording and scaling interpretation; not context-specific)

13. Usefulness of Observational Methods Advantages Greater Objectivity - Assessing actual rather than reported behaviour Contextual sensitivity; intensity & frequency of behaviour; Predictive Validity & Sensitive to change is well-documented Disadvantages Subject(s) Bias: reactivity effects, e.g., to the camera or researcher Method Error: 1) Context- specificity (behaviour sampled at one point in time); 2) Observer Bias Expensive and time- consuming

14. To Observe or Not To Observe? Using a single-source approach is not now regarded as scientifically acceptable because any method is affected by sizable error Using a single-source approach is not now regarded as scientifically acceptable because any method is affected by sizable error The study of complex constructs such as parenting necessarily require adoption of a multi-method multi-informant approach On the other hand, the overlap between methods is usually modest, at best

15. Primary Age Learning Skills (PALS Project) Scott, OConnor & Futh, 2005) Primary Age Learning Skills (PALS Project) (Scott, OConnor & Futh, 2005) Evaluates effectiveness of a behavioural parenting programme for an at-risk population of school-age children Multi-method assessment at pre, post and follow- up 1) Respondent-based methods (Questionnaire & Interview) - evaluate subjective change 1) Respondent-based methods (Questionnaire & Interview) - evaluate subjective change 2) Direct observation (development of new coding scheme) - evaluates objective change 2) Direct observation (development of new coding scheme) - evaluates objective change

16. A Focus on Attachment Measurement using Direct Observation: The CARP (Matias et al, 2005) What do we mean by attachment-related parenting? Core concept of Sensitive Responding - Operationalisation: Responsiveness to child Child Mindedness Facilitation Responsive Engagement Promoting Autonomy/Encouraging Warmth

17. The PALS Parenting Battery Observation of parent-child interactions in semi- structured play, structured task, and clean-up 25+ Parent questionnaires of instrumental and expressive warmth, conflict, coercion, punitiveness 15+ Parent interviews of discipline strategies and normal routines 20+ Parent interviews to assess the parents working model of the child 25+ Attachment story stem narratives with the child 25+ NB: length of assessment in small print; for observations, add 40+ for coding; for interviews add 30+ for coding; for attachment narratives add 30+ for coding

18. What is Novel about PALS? Multi-method, multi-theory prevention study in a multi-ethnic sample Multi-method Observation, interview & questionnaire of parent-child relationship and outcome Multi-theory Attachment and social learning theory approach to studying parenting in everyday situations Attachment is a neglected area of research in interventions with school-age children Attachment is a neglected area of research in interventions with school-age children Will improvements in one measure or theory translate to improvements in another? Will improvements in one measure or theory translate to improvements in another?Multi-ethnic A majority of parents (Southwark) were minority and poor

19. PALS Summary of Findings from Report Methods Changes in Parenting Changes in Parenting No change on questionnaire Interview - Increases in reported use of calm discipline and decreased use of criticism but no change in reported use of praise Changes in Child Behaviour Questionnaire & Interview: No improvement in child antisocial behaviour, Hyperactivity /ADHD & child emotional symptoms

20. Is that the end of the story?

21. DIRECT OBSERVATION: Parenting Outcome (i) 1. Change in Sensitive Responding Intention to Treat Analysis Graph 1 Effect size(es) = 0.37

22. DIRECT OBSERVATION: Parenting Outcome (ii) 2. Change in Sensitive Responding Per Protocol Analysis Graph 2 Effect Size (es) = 0.67

23. DIRECT OBSERVATION - Child Outcome (i) 3. Change in Child Attention on Task Intention to Treat Analysis Graph 3 Effect Size (es) =. 32

24. DIRECT OBSERVATION - Child Outcome (ii) 4. Change in Child Attention on Task Per Protocol Analysis Graph 4 Effect Size (es) =.78

25. Direct Observation 5. Ethnicity Effects (i) - Mean Differences at Baseline ANOVA showed that West African group scored significantly lower in SR compared to White British (p<.01) & Black Afro- Caribbean groups (p<.05)

26. Direct Observation Ethnicity Effects (ii) - Intervention Outcome 6. Despite ethnic differences at baseline, intervention was equally successful in increasing Sensitive Responding across all ethnic groups

27. PALS Study Summary: Value of Observational Methods A Social-Learning based Intervention increased Sensitive Responding, an Attachment derived concept Interventions effects may generalize across method & theory Ethnic differences in observed Sensitive Responding were not found in self-report measures How families present and represent themselves requires sensitive assessment Group differences prior to treatment does not imply group moderation of treatment effects Observed Attention on Task increased, but no change in child behaviour was detected in respondent-based measures Observed data imply real observable change, but the definitions of robust change may imply more than mono-method evidence

28. Main Conclusions Studies relying on a single method are suspect Multi-method assessments of parent-child relationship quality and outcomes are scientifically required, but difficult to attain The call for a multi-method approach implies the lack of a single gold standard: for detecting change and a successful intervention for detecting change and a successful intervention for assessing cultural variations for assessing cultural variations for defining a meaningful effect size for defining a meaningful effect size The application of findings from multi-method investigations to a clinical or social policy context will be neither direct nor easy