Presentation on theme: "How do measures measure up?. What is the Centre? Knowledge Bringing people and knowledge together to promote the best mental health and well-being for."— Presentation transcript:
How do measures measure up?
What is the Centre? Knowledge Bringing people and knowledge together to promote the best mental health and well-being for every child and youth CapacityPartnerships Full continuum of effective and accessible mental health services for children and youth.
Brief Introductions Name and affiliation Program to be evaluated One outcome of interest
Objectives To provide an overview of concepts on measurement To provide guidelines for assessing and selecting measures To reduce fear on the topic of measures and indicators
Outline A.What is measurement? B.What are the different sources of data and types of measures? C.What affects the quality of measures? D.How can we select appropriate measures for our evaluation?
A. What is measurement?
Getting Ready for Evaluation Evaluatio n Heaven! Develop Logic Model Identify Priority Outcomes Review and Select Measures Develop Data Collection Procedures
What is measurement? Measurement refers to the process of operationalizing the evaluation question Evaluation question Information needed Sources of information Types of data Evaluation Measure
Sample Evaluation Question: What are clients perceptions of our services? Has client satisfaction with our services improved? What areas of the program do we need to improve on? Evaluation question Information needed Sources of information Types of data Evaluation Measure
Information from parents and/or youth on: Satisfaction with location Ease in accessing the services Perceptions of outcomes as a result of the service Appropriateness of the services Perceptions on therapeutic alliance or relationship Evaluation question Information needed Sources of information Types of data Evaluation Measure
B. What are the different sources of data and types of measures?
Levels of measurement Nominal Ordinal Interval Ratio
Sources of information Questionnaires Interviews or focus groups Observation Administrative data and/or Health records
Sources of information from parents and/or youth on how they experience our services: Interview Focus groups Administrative data: number and type of complaints Self-administered survey Standardized questionnaire Evaluation question Information needed Sources of information Types of data Evaluation Measure
Qualitative data: verbal and pictorial Numeric scores: Basic units such as frequency, duration, length Scores from rating scales Standardized scores (z scores) Age- or grade-equivalent or adjusted scores Criteria for evaluating scores: Standardization: procedural comparability Psychometric properties Types of Data
Type of data on clients satisfaction or perception of care Scores on a standardized measure on client perception of care: Average of ratings on all items Changes in average ratings every 6 months Evaluation question Information needed Sources of information Types of data Evaluation Measure
C. What affects the quality of measures?
Cultural and Historical Context Use of tests and measures as a cultural tool Cultural appropriateness of the measures Revisions or updates to measures to accommodate changes in scores in the population
Psychometric Properties: Validity Does the measure REALLY measure what it is supposed to measure? Face validity Content Validity Construct, Criterion and Factorial Predictive Concurrent Discriminant or Divergent
Psychometric Properties: Reliability = Dependable = Trustworthy = Same old, Same old
Cost and availability Time for administering, scoring and analyzing Staff involvement Information management Psychometric Properties: Feasibility
Clinical utility Culturally appropriate Administrative uses for decision-making Usefulness for improving program Usefulness for information on public reports Psychometric Properties: Relevance
Other measurement considerations when involving children Developmental level Educational level Health status Family
D. How can we select appropriate measures for our evaluation?
Criteria for selecting measures Evidence-based Feasible Relevant and meaningful
Evidence-based Sound psychometric properties: valid and reliable Used in similar settings Recommended by experts, if no previous literature
Sample Summary Tables
Sample Summary Table Name of Measure Reference or Developer Areas measured Evidence on reliability and validity Feasibility (cost, training, who administers) RelevanceComments
Evaluation question Information needed Sources of information Types of data Evaluation Measure General Information Developer/ AuthorsChildrens Indicator Workgroup of Sixteen State Study, Centre for Mental Health Services. Date of publication, versions available Separate forms for parents and youth. Constructs measuredThe YSS is used as a measure of youth service usage and satisfaction with services. The YSSF is used as a measure of parent report of youth service usage and satisfaction with services. Population for which designed Adolescents ages 13 and up. Method of administration Self-report questionnaire, paper or telephone interview. Subscales and number of items 25 items. Five scores based on a factor analysis can be obtained: Good access to services Participation in treatment Cultural sensitivity of staff Appropriateness of services Positive outcome of service Estimated time to administer Not mentioned. Costs, availability and permission to use In public domain. See online resources below. Examiner qualifications and training requirements Information not available at this time. Youth Services Surveys (YSS, YSSF)
Technical Information Sample for development of norms The YSS was created as part of the Mental health Statistics Improvement Program for the Sixteen State Study. To date 430 youth have completed the survey. ReliabilityReliability indices for the 5 factors of the survey are good: Access =.713, Participation in Treatment =.776, Cultural Sensitivity =.863, Appropriateness =.863, Outcome =.893 (Brunk et al 2000) ValidityThis indicator was developed by the U.S. Substance Abuse and Mental Health Services Administration (SAMHSA) for the National Mental Health Performance Measures, and feasibility was assessed across 16 states (Lutterman et al, 2003). The Virginia Department of Mental Health, Mental Retardation, and Substance Abuse Services coordinated the development of the surveys for children and youth (Brunk et al 2000). The survey is widely used in the U.S. and reports by various states are available in the grey literature. UsesThe YSS may be used for the following: (1) As a data collection method for the following: Client perception of care, consumers linked to physical health services, children in family-like arrangements and other 24- hour residential care programs. (2) As a clarification method for findings from other indicators. (3) As a measure of penetration (when providers serve a higher than average percentage of youth with a particular problem than occurs in the general population. Review and Comments Applicability for Ontario Information not available at this time. Other considerations Information not available at this time. Resources Key ReferencesBrunk, M. (2001). Youth Services surveys Guidelines for administration. [Power Point presentation]. Brunk, M., Koch, J.R., McCall, B. (2002). Report on Parent satisfaction with Services at Community Services Boards. Virginia Department of Mental Health, Mental Retardation and Substance Abuse Services. Online Resourceshttp://www.mhsip.org Contact information Youth Services Surveys (YSS, YSSF)
Evaluation question Information needed Sources of information Types of data Evaluation Measure Measures of Therapeutic Alliance Blatt & Zuroff 2005: Importance of including therapeutic alliance as a factor moderating treatment outcomes science direct link science direct link Elvins & Green 2008: Review of concept and measures on therapeutic alliance science direct link2 science direct link2 Green 2006: Measures on therapeutic alliance for child and youth mental health interscience link interscience link Blatt & Zuroff 2005: Importance of including therapeutic alliance as a factor moderating treatment outcomes science direct link science direct link Elvins & Green 2008: Review of concept and measures on therapeutic alliance science direct link2 science direct link2 Green 2006: Measures on therapeutic alliance for child and youth mental health interscience link interscience link
Evaluation question Information needed Sources of information Types of data Evaluation Measure Measures on Cultural Competency Cultural Competency Self-Evaluation Questionnaire friends link friends link
Some common issues To translate or not Picking and choosing items from various measures Making sense of data: indicators, benchmarks and standards
Decision process for identifying, selecting and using measures Specify topic of inquiry IdentifyingSelectingUsing Conduct literature review Identify existing measures Identify Gap: NO existing measures Examine evidence, feasibility & relevance Use measure Revise measure Develop new measure Collect & Analyze Data (as Developed) Gather new evidence on validity and reliability
Steps in Translating Measures Translate to second language Back translate to first language Pilot-test with experts & end-users Gather new evidence on validity and reliability Compare results with original version
Measures Matrix Evaluation question Outcome or process variable Indicator or measure Where data comes from How data is collected data & how frequently Who collects data When data is collected How data will be analyzed
Sample Measures Matrix Evaluation questionDo clients perceive outcomes as a result of the care/ services they receive? Process or Outcome variable Client perception of care Indicator/ MeasureYouth Services Survey Source of dataYouth 16 to 18 years, and Parents of children and youth below 16 years How data is collectedSelf-administered questionnaire, in-person and by mail Who collects dataReceptionist When data is collectedAt termination of the program, at least 3 months in program Plan for analysisCompare means of total scores for those in Program A with those in Program B
Questions? Key insights?
Summary Measurement involves specifying or operationalizing the evaluation question. Process or outcome variables can be assessed using numerical or qualitative data. Sources of information include questionnaires, interviews, focused groups and administrative data.
Summary, Cont. Summary tables assist in organizing information about various measures, and help in making informed decisions on the most appropriate measure for the evaluation. A matrix of the indicators and measures to be used in the evaluation summarizes how, where and when data is to be collected.
Next Steps Review logic model and identify evaluation questions Specify process and outcome variables Summarize literature review Review and select measures Create indicators and measures matrix Develop protocol for collecting data Develop plan for analyzing data
Future webinars? February 2009: Using Excel and SPSS for basic statistics March 2009: Writing the Final Report
For more information Evangeline Danseco, PhD Head, Evaluation and Research , ext