Presentation is loading. Please wait.

Presentation is loading. Please wait.

Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July.

Similar presentations


Presentation on theme: "Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July."— Presentation transcript:

1 Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July 11-12, 2005

2 Are measures important?  Measures are how theory is translated into data for modeling  Unlike art, where we want to engage individual interpretation as a major component of the work  Measures need to be unambiguously interpreted in precisely the same way by all

3 Why Scales?  Combining the results of several questions is more reliable and precise, and reduces measurement error (Spector, 1992)

4 What makes a good scale?  It is reliable The same measurement, produces similar results across time and persons The questions or items that make up the scale are consistent with one another  It is valid It measures the concept that it is intended to measure  It measures one concept  It is regularly distributed

5 Reliability  Internal Reliability Internal consistency: the items should inter- correlate (Cronbach’s Alpha)  Test-Retest Reliability Same measure taken again should produce consistent results

6 Validity  Rationale The items must be consistent with theory and the concept under consideration  Criterion Validity The items should vary in the same way to other variables that are theoretically related to the concept Canonical correlation

7 Uni-dimensional, Efficient  Scales that measure a single concept are more useful in establishing causal relationships  Efficient scales use only enough items to reliably measure a concept  Regular distributions allow standard modeling techniques

8 Research Question  If my scales are not uni-dimensional, are there sub-scales which are and meet the criteria associated with good scales?  Investigated using data gathered for the Canadian National Longitudinal Survey of Children and Youth by Statistics Canada.

9 Data and Method  The NLSCY cycles 1, 2 and 3 Measures on parenting and child behaviour taken every two years (allows the investigations of test- retest reliability) N > 13,000.  Common factor analysis Dimensions, alpha  Canonical correlation Criterion validity

10 Problems  There are a lot of candidate scales 2 k – 1, if scales of all sizes are considered  The testing procedure is lengthy Factor analysis (dimensionality) Alpha (internal reliability) Canonical Correlation (external validity) Re-test (all of the above, twice) More than 13 scales

11 Solution: optifact  A Stata software component to analyze a list of candidate items for the creation of a summated rating scale

12 -optifact- Specification optifact varlist [weight] [if exp] [in range] [, top(#) smallest(#) largest(#) maxfact(#) minslope(#)] Varlist: the list of candidate items (numeric) Options: Top: the number of candidate scales to list (10) Smallest: the smallest k allowed (3) Largest: the smallest k allowed (all items) Maxfact: the maximum number of factors allowed (1)

13 -optifact- Output optifact abecq6b abecq6n abecq6w abecq6p abecq6i abecq6qq abecq6hh abecq6s [aweight=awtcw01c], top(20) criteria(childsex ammcq01) 8 items will be processed There are 219 potential scales 56 combinations of 3 items had one factor, 0 had more than one factor Top 5 scales using 3 items Avg. Can K Alpha Cov. Chk Items - ----- ----- --- ----- 1 3 0.774 0.201 Yes abecq6p abecq6i abecq6qq 2 3 0.769 0.203 Yes abecq6p abecq6i abecq6hh 3 3 0.744 0.231 Yes abecq6b abecq6n abecq6hh 4 3 0.742 0.169 Yes abecq6p abecq6qq abecq6hh 5 3 0.734 0.213 Yes abecq6n abecq6p abecq6i 55 combinations of 4 items had one factor, 15 had more than one factor Top 5 scales using 4 items Avg. Can K Alpha Cov. Chk Items - ----- ----- --- ----- 1 4 0.802 0.185 Yes abecq6p abecq6i abecq6qq abecq6hh 2 4 0.784 0.199 Yes abecq6n abecq6p abecq6i abecq6hh 3 4 0.776 0.191 Yes abecq6n abecq6p abecq6i abecq6qq 4 4 0.775 0.198 Yes abecq6b abecq6p abecq6i abecq6hh 5 4 0.768 0.172 Yes abecq6p abecq6i abecq6qq abecq6s

14 Behaviour Scales VariableConceptskAlpha Hyperactivity (becs06) 380.84 Prosocial Behaviour (becs07) 4100.76 Emotional Disorder /Anxiety (becs07) 380.78 Aggression (becs09 260.78 Indirect Aggression (becs10) 150.78 Property Offences (becs11) 260.62

15 Parenting Scales VariableConceptskAlpha Positive interaction (0-23 months) (prcs01) 250.73 Ineffective parenting style (0-23 months) (prcs02) 120.37 Positive interaction (2-11 years) (prcs03) 250.81 Ineffective parenting style (2-11 years) (prcs04) 370.70 Consistency (2-11 years) (prcs05) 250.68 Rational parenting style (2-11 years) (prcs06) 260.62 PMK Depression (dpps01) 4120.82

16 Results  Of these 13 scales only one was uni- dimensional  Most had larger number of items than desirable for the value of alpha

17 New Behaviour Scales VariableConceptskAlphaConcepts’k’Alpha’ Hyperactivity (becs06) 380.84130.80 Prosocial Behaviour (becs07) 4100.76140.77 Emotional Disorder /Anxiety (becs07) 380.78130.62 Aggression (becs09) 260.78130.67 Indirect Aggression (becs10) 150.78130.70 Property Offences (becs11) 260.62120.63

18 New Parenting Scales VariableConceptskAlphaConcepts’k’Alpha’ Positive interaction (0-23 months) (prcs01) 250.73130.63 Ineffective parenting style (0- 23 months) (prcs02) 120.37 Positive interaction (2-11 years) (prcs03) 250.81130.78 Ineffective parenting style (2- 11 years) (prcs04) 370.70 Consistency (2-11 years) (prcs05) 250.681 3 0.66 Rational parenting style (2-11 years) (prcs06) 260.62 PMK Depression (dpps01) 4120.82130.75

19 Implications  These scales are part of a large survey conducted every 2 years in Canada on more than 20,000 children  The survey has completed 6 cycles of data collection  Each question has been asked more than 120,000 times

20 Implications (Cont’d)  The scales required 85 questions  The revised scales require 30 questions  More than 55,000,000 questions

21 Caveats  Most scales in the NLSCY are not the best that can be made from the available data  Some of these scales should not be used  The NLSCY is an excellent survey, conducted by competent statisticians  Similar results might be expected in other major surveys

22 Conclusions  -optifact- can help find better measures Uni-dimensional Valid Reliable  -optifact- can reduce costs Equivalent or better measure for same money


Download ppt "Creating Valid and Effective Measures Using –optifact- to Create Better Summated Rating Scales North American Stata Users' Group Meetings Boston, July."

Similar presentations


Ads by Google