Reliability & Validity “dependability” is the indicator consistent? same result every time? Does not necessary measure what you think it measures (ex. May consistently measure something other than the concept) Validity measurement validity - how well the conceptual and operational definitions mesh with each other does measurement tool measure what we think ?

Relationship between Measurement Reliability and Validity reliability necessary for validity but does not guarantee it (“necessary but not sufficient”) measure can be reliable but invalid Source: Neuman (2000: 171)

Types of Reliability stability over time representative across different subgroups of a population (ex. of problems: young people may exaggerate their ages, older people may reduce theirs) equivalence Comparable results from multiple indicators intercoder reliability

Improving Reliability clearly conceptualize constructs increase level of measurement use pretests, pilot studies e.g. use multiple indicators : Measurement Using Multiple Indicators Neuman (2000: 167) Dependent Variable Measure Independent Empirical Association? a2 a3 a1 b1 b2 A B Specific Indicators

Selected Types of Validity: Face Validity judgement by the scientific community that indicator measures the concept Construct/concept Measure ? Scientific Community

Content Validity measure represents all the aspects of conceptual definition of construct. judgement concerning how adequately a test samples behavior representative of the universe of behavior the test was designed to sample. Love Beliefs & Values

Criterion Validity The validity of an indicator is verified by comparing it with another measure of the same construct in which a researcher has confidence. Predictive and concurrent validity Future Behavior Predictive Construct Measure Old Measure Concurrent Similar Results ?

Construct Validity A type of measurement validity that uses multiple indicators convergent and discriminate

Other Dimensions of Validity Internal Validity no error of logic internal to research design External Validity results can be generalized Statistical validity correct statistical methodology chosen ? assumptions fully met

More terms & concepts for assessing validity (usually in experimental research) Threats to Internal Validity Selection bias History effects (something in context changes) Maturation Testing effect Instrumentation Mortality Statistical regression Contamination Compensatory behaviour Experimenter expectanc Threats to External Validity Realism Reactivity

Another Aspect of Measurement Design: Levels of Measurement 1-Nominal (ex. Mother tongue) different categories (names, labels, images) not ranked 2-Ordinal (county fair prizewinners ranked by first, second & third prize) different categories rank-ordered attributes indicate relatively more or less of that variable distance between the attributes of a variable is imprecise

Levels of Measurement (cont’d) 3- Interval Measures (age measured by 5 year age groups) different categories ranked in order Can tell amount of difference between categories Usually no true zero 4- Ratio Measures (age measured by date of birth) amount of difference between categories also possible to state proportion (have a true zero) Relations between levels --can collapse from higher into lower, not vice versa

The Research Process Babbie (1995: 101)