MEASUREMENT: RELIABILITY AND VALIDITY Reliability: dependability or consistency The numerical results produced by by an indicator do not vary because of characteristics of the measurement process or measurement instrument itself. 3 types of reliability: Stability reliability Representative reliability Equivalence reliability

Stability reliability Rel. across time. Does the measure deliver the same answer when applied in different time periods? Weight-scale example Test-retest (reliability Representative reliability Rel across subpopulations or groups. Does the indicator deliver the same result when applied to different groups? Ex: measure of critical thinking---for gender/religion...

Equivalence reliability: Applies when researchers use multiple measures (several items in a questionnaire measuring the same construct) Does the measure yield consistent result across different indicators? Ex: nationalism –10 items İntercoder reliability When observers or raters agree with each other Ex: content analysis studies

VALIDITY How well the conceptual and operational definitions mesh with each other The better the fit, the greater the validity More difficult to achieve than reliability Cannot have absolute confidence in validity Same as epistemic relation Hypothetical correlation between the indicator and the construct

Four types of validity: Face validity Content validity Criterion validity Concurrent validity Predictive validity Construct validity Divergent discriminant

Face val: judgement of the researcher Content val: is the full content of the definition represented in a measure? Ex: definition of feminism—equality in arts, politics, family, work, authority relations.... Criterion : uses some standard or criterion to indicate a construct accurately The validity of an indicator is verified by comparing it with another measure of the same construct 2 types 1. concurrent: an indicator must be associated with a preexisting indicator that is judged to be valid. Ex: two diiferent measures of totalitarianism should be highly correlated

2. predictive validity: An indicators predicts a future event that is logically related to a construct İt cannot be used for all measures Should not be confused with prediction in hypothesis testing Ex: GRE test (or LES) has high predictive validity if the graduate students who took higher grades in GRE or LES do well in graduate program. Ex: measure of political conservatism—should give higher scores with AKP members, but lower scores with DTP members.

Construct validity İf the measure is valid do the various indicators operate in a consistent manner? For measures with multiple indicators Convergent validity Applies when multiple indicators converge or associate with one another Whether multiple measures of the same construct hang together or operate in a similar manner Discriminant validity Opposite of convergent Means that the indicators of one construct hang together , but also diverge or negatively associated with opposing constructs Ex: conservatism—liberalism

Relationship between reliability and validity Reliability is necessary for validity But it does not guarantee that a measure will be valid Not a suficient condition for validity A measure may produce the same result over and over again, but it does not mean that it fits the construct Sometimes as validity increases reliability is more diificult to attain. This occurs when a construct is highly abstract (alienation ex)

Other uses of the terms reliability and validity Internal validity: There are no errors internal to the design of the research External validity: İs used primarily in experimental research It is the ability to generalize findings from one setting to a broad range of settings Statistical validity: Means that the correct statistical procedure is chosen and its assumptions are fully met. Ex: race 1.white 2. african-american 3. asian İt does not make any sense to say that the average race is 1,5.

Levels of Measurement Continous and discrete variables: Continous variables have an infinite number of values or attributes that flow along a continuum. Ex: temparature, age, income, amount of schooling,... Discrete variables: Have relatively fixed set of seperate values or variable attributes İnstead of a continuum, they contain distinct categories Ex: gender (male/female); religion (muslim/protestant/catholic...) Whether a variable continous or discrete affects its measurement level

Nominal: indicates only that there is a difference among categories Nominal measures Ordinal measures İnterval measures Ratio measures Nominal: indicates only that there is a difference among categories Ex: religion, marital status, gender Ordinal: indicates a difference and categories can be ordered or ranked Ex: letter grades (A B C), strongly agree-strongly disagree İnterval: it can specifiy the amount of distance between categories Ex: (IQ scores, scores on several scales) Has arbitrary zero Ratio: there is a true zero which makes it possible to state relations in terms of proportion or ratios Ex: income, years of schooling,

Nominal—discrete Ordinal—discrete Continous variables can be measured at the interval or ratio level A ratio level measurement can be turned into an internal, ordinal, or nominal. The interval level can always be turned into an ordinal or nominal, but not the other way around. The ratio level is rarely used in social sciences