1. Research Methods in MIS
Dr. Deepak Khazanchi

2. Measurement of Variables: Scaling, Reliability and Validity

3. Major Sources of Errors in Measurement
Since 100% control for precise and unambiguous measurement of variables is unattainable, error does occur. Much potential error is SYSTEMATIC (results from a bias) while the remainder is RANDOM (occurs erratically).
Sources of Measurement Differences
Respondents
Situational factors
Measurer or researcher
Data collection instruments

4. Validity and Reliability
Accuracy of measurement. The degree to which an instrument measures that which is supposed to be measured.
Validity Coefficient: An estimate of the validity of a measure in the form of a correlation coefficient.
Reliability:
Consistency of measurement. The degree to which an instrument measures the same way each time it is used under the same conditions with the same subjects.
Reliability Coefficient: An estimate of the reliability of a measure usually in the form of a correlation coefficient.
From Sproull (1995):
Validity is an extremely important and perhaps the most important, aspect of a measure. If an instrument does not measure accurately what it is supposed to measure, there is no reasons to use it, even if it does measure consistently.
Of validity and reliability, validity is far more important than reliability for the reason given in #1.
It is questionable practice to use measures for which: (1) there is no evidence of validity or (2) there is evidence of validity but the validity estimates are low.
Validity is specific. An instrument or measure can be valid for a specific criterion but not for other criteria, for a specific group of people but not other groups.
There are several types of validity and reliability estimates. Because of the confusion in using different names and meanings for the various types of estimates, standard labels and standard definitions for the various types of both validity and reliability estimates have been determined.
A single instrument may have several types of validity and reliability estimates.
Each type of validity and reliability estimate has a different purpose.
Validity and reliability are always estimated, not proven.
The statistical symbol for a validity coefficient is rxy indicating correlation of two different measures. The statistical symbol for a reliability coefficient is rxx indicating two measures of the same variables.
A typical validity coefficient would be approximately .45 or higher. Higher would be better, However, validity coefficients rarely exceed .60 and many are in the range of .30 to .40.
A typical reliability coefficient for a a researcher designed instrument is approximately .70 or higher. For an instrument designed by a testing service, one would expect .90 or higher.
If the researcher designs an instrument, validity and reliability estimates should be made by the researcher. If an instrument is purchased, the company from which it is purchased should provide validity and reliability information which the researcher examines prior to purchase.

5. Possible Conditions of Validity and Reliability
When examining an instrument for validity and reliability remember that three types of conditions may exist. An instrument might show evidence of being:
Both valid and reliable or
Reliable but not valid or
Neither reliable nor valid
NOTE: An instrument which is valid will also have some degree of reliability.

6. About Reliability and Validity Coefficients
Validity and reliability are estimated by using correlation coefficients. These statistics estimate the degree of validity or reliability.
Thus, it is not a question of an instrument having or not having validity or reliability;
It is a question of to what degree is an instrument valid for a specific purpose and to what degree does the instrument evidence specific types of reliability.
Reliability estimates are done after validity is assessed.
We will discuss the notions of internal and external validity in the context of experimental designs.

7. Types of Validity Content Validity
The representativeness of the content of the instrument to the objectives of using the instrument.
Usual Process:
1. Examine objectives;
2. Compare objectives to content of instrument.

8. Types of Validity (cont’d)
Criterion-Related Validity
Predictive: The degree to which a measure predicts a second future measure.
Usual Process:
1. Assess validation sample on predictor;
2. Assess validation sample on criterion at later time;
3. Correlate scores
Concurrent: The degree to which a measure correlates with another measure of the same variable which has already been validated.
1. Assess validation sample on new measure;
2. Assess validation on already validated measure of same variable at about the same time;
3. Correlate scores.

9. Types of Validity (cont’d)
Construct Validity
The degree to which a measure relates to expectations formed from theory for hypothetical constructs
Usual Process
1. Assess validation sample on major variable
2. Assess validation sample on several hypothetically related variables
3. Analyze to see of major variable differentiates Ss on the related variables

10. Types of Reliability (Consistency) Estimates
V.IMP: A MEASURE CAN BE RELIABLE BUT TOTALLY LACK VALIDITY.
Stability
Test-retest
Equivalence
Parallel forms
Internal Consistency
Split-half
KR20 (Kuder-Richardson)
Coefficient (Cronbach’s) alpha
Interater reliability

11. Types of Reliability (Consistency) Estimates: STABILITY
Test-Retest Reliability
Used to assess the stability of a measure over time.
Usually indicated by a correlation coefficient.
Number of forms (of instrument): 1
Number of administrations: 2
Usual Process:
Administer the instrument to the reliability sample at Time 1.
Wait a period of time (e.g., 2-4 weeks)
Administer copies of the same instrument to the same sample at Time 2.
Correlate the scores from Time 2 and Time 1.

12. Types of Reliability (Consistency) Estimates: EQUIVALENCE
Equivalence Forms Reliability (Also known as Parallel Forms or Alternate Forms Reliability).
Used to assess the equivalence of two forms of the same instrument.
Usually indicated by a correlation coefficient.
Number of forms (of instrument): 2
Number of administrations: 2
Usual Process:
Administer Form A of the instrument to the reliability sample
Break the sample for a short rest period (10-20 minutes)
Administer Form B of the instrument to the same reliability sample
Correlate the scores from Form A and Form B
Needed when two or more versions (Forms) of the instrument will be used.

13. Types of Reliability (Consistency) Estimates: INTERNAL CONSISTENCY
Split-Half Reliability
Used to assess the internal consistency or equivalence of two halves of an instrument.
Usually indicated by a correlation coefficient plus Spearman-Brown Prophecy Formula.
Number of forms (of instrument): 1
Number of administrations: 1
Usual Process:
Obtain or generate an instrument in which the two halves were formulated to measure the same variable.
Administer the instruments to the reliability sample.
Correlate the summed scores from the first half (often the odd numbered items) with the summed scores from the second half (often the even numbered items).
Computer the Spearman-Brown Prophecy Formula to correct for splitting one instrument into halves.

14. Types of Reliability (Consistency) Estimates: INTERNAL CONSISTENCY
KR20 (Kuder-Richardson Reliability)
Used to assess the internal consistency of items on an instrument when responses are dichotomous.
Usually indicated by the correlation generated using the KR-20 formula (There are other forms of this formula. This is used when there are two responses: correct or incorrect).
Number of forms (of instrument): 1
Number of administrations: 1
Usual Process:
Generate or select an instrument.
Administer the instrument to the reliability sample.
Compute the variance (x)2 of the scores.
Computer the proportion of correct and incorrect responses to each item.
Compute the KR-20 formula.

15. Types of Reliability (Consistency) Estimates: INTERNAL CONSISTENCY
Coefficient Alpha (Cronbach’s Alpha)
Used to assess internal consistency of items on an instrument when responses are nondichotomous.
Usually indicated by the coefficient generated using Cronbach’s formula (more generic version of the KR-20 formula)
Number of forms (of instrument): 1
Number of administrations: 1
Usual Process: Same as previous slide.

16. Types of Reliability (Consistency) Estimates: INTERSUBJECTIVE
INTERRATER RELIABILITY
Used to assess the degree to which two or more judges (raters) rate the same variables in the same way.
Usually needed when two or more judges (raters) will be used in a research study.
Usual Process:
Select or generate an instrument
Randomly select a number of objects or events to be rated
Train the raters
Have each rater rate each object or event independently
Correlate the scores of the two raters.

17. Practicality of Measurement
Practicality has been defined in terms of the following three characteristics: Economy, Convenience and Interpretability
Economy
Some trade-off between ideal needs and budget
Instrument length (limiting factor: cost)
Choice of data collection method
Need for Fast and economical scoring
The scientific requirements of a project call for the measurement processes to be reliable and valid, while the operational requirements call for it to be practical.

18. Practicality of Measurement (cont’d)
Convenience
A measuring device passes the convenience test if it is easy to administer.
Detailed and clear instructions with examples, if needed.
Pay close attention to design and layout
Avoid crowding of material, carryover of items from one page to another
Interpretabililty
Relevant when persons rather than test designers interpret results. In that case, test designers should include:
A statement of the functions the test was designed to measure and the procedures by which it was developed
Detailed instructions for administering and scoring
Evidence of reliability etc.