1. Statistics for Education Research Lecture 10 Reliability & Validity Instructor: Dr. Tung-hsien He the@tea.ntue.edu.tw

2. Chi-Square (  2) Chi-Square (  2) 1. Features: a. A nonparametric test ( 無母數分析 ): (1) Assumptions of normality & homogeneity are not met. (2) Sample size is very limited. b. To test whether differences in the observed frequency and expected frequency are significant or not.

3. c. Expected frequency is determined by the researchers. Usually it is the averaged frequency. d. Appropriate for nominal data only. e.  2 is a family of distributions that skew positively f.  2 can be used to test the magnitudes of the expected causal relation between two variables; however, it is not used to identify causality of two variables. 2. Formula: 20.1, p. 536

4. 3. Example: a. A researcher is interested in knowing whether male and female subjects will respond to questionnaire items differently. So, the researcher chooses 20 subjects to answer survey questions. b. Appropriate Stat Technique: Chi-square c. SPSS Procedures:

5. Reliability Reliability a. Meaning: The extent to which a scale may produce consistent scores. b. Types: Many types of reliability. Two widely used types are Cronbach’s α and interrater reliabiliaty. 1. Cronbach’s α = Internal Consistency of a Scale 2. Interrater Reliability (Consistency in Scores of Two or More Raters) = intraclass coefficient (ICC)

6. Factor Analysis Factor Analysis 1. Features: a. To pinpoint the construct validity of Likert scales; b. Exploratory Factor Analysis (EFA): Scales are developed as no strong theories are applied; EFA is usually run in the phase of “pilot study”; to explore the construct validity of the scale;

7. c. Confirmatory Factor Analysis (CFA): Scales are developed based on sound and clear theories or based on satisfactory results of EFA; to confirm the construct validity obtained by running EFA. d. Latent Factors: For EFA, to test whether the expected latent factors can be successfully identified by using rotation and extraction methods; for CFA, to test whether the identified latent factors can adequately account for the portion of variances

8. 2. Example a. Scenario: A researcher develops a 6-item scale. Item1 to item3 are designed to measure attitudes, whereas item4 to item 6 are designed to measure motivation. 30 subjects are selected to pilot this scale. What is reliability and validity of this scale? b. Appropriate STAT Technique: EFA C. SPSS Procedures:

9. 3P 3P PP: Using request strategies may be important for any language learners since language is used for communication. IP: However, EFL learners’ use of request strategies may be influenced by: (a) L1 and (b) social variables such as status and familiarity. SP: These variables will exercise effects on young EFL learners’ using request strategies.

10. Literature Review Literature Review 1. Significances of variables being studied? 2. Links between PP & IP? 3. Theoretical Framework? Method Method 1. Experimental(?) 2. Subjects: 12 Chinese children & 6 native speakers (Randomly Selected? Why Native Speakers? Size is large enough?]

11. 3. Statistical Techniques: Chi-Square (See Table1, p. 420) a. Two-Way ANOVA? b. p .05? c. Effects?

12. Factor Analysis Factor Analysis 1. EFA: Table 5, p. 19 2. CFA: Figure 1, p. 20 Criticisms: 1. Why EFA & CFA on the same subjects? 2. Are loadings of latent factors high enough? (See CFA, Item1 & Perform, p. 20). 3. Any fit-of-goodness indexes?