2. Cultural Nuances, Assumptions and the Butterfly Effect: A Prelude A Brief Introduction to Generalizability Theory for the Uninitiated

3. What is Generalizability Theory? Generalizability Theory—the theory that deals with the degree to which one can say that conclusions drawn on the basis of data collected from sampling different domains can be applied with confidence to those domains.

4. Still Lost? What we’re talking about you best know as instrument reliability. Commonly known examples: inter-rater reliability, test-retest reliability, split-halves reliability, Kuder-Richardson reliabilities, Hoyt reliability, and Cronbach’s alpha

5. Test (True) Score Theory Observed scores are composed of true scores and error. The variance of the observed scores is partitioned. Then estimates are combined to produce a coefficient. X o = X t + X e X=score, t=true, o=observed, e=error S o 2 = S t 2 + S e 2 S 2 = variance r 11 = S t 2 /S o 2

6. True Score Theory and Generalizability Generalizabilty is an extension of true- score theory. Sources of Variation are viewed from different perspectives. S 2 O = S 2 LG + S 2 TG + S 2 LS + S 2 TS L=long term, T=temporary G=general, S=specific

7. So what does this have to do with Multivariate Analysis? All techniques we have studied deals with the degree of overlap of variance (information) for sets of variables (Tabachnick & Fidell, 2001). In the present case the overlap of interest is between observed variance for evaluators’ opinions about the spheres involved and the actual underlying dimensions influencing interpersonal interactions perceived by the evaluators--“true-score” variance. S 2 TOT = S 2 E + S 2 V + S 2 Sc + S 2 e E=evaluator, V=vignette,Sc=scale, e=error

8. Relationship of Gerneralizability to Multivariance To produce the variance estimates needed ANOVA is used. In this situation a 6 x 7 x 46 Evaluator (E) x Sphere + Primary Category (S) x Vignette (V) Mixed Effects Repeated Measures ANOVA is used. Evaluator and Vignette are considered random, Sphere fixed.

9. ANOVA DESIGN Doubly Repeated Measures Scale and Vignette Within Evaluator Between

10. Table 1 Variance Components Derived from the SPSS GLM repeated Measures Calculation for Overall Evaluations Source of Variation SS df MS Vignette (V) 29.220 45.649 Sphere (S)639.865 6106.644 Evaluator (E) 61.972 5 12.394 V x S 185.516270.687 V x E 96.528225.429 S x E 340.184 30 11.339 V x S x E Error (e) Residual 648.149 1350.480

11. Table 2 Variance Components and Expected Mean Squares for Overall Evaluations Source of VariationMeans Square (MS)Expected Mean Square (EMS)Estimated Variance Vignette (V).649  2 e + +  2 VE +  2 V.220 Sphere (S) 106.644  2 e + +  2 VS +  2 SE +  2 S ≥94.618 Evaluator (E) 12.394  2 e + +  2 VE +  2 E 11.965 V x S.687  2 e + +  2 VS ≤.207 V x E.429  2 e + +  2 VE ≤-.059 S x E 11.339  2 e + +  2 SE ≤10.859 V x S x E  2 e +  2 VSE ≥.480 Error (e)  2 e ≥.480 Residual.480.480 Total (TOT)  2 e +  2 S +  2 V +  2 E ≤107.763 ≥107.283  = S 2 S / S 2 TOT ≥ 94.618 / 107.763 = 0.879 0.879 ≤  ≤ 0.886 ≤ 95.094 / 107.283 = 0.886

12. Table 3 Variance Components Derived from the SPSS GLM repeated Measures Calculation for Spheres Source of Variation SS df MS Vignette (V) 12.426 45.276 Sphere (S) 22.575 5 4.515 Evaluator (E) 16.996 5 3.399 V x S 73.730225.328 V x E 27.643225.123 S x E 24.769 25.991 V x S x E Error (e) Residual 168.092 1125.149

13. Table 4 Variance Components and Expected Mean Squares for Spheres Source of VariationMeans Square (MS)Expected Mean Square (EMS)Estimated Variance Vignette (V).276  2 e + +  2 VE +  2 V.153 Sphere (S) 4.515  2 e + +  2 VS +  2 SE +  2 S ≥3.345 Evaluator (E) 3.399  2 e + +  2 VE +  2 E 3.276 V x S.328  2 e + +  2 VS ≤.179 V x E.123  2 e + +  2 VE ≤-.026 S x E.991  2 e + +  2 SE ≤.842 V x S x E  2 e +  2 VSE ≥.149 Error (e)  2 e ≥.149 Residual.149.149 Total (TOT)  2 e +  2 S +  2 V +  2 E ≤6.923 ≥7.072  = S 2 S / S 2 TOT ≥ 3.345 / 7.072 = 0.473 0.473 ≤  ≤ 0.505 ≤ 3.494 / 6.923 = 0.505

14. A Sample Calculation As an interesting example of how the  ’s are calculated here is one (see Table 2 for the data) Sphere (MS) = 106.644  2 e + +  2 VS +  2 SE +  2 S -V x S (MS) =.687  2 e + +  2 VS -S x E (MS) = 11.339  2 e + +  2 SE +Error (MS) ≤.480  2 e Sphere (Variance) ≤ 95.094  2 S -Error (MS) ≤.480  2 e Sphere (Variance) ≥ 94.618 Total (TOT) ≤ 107.763  2 e +  2 S +  2 V +  2 E ≥107.283  = S 2 S / S 2 TOT ≥ 94.618 / 107.763 = 0.879 ≤ 95.094 / 107.283 = 0.886 0.879 ≤  ≤ 0.882

15. The Conclusion What we get is three little numbers that are not as simple as they look: Primary Influence:  =.966 Spheres:  =.473 Overall:  =.879