Download presentation

Presentation is loading. Please wait.

Published byCorinne Allis Modified over 2 years ago

1
William van der Veld (University of Amsterdam) Willem Saris (ESADE Business School) Barcelona, 2005, European Association for Survey Research THE NATURE OF MEASUREMENT ERROR IN PANEL DATA Reliability and Opinion Stability

2
2 Overview The RUSSET panel (www.vanderveld.nl) Reliability estimation in a panel design Reliability estimation in a cross-section design Comparison of Panel & Cross-section The VAS Model Conclusion

3
3 The RUSSET panel Please, tell me, how satisfied are you with your current life as a whole? Not at all Very Satisfied Satisfied Aggregate level [means]Individual level (Correlations)

4
4 Reliability estimation in a panel design Quasi-simplex model: Assumptions The random error variance is the identical for the repeated measures; For each respondent the attitude changes according to a lag-1 quasi- simplex; The repeated measures are independent. Interpretation y: The observed variable e: the random error in y T = y + e When T and y standardized, β : the stability coefficient λ : reliability coefficient

5
5 Reliability estimation in a panel design Estimation of Reliability & Stability RUSSET Data: 3 waves, 3 times the question: How satisfied are you with your current life…? n=837; Chi-square=0; df=0 [Lisrel, ML estimation] Completely Standardized Solution λ1λ1 λ2λ2 λ3λ3 s 21 s

6
6 Reliability estimation in a panel design The observed stability is: 0.34 & After correction for measurement error, the stability is: 0.99 & The reliability coefficient of a simple question as How satisfied are you with your current life as a whole? is quite poor: ±0.6 (reliability=0.36) Is there an alternative way to estimate the reliability, so that we can compare both estimates?

7
7 Reliability estimation in a cross-section design Parallel-test model: Assumptions The random error variance is the identical for the repeated measures; For each respondent the attitude has not changed during the interview; The repeated measures are independent. Interpretation y: The observed variable e: the random error in y T = y + e When T and y standardized, λ : reliability coefficient

8
8 Reliability estimation in a cross-section design Estimation of Reliability (not Stability) RUSSET Data: 1 wave, 2 times the question: How satisfied are you with your current life…? Same respondents (exactly), same question. n=837; Chi-square=1; df=1 [Lisrel, ML estimation] Completely Standardized Solution λ1λ1 λ 2-6 λ λ3λ

9
9 Reliability estimation in a cross-section design THATS STRANGE!!!! Same question, same respondents, same moment Same definition: Still estimated reliability PTR(6)=0.85 vs. QSR(6)=0.58 No confirmation because they are different. So, this result does not help Is there a way to check which reliability is correct?

10
10 Comparison of QSR & PTR Use the reliability-estimates to correct the observed correlation between different variables for measurement error. ρ 21 = r 21 /( λ Tx1 * λ Tx2 ) We have observed the correlation: r 21 We have estimated the reliability coefficients with QS & PT model. We are interested in: ρ 21 Which ρ 21 is more plausible? That obtained with the QS-estimates or that with the PT-estimates

11
11 Comparison of QSR & PTR Exactly the same data (n=837), Wave 1995(6,7). Correction of the observed correlation between: Satlife & Satinc. Satlife: How satisfied are you with your current life as a whole? Satinc: How satisfied are you with your familys current financial situation? Observed correlation between Satlife & Satinc = 0.51 The reliability coefficient estimates are:

12
12 Comparison of QSR & PTR Standardized estimates of reliability coefficients QSRCPTRC λ1λ1 λ 2-6 λ3λ3 λ 2-m2 SATLIFE SATINC Cor(Satlife, Satinc)=.51 ρ 21 =

13
13 Comparison of QSR & PTR Standardized estimates of reliability coefficients QSRCPTRC λ1λ1 λ 2-6 λ3λ3 λ 2-m2 SATLIFE SATINC Cor(Satlife, Satinc)=.51 ρ 21 =.71

14
14 Comparison of QSR & PTR Standardized estimates of reliability coefficients QSRCPTRC λ1λ1 λ 2-6 λ3λ3 λ 2-m2 SATLIFE SATINC Cor(Satlife, Satinc)=.51 ρ 21 =

15
15 Comparison of QSR & PTR It appears that the QSR-estimates are wrong. Whats wrong with the quasi-simplex reliability? The key is provided by the VAS model (Van der Veld & Saris, 2004) It can be derived that: QSR=q(vas)*c(vas) This is proven in the accompanying paper, but it can be illustrated too.

16
16 The VAS Model For details, visit Session: Nonattitudes and informed opinions at Thursday 11:30 Chair: Peter Neijens

17
17 The quasi-simplex model

18
18 The VAS Model For details, visit Session: Nonattitudes and informed opinions at Thursday 11:30 Chair: Peter Neijens

19
19 The VAS Model - Estimation of Q, S, & C RUSSET Data: 3 waves, 6 times the question (n=627). Satlife: How satisfied are you with your current life as a whole? Satinc: How satisfied are you with your familys current financial situation?

20
20 Comparison of QSR & PTR & VAS Standardized estimates of reliability coefficients QSPTVAS λ 2-6 q 2-6 c 2-6 q*c SATLIFE SATINC QSRC=q*c QSRCsatlife=.81*.75=.60 QSRCsatinc=.78*.70=.54

21
21 Conclusion The quasi-simplex model has parameters for –Stability –Reliability The reliability in that model is not correct, It is the product of: –The quality of the measurement instrument –The opinion crystallization The stability is correct [see paper].

22
22 Conclusion How do you feel about public security these days? With the QSM all the considerations that are unique for a specific time are not in the variable S. Hence such considerations are not part of the stability. One should be aware of this. It depends on the object of study, whether this model is appropriate to estimate the stability! It is definitely not correct for reliability estimation!

Similar presentations

© 2016 SlidePlayer.com Inc.

All rights reserved.

Ads by Google