1. Propensity Score Adjustments for Internet Survey of Voting Behavior:
A Case in Japan
Tetsuro Kobayashi
National Institute of Informatics, Japan
General Online Research 09 April 7, 2009 Vienna

2. Expectations toward Internet Surveys
Deterioration of survey environments
Changes in living environment
Increasing “privacy” sensitivity
⇒Expectations toward Internet surveys
Definition of Internet survey
Survey methods in which research entities (normally research companies) recruit potential survey respondents and then have a selected group of respondents fill out one-line questionnaires

3. Potential problems in applying to social survey
Ambiguous definition of population
Originates from panel construction
Purposive sampling
Non-negligible divergences from probability sampling
Problem of representativeness
larger percentages of housewives in their thirties
“professional respondents”
⇒Quata sampling
Divergences still remain
Advantages
Collecting samples satisfying specific conditions

4. Propensity Score Adjustment
Propensity Score (Rosenbaum and Rubin, 1983)
The probability of a given respondent belonging to a particular group, which is calculated from covariates information.
‘Typicality’ of a given respondent in a particular group
Covariates
Variables that can be considered to affect variables that are targets of adjustment AND survey assignments (probability sampling surveys vs. internet surveys)

5. Propensity Score Adjustment
Probability sampling
survey
Internet
Covariates
Logistic (or Probit) regression
Independent vars: Covariates
Dependent vars: Survey assignment
probability sampling surveys vs. internet surveys
Predict the probability(e) for each respondents
e = propensity score

6. Strongly Ignorable Treatment Assignment Condition
Selection of covariates is extremely important
But unclear in past studies
Effective selection criteria for survey data
Hoshino & Shigemasu(2004), Hoshino & Maeda(2006)
Strongly Ignorable Treatment Assignment
Survey assignments depend on covariates yet do NOT directly depend on the dependent variables (the variables that are targets of adjustments)

7. Strongly Ignorable Treatment Assignment Condition
1. Select covariates that are stable within individuals and have the potential to be used continuously in both internet surveys and probability sampling surveys. 2. Select covariates that differentiate internet surveys from probability sampling surveys (in particular, select covariates that yield larger standardized partial regression coefficients and smaller p-values for probability significance, based on the results of analyses such as student’s t-test and logistic repression analysis). 3. Select covariates whose partial repression coefficients in predicting the target variables of adjustments are either positive or negative in both of the two sets of data. Especially, select covariates whose absolute values of standardized partial regression coefficients are relatively large. 4. Remove covariates from the set of covariates selected with the above criteria so as to make the sum of square error smaller (or minimize the increase).

8. Strongly Ignorable Treatment Assignment Condition
Criterion 3 is particularly important
Adjustments can have adverse effects if covariates are selected only to maximize the predicting power of a logistic regression analysis that predicts survey assignments (Hoshino & Maeda, 2006)
We follow the four when determining covariates for calculating propensity scores and verify the effectiveness of propensity score adjustments.

9. Survey Data Internet Survey Probability Sampling Survey
Three-wave panel design survey conducted during the last days before the Upper House election of Japan in summer 2007.
Respondents
Those who “regularly obtain information about social and political issues from internet resources”
Purposive sampling
An intensive panel survey that spans three days would have been difficult to conduct by traditional interview or telephone surveys
Probability Sampling Survey
Two-stage stratified random sampling; N=1373（54.9%）
Sponsored by the Ministry of Education, Science, Sports and Culture, Grant-in-Aid(PI: Ken’ichi Ikeda; assignment number )

10. Dependent variables: The targets of adjustments
Party ID

11. Dependent variables: The targets of adjustments
Political parties for which Rs voted in the 2007 Upper House election
Propositional representation seats
Electoral district seats

12. Candidates for Covariates
Gender
Age
Education level
Annual household income
Occupation
Subjective social class
Conservative vs. Progressive ideology
Social tolerance
Participation in organizations
Participation in informal groups
Openness and verticalness of the participating organizations and informal groups
Exposure to election campaigns

13. Selection of Covariates
1. Select covariates that are stable within individuals and have the potential to be used continuously in both internet surveys and probability sampling surveys.
⇒OK
2. Select covariates that differentiate internet surveys from probability sampling surveys (in particular, select covariates that yield larger standardized partial regression coefficients and smaller p-values for probability significance, based on the results of analyses such as student’s t-test and logistic repression analysis).

14. Selection of Covariates
3. Select covariates whose partial repression coefficients in predicting the target variables of adjustments are either positive or negative in both of the two sets of data. Especially, select covariates whose absolute values of standardized partial regression coefficients are relatively large.
Multi-nominal logit models
Dependent vars: Party ID and Votes
Independent vars: Each of the covariate candidates

15. Selection of Covariates

16. Calculation of Propensity Score
Method 1: Only the demographic variables (gender, age, annual household income, and subjective social class)
Method 2: Besides the demographic variables, political attitudes and behaviors were also included in this method. (gender, age, annual household income, subjective social class, conservative or progressive ideology, participation in organizations, exposure to political campaigns)
Method 3: Criterion 3 was disregarded. With all the covariate candidates as independent variables, a logistic regression analysis that predicts survey assignments was performed, and covariates were selected with a variable reduction technique with 0.05 as the threshold for p-values.

17. Calculation of Propensity Score
Logistic regressions
Dependent var: Survey assignment
probability sampling surveys vs. internet surveys
Independent vars: Each sets of covariates
Calculate the predicted probability (e) of each R being assigned to Internet survey
If e<.10, then replace e with .10 as the lower limit.
Calculate (1-e)/e, and use it as the weight for Internet survey respondents.

18. Adjustment of Party ID
Setting the distribution of probability sampling survey as the true value, the squared errors with the three methods were calculated.
The adjustments were unsuccessful because the square errors were larger after the adjustments than before.

19. Adjustment of Party ID

20. Adjustment of Votes The squared errors of electoral district
The squared errors of proportional representation

21. Adjustment of Votes Propositional representation seats
Electoral district seats

22. Why failed in Party ID?
Case 1: The selected covariates have explanatory power over assignments but do NOT have explanatory power over dependent variables that are the targets of adjustments.
Unlikely. Pseudo R2s of multi-nominal logit models do not differ much between party ID and votes.
Case 2: The data from the probability sampling survey diverged from the true values of the population
Case 3: The number of covariate candidates was too small
Case 4: The check of Strongly Ignorable Treatment Assignment is retroactive

23. Why failed in Party ID?
Strongly Ignorable Treatment Assignment Condition is not satisfied if survey assignment directly depends on party ID.
Those who “regularly obtain information about social and political issues from internet resources”
Strong interests in politics and tended to support particular political parties
The people who support a particular political party might have a stronger tendency to participate in the internet survey

24. Why failed in Party ID?
If survey assignments depend on variables that are not reducible to basic social attributes such as gender and age, adjustments with covariates only would be difficult.

25. Dilemma of Adjustments in Social Survey
Advantages of Internet survey
Purposively extracting samples that meet certain conditions that are difficult to capture with probability sampling surveys
Intensive panel design with short intervals
However, Strongly Ignorable Treatment Assignment condition may be violated if the criteria for purposive sampling themselves depend on dependent variables that are the targets of adjustment
Adjustments of internet surveys in voting behavior and public opinion research can be effective if they do not involve purposive sampling with particular political inclinations