Finding the Perfect Match Minimizing Bias & Increasing Representativness Through Sample Matching Lynn Vavreck UCLA Political Science & Polimetrix

Lynn Vavreck UCLA Political Science The Representative Sample 1. How to Judge Representativeness 2. Ways that Surveys Miss the Target 2. Methods that Generate Representativeness in Theory 3. How to get Representative Samples in Practice 4. Does it “Work”? Representativenes s Practice Theory

Lynn Vavreck UCLA Political Science How to Judge Representativeness  1 Possibility: Know population characteristics, compare sample to population  2 Possibility: Identify target that represents population, compare sample to target General Population – Census, US CPS Registered Voters – Exit Polls, Registered Voter Files  What we often do: Comparing Internet to Phone samples -- comparing 1 sampling mode/design to another instead of a sampling mode/design to the target.

Lynn Vavreck UCLA Political Science Ways we can Miss the Target  Sampling Error The standard deviation of estimates obtained through repeated sampling following the same procedure. Error due to chance.  Bias Error as a result of gaps in the data are not benign but are systematically related to the phenomenon being modeled. Violation of ignorability.

Lynn Vavreck UCLA Political Science Ways to Get Close to Target  Theory Simple Randomness – Each observation has an equal chance of being selected into the sample  Practice Various sampling techniques (cluster, stratify, choose blocks, households, and then people, RDD, List-based methods … ) Weights to compensate for non-response (different techniques, different constructions) As usable sample data deviates farther from targets, weights get bigger

Lynn Vavreck UCLA Political Science Sample Matching  As a method of reducing sample bias  Can be used with various sampling methods  Involves generating matches among people on a given set of characteristics that are known before the survey is administered

Lynn Vavreck UCLA Political Science How it Works  Enumerate target population at individual level (with data on characteristics) if possible  Draw a random sample, T,of size N from the target population.  For every i in T, find the closest match in an available pool (panel) by minimizing a distance function, d(x,y).  More than one match per i can be identified if desired (this allows for replacement)

Available Respondents Population Target Sample Matched Sample

Lynn Vavreck UCLA Political Science What is Needed to Do this?  A large number of people from whom to select matches  For Political Scientists, coverage of specific types of people Low interest/political sophistication/knowledge Independents/Moderates Low propensity voters  How do you ensure coverage of these parts of the population?

Lynn Vavreck UCLA Political Science Everyone Cares about Something  People like to give their opinions … about something!  Find things that people identify with and ask them about those things Sports Hobbies Entertainment Celebrity Gossip True Crime

Lynn Vavreck UCLA Political Science Other Methods  Many other methods for finding hard to reach populations  Panel Managements is critical Incentives Frequency of Surveys Signals that Someone is listening Interesting tools/widgets

Lynn Vavreck UCLA Political Science Even if You Get Them …  Are low propensity voters who are willing to take surveys on the Internet “like” low propensity voters in general?  You can get the marginals right, but do the mechanisms still work?

Lynn Vavreck UCLA Political Science Party ID and Ideology  Using NES 04, Annenberg 04, and CCES 06 we looked for the relationship between party identification and ideology for different levels of political knowledge Looking for degree of “constraint” among these two things A relationship/mechanism we know exists in regularity (different modes, samples, years, question wordings)

Lynn Vavreck UCLA Political Science Hill, Lo, Vavreck, & Zaller

Lynn Vavreck UCLA Political Science 2004 Bush Vote Share by State